Remote NetCDF Dataset Access via OPeNDAP

EsgDataset provides a unified interface for accessing NetCDF data remotely via OPeNDAP protocol. It wraps RNetCDF functions and provides convenient methods for subsetting, slicing, and reading data without downloading entire files.

The class supports three levels of interfaces:

Basic layer: Direct wrappers around RNetCDF functions
Middle layer: Convenient methods for subsetting by time/space
High layer: Data manipulation and format conversion

It also supports aggregating multiple files into a single logical dataset, automatically handling time dimension concatenation.

Author

Hongyuan Jia

Active bindings

url: The OPeNDAP URL(s)
is_open: Whether the connection is open
is_aggregated: Whether the dataset contains multiple files
file_count: Number of files in the dataset
time_filter: A result-level time filter recorded by EsgResultFile$filter_time() or EsgResultAggregation$filter_time(), or NULL.

Methods

Method `new()`

Create a new EsgDataset object

Usage

EsgDataset$new(urls)

Arguments

urls: A character vector of OPeNDAP URLs. Can be a single URL or multiple URLs for a multi-file dataset.

Returns

An EsgDataset object.

Examples

\dontrun{
# Single file
ds <- EsgDataset$new("https://example.com/data.nc")

# Multiple files
ds <- EsgDataset$new(c("url1.nc", "url2.nc"))
}

Method `open()`

Open OPeNDAP connection(s)

Usage

EsgDataset$open(
  async = FALSE,
  timeout = NULL,
  progress = getOption("epwshiftr.progress", interactive())
)

Arguments

async: If TRUE, first validates opening in a one-shot worker, then re-opens caller-owned handles before returning so the dataset remains opened after open() returns. The caller still receives the final EsgDataset object itself rather than a Mirai/Future-like handle. Default: FALSE.
timeout: Optional positive number of seconds for the async worker pre-open phase. Only supported when async = TRUE. It does not limit the final caller-owned reopen that makes the returned dataset stay opened.
progress: Whether to show a progress bar while opening NetCDF/OPeNDAP handles. By default the package option epwshiftr.progress is used, falling back to interactive().

Returns

The EsgDataset object itself, invisibly.

Examples

\dontrun{
ds$open()
# Returns the opened dataset directly; no Mirai/Future to collect.
ds$open(async = TRUE, timeout = 10)
}

Method `close()`

Close OPeNDAP connection(s)

Usage

EsgDataset$close()

Returns

The EsgDataset object itself, invisibly.

Examples

\dontrun{
ds$close()
}

Method `slice()`

Select files from this dataset by file position.

$slice() creates a new EsgDataset with a subset of the current dataset URLs. This is a file/URL-level operation; NetCDF variable, dimension, time, and spatial slicing is still performed by $var_get(), $read_array(), $read_data_table(), and $read_region().

Usage

EsgDataset$slice(i, reopen = FALSE)

Arguments

i: A positive or negative integer vector, or a logical vector with one value per file.
reopen: Whether to open the returned dataset when the current dataset is already open. If FALSE (default), slicing an open dataset raises an error because RNetCDF handles cannot be safely shared between dataset objects.

Returns

A new EsgDataset object.

Method `reachable()`

Probe whether this dataset's current files or URLs are reachable.

$reachable() checks the actual URLs or local paths stored in the dataset. It does not reuse reachability checks from an EsgResult; opened datasets, fallback downloads, and manually created datasets are always evaluated from their current url values.

Usage

EsgDataset$reachable(level = c("data_node", "url"), probe = NULL)

Arguments

level: Probe level. "data_node" probes the root URL of each remote data node; "url" probes the actual dataset URL. Default: "data_node".
probe: Optional named list of probe settings. Supported fields are timeout, concurrency, network_policy, cache_seconds, and cache_failures_seconds.

Returns

A data.table with columns file_index, source_index, data_node, service, url, reachable, latency_ms, error, probe_level, probe_url, and probe_cached.

Method `file_inq()`

Get file information

Usage

EsgDataset$file_inq(index = 1L)

Arguments

index: File index for multi-file datasets. Default: 1L.

Returns

A list with file information.

Examples

\dontrun{
info <- ds$file_inq()
}

Method `var_inq()`

Get variable information

Usage

EsgDataset$var_inq(var, index = 1L)

Arguments

var: Variable name or ID.
index: File index for multi-file datasets. Default: 1L.

Returns

A list with variable information.

Examples

\dontrun{
var_info <- ds$var_inq("tas")
}

Method `dim_inq()`

Get dimension information

Usage

EsgDataset$dim_inq(dim, index = 1L)

Arguments

dim: Dimension name or ID.
index: File index for multi-file datasets. Default: 1L.

Returns

A list with dimension information.

Examples

\dontrun{
dim_info <- ds$dim_inq("time")
}

Method `att_get()`

Get attribute value

Usage

EsgDataset$att_get(var, att, index = 1L)

Arguments

var: Variable name or ID, or "NC_GLOBAL" for global attributes.
att: Attribute name.
index: File index for multi-file datasets. Default: 1L.

Returns

The attribute value.

Examples

\dontrun{
units <- ds$att_get("tas", "units")
}

Method `var_get()`

Read variable data

Usage

EsgDataset$var_get(
  var,
  start = NULL,
  count = NULL,
  index = 1L,
  collapse = FALSE,
  async = FALSE,
  timeout = NULL
)

Arguments

var: Variable name or ID.
start: Starting indices (1-based). If NULL, starts from beginning.
count: Number of values to read. If NULL, reads all.
index: File index for multi-file datasets. Default: 1L.
collapse: Whether to collapse result. Default: FALSE.
async: If TRUE, perform the variable read in a one-shot worker and return the final array directly once complete. No Mirai/Future object is exposed. Default: FALSE.
timeout: Optional positive number of seconds for the async worker phase. Only supported when async = TRUE.

Returns

An array with variable data.

Examples

\dontrun{
data <- ds$var_get("tas")
data_subset <- ds$var_get("tas", start = c(1, 1, 1), count = c(10, 10, 1))
# Returns the final array directly; no Mirai/Future handling required.
data_async <- ds$var_get("tas", async = TRUE, timeout = 10)
}

Method `get_variables()`

List all variables in the dataset

Usage

EsgDataset$get_variables(index = 1L)

Arguments

index: File index for multi-file datasets. Default: 1L.

Returns

A character vector of variable names.

Examples

\dontrun{
vars <- ds$get_variables()
}

Method `get_dimensions()`

List all dimensions in the dataset

Usage

EsgDataset$get_dimensions(index = 1L)

Arguments

index: File index for multi-file datasets. Default: 1L.

Returns

A character vector of dimension names.

Examples

\dontrun{
dims <- ds$get_dimensions()
}

Method `get_time_axis()`

Get time axis information

Usage

EsgDataset$get_time_axis(index = 1L)

Arguments

index: File index for multi-file datasets. Default: 1L.

Returns

A list containing time values, units, and calendar.

Examples

\dontrun{
time_info <- ds$get_time_axis()
}

Method `get_spatial_grid()`

Get spatial grid information (latitude and longitude)

Usage

EsgDataset$get_spatial_grid(index = 1L)

Arguments

index: File index for multi-file datasets. Default: 1L.

Returns

A list containing latitude and longitude values.

Examples

\dontrun{
grid <- ds$get_spatial_grid()
}

Method `read_array()`

Read variable data as a list of arrays (one per file)

Usage

EsgDataset$read_array(
  variable,
  start = NULL,
  count = NULL,
  collapse = FALSE,
  async = FALSE,
  timeout = NULL
)

Arguments

variable: Variable name.
start: Starting indices. If NULL, starts from beginning.
count: Number of values to read. If NULL, reads all.
collapse: Whether to collapse result. Default: FALSE.
async: If TRUE, read array values in a one-shot worker and return the final list directly once complete. No Mirai/Future object is exposed. Default: FALSE.
timeout: Optional positive number of seconds for the async worker phase. Only supported when async = TRUE.

Returns

A list of arrays with variable data. Each element corresponds to a file in the dataset.

Examples

\dontrun{
data_list <- ds$read_array("tas")
data <- data_list[[1]]
# Returns the final list directly; no Mirai/Future handling required.
data_list_async <- ds$read_array("tas", async = TRUE, timeout = 10)
}

Method `read_data_table()`

Read variable data as a list of data.table (one per file)

Usage

EsgDataset$read_data_table(
  variable,
  start = NULL,
  count = NULL,
  rbind = FALSE,
  async = FALSE,
  timeout = NULL
)

Arguments

variable: Variable name.
start: Starting indices. If NULL, starts from beginning.
count: Number of values to read. If NULL, reads all.
rbind: If TRUE, return a single data.table by row-binding the per-file results with data.table::rbindlist(..., idcol = "file_index"). Default: FALSE.
async: If TRUE, offload the array read phase to a one-shot worker and still return the final data.table result directly. No Mirai/Future object is exposed. Default: FALSE.
timeout: Optional positive number of seconds for the async worker phase. Only supported when async = TRUE.

Returns

If rbind = FALSE, a list of data.table (one per file). If rbind = TRUE, a single data.table with an extra file_index column.

Examples

\dontrun{
dt_list <- ds$read_data_table("tas")
dt <- dt_list[[1]]
dt_all <- ds$read_data_table("tas", rbind = TRUE)
# Returns the final data.table directly; no Mirai/Future handling required.
dt_async <- ds$read_data_table("tas", async = TRUE, timeout = 10)
}

Method `read_region()`

Read variable values near a target coordinate and optional time range

Usage

EsgDataset$read_region(
  variable,
  lon,
  lat,
  time = "auto",
  method = "nearest",
  rbind = TRUE,
  async = FALSE,
  timeout = NULL
)

Arguments

variable: Character vector of variable names.
lon: Target longitude.
lat: Target latitude.
time: Time range to read. Use "auto" to reuse the time range recorded by EsgResultFile$filter_time() or EsgResultAggregation$filter_time() when available; if no recorded range exists, all times are read. Use NULL to always read the full time axis. A length-2 character, Date, or POSIXt range is parsed in UTC and used explicitly. Default: "auto".
method: Grid extraction method. One of "nearest", "idw", "bilinear", or "mean". Default: "nearest".
rbind: If TRUE, return one data.table. If FALSE, return a list of per-file, per-variable data.tables. Default: TRUE.
async: If TRUE, offload each NetCDF variable read to a one-shot worker. Default: FALSE.
timeout: Optional positive number of seconds for each async read. Only supported when async = TRUE.

Returns

A data.table or list of data.tables with columns including file_index, variable, time, lon, lat, method, and value. The "grid_sources" attribute records contributing grid coordinates and weights.

Examples

\dontrun{
dt <- ds$read_region(
    variable = c("tas", "hurs"),
    lon = 103.98,
    lat = 1.37,
    time = c("2050-01-01", "2050-12-31")
)
}

Method `selection()`

Return file selection provenance for this dataset.

$selection() maps the current dataset file positions back to the result rows that produced the dataset when that information is available. It does not record intermediate filter steps.

Usage

EsgDataset$selection()

Returns

A list with source_count, source_num_found, and source_indices.

Method `print()`

Print dataset summary

Usage

EsgDataset$print()

Returns

The EsgDataset object itself, invisibly.

Method `clone()`

The objects of this class are cloneable with this method.

Usage

EsgDataset$clone(deep = FALSE)

Arguments

deep: Whether to make a deep clone.

Examples


## ------------------------------------------------
## Method `EsgDataset$new`
## ------------------------------------------------

if (FALSE) { # \dontrun{
# Single file
ds <- EsgDataset$new("https://example.com/data.nc")

# Multiple files
ds <- EsgDataset$new(c("url1.nc", "url2.nc"))
} # }

## ------------------------------------------------
## Method `EsgDataset$open`
## ------------------------------------------------

if (FALSE) { # \dontrun{
ds$open()
# Returns the opened dataset directly; no Mirai/Future to collect.
ds$open(async = TRUE, timeout = 10)
} # }

## ------------------------------------------------
## Method `EsgDataset$close`
## ------------------------------------------------

if (FALSE) { # \dontrun{
ds$close()
} # }

## ------------------------------------------------
## Method `EsgDataset$file_inq`
## ------------------------------------------------

if (FALSE) { # \dontrun{
info <- ds$file_inq()
} # }

## ------------------------------------------------
## Method `EsgDataset$var_inq`
## ------------------------------------------------

if (FALSE) { # \dontrun{
var_info <- ds$var_inq("tas")
} # }

## ------------------------------------------------
## Method `EsgDataset$dim_inq`
## ------------------------------------------------

if (FALSE) { # \dontrun{
dim_info <- ds$dim_inq("time")
} # }

## ------------------------------------------------
## Method `EsgDataset$att_get`
## ------------------------------------------------

if (FALSE) { # \dontrun{
units <- ds$att_get("tas", "units")
} # }

## ------------------------------------------------
## Method `EsgDataset$var_get`
## ------------------------------------------------

if (FALSE) { # \dontrun{
data <- ds$var_get("tas")
data_subset <- ds$var_get("tas", start = c(1, 1, 1), count = c(10, 10, 1))
# Returns the final array directly; no Mirai/Future handling required.
data_async <- ds$var_get("tas", async = TRUE, timeout = 10)
} # }

## ------------------------------------------------
## Method `EsgDataset$get_variables`
## ------------------------------------------------

if (FALSE) { # \dontrun{
vars <- ds$get_variables()
} # }

## ------------------------------------------------
## Method `EsgDataset$get_dimensions`
## ------------------------------------------------

if (FALSE) { # \dontrun{
dims <- ds$get_dimensions()
} # }

## ------------------------------------------------
## Method `EsgDataset$get_time_axis`
## ------------------------------------------------

if (FALSE) { # \dontrun{
time_info <- ds$get_time_axis()
} # }

## ------------------------------------------------
## Method `EsgDataset$get_spatial_grid`
## ------------------------------------------------

if (FALSE) { # \dontrun{
grid <- ds$get_spatial_grid()
} # }

## ------------------------------------------------
## Method `EsgDataset$read_array`
## ------------------------------------------------

if (FALSE) { # \dontrun{
data_list <- ds$read_array("tas")
data <- data_list[[1]]
# Returns the final list directly; no Mirai/Future handling required.
data_list_async <- ds$read_array("tas", async = TRUE, timeout = 10)
} # }

## ------------------------------------------------
## Method `EsgDataset$read_data_table`
## ------------------------------------------------

if (FALSE) { # \dontrun{
dt_list <- ds$read_data_table("tas")
dt <- dt_list[[1]]
dt_all <- ds$read_data_table("tas", rbind = TRUE)
# Returns the final data.table directly; no Mirai/Future handling required.
dt_async <- ds$read_data_table("tas", async = TRUE, timeout = 10)
} # }

## ------------------------------------------------
## Method `EsgDataset$read_region`
## ------------------------------------------------

if (FALSE) { # \dontrun{
dt <- ds$read_region(
    variable = c("tas", "hurs"),
    lon = 103.98,
    lat = 1.37,
    time = c("2050-01-01", "2050-12-31")
)
} # }

Author

Active bindings

Methods

Public methods

Method new()

Usage

Arguments

Returns

Examples

Method open()

Usage

Arguments

Returns

Examples

Method close()

Usage

Returns

Examples

Method slice()

Usage

Arguments

Returns

Method reachable()

Usage

Arguments

Returns

Method file_inq()

Usage

Arguments

Returns

Examples

Method var_inq()

Usage

Arguments

Returns

Examples

Method dim_inq()

Usage

Arguments

Returns

Examples

Method att_get()

Usage

Arguments

Returns

Examples

Method var_get()

Usage

Arguments

Returns

Examples

Method get_variables()

Usage

Arguments

Returns

Examples

Method get_dimensions()

Usage

Arguments

Returns

Examples

Method get_time_axis()

Usage

Arguments

Returns

Examples

Method get_spatial_grid()

Usage

Arguments

Returns

Examples

Method read_array()

Usage

Arguments

Returns

Examples

Method read_data_table()

Usage

Arguments

Method `new()`

Method `open()`

Method `close()`

Method `slice()`

Method `reachable()`

Method `file_inq()`

Method `var_inq()`

Method `dim_inq()`

Method `att_get()`

Method `var_get()`

Method `get_variables()`

Method `get_dimensions()`

Method `get_time_axis()`

Method `get_spatial_grid()`

Method `read_array()`

Method `read_data_table()`

Method `read_region()`

Method `selection()`

Method `print()`

Method `clone()`