EsgDataset provides a unified interface for accessing NetCDF data
remotely via OPeNDAP protocol. It wraps RNetCDF functions and provides
convenient methods for subsetting, slicing, and reading data without
downloading entire files.
The class supports three levels of interfaces:
Basic layer: Direct wrappers around RNetCDF functions
Middle layer: Convenient methods for subsetting by time/space
High layer: Data manipulation and format conversion
It also supports aggregating multiple files into a single logical dataset, automatically handling time dimension concatenation.
urlThe OPeNDAP URL(s)
is_openWhether the connection is open
is_aggregatedWhether the dataset contains multiple files
file_countNumber of files in the dataset
time_filterA result-level time filter recorded by
EsgResultFile$filter_time() or
EsgResultAggregation$filter_time(), or NULL.
EsgDataset$new()Create a new EsgDataset object
EsgDataset$new(urls)urlsA character vector of OPeNDAP URLs. Can be a single URL or multiple URLs for a multi-file dataset.
# Single file
ds <- EsgDataset$new("https://example.com/data.nc")
# Multiple files
ds <- EsgDataset$new(c("url1.nc", "url2.nc"))EsgDataset$open()Open OPeNDAP connection(s)
EsgDataset$open(
async = FALSE,
timeout = NULL,
progress = getOption("epwshiftr.progress", interactive())
)asyncIf TRUE, first validates opening in a one-shot worker,
then re-opens caller-owned handles before returning so the
dataset remains opened after open() returns. The caller still
receives the final EsgDataset object itself rather than a
Mirai/Future-like handle. Default: FALSE.
timeoutOptional positive number of seconds for the async
worker pre-open phase. Only supported when async = TRUE.
It does not limit the final caller-owned reopen that makes the
returned dataset stay opened.
progressWhether to show a progress bar while opening
NetCDF/OPeNDAP handles. By default the package option
epwshiftr.progress is used, falling back to interactive().
EsgDataset$slice()Select files from this dataset by file position.
$slice() creates a new EsgDataset with a subset of the current
dataset URLs. This is a file/URL-level operation; NetCDF variable,
dimension, time, and spatial slicing is still performed by
$var_get(), $read_array(), $read_data_table(), and
$read_region().
iA positive or negative integer vector, or a logical vector with one value per file.
reopenWhether to open the returned dataset when the current
dataset is already open. If FALSE (default), slicing an
open dataset raises an error because RNetCDF handles cannot
be safely shared between dataset objects.
EsgDataset$reachable()Probe whether this dataset's current files or URLs are reachable.
$reachable() checks the actual URLs or local paths stored in the
dataset. It does not reuse reachability checks from an EsgResult;
opened datasets, fallback downloads, and manually created datasets
are always evaluated from their current url values.
EsgDataset$reachable(level = c("data_node", "url"), probe = NULL)levelProbe level. "data_node" probes the root URL of each
remote data node; "url" probes the actual dataset URL.
Default: "data_node".
probeOptional named list of probe settings. Supported fields
are timeout, concurrency, network_policy,
cache_seconds, and cache_failures_seconds.
A data.table with columns
file_index, source_index, data_node, service, url,
reachable, latency_ms, error, probe_level,
probe_url, and probe_cached.
EsgDataset$att_get()Get attribute value
EsgDataset$var_get()Read variable data
EsgDataset$var_get(
var,
start = NULL,
count = NULL,
index = 1L,
collapse = FALSE,
async = FALSE,
timeout = NULL
)varVariable name or ID.
startStarting indices (1-based). If NULL, starts from beginning.
countNumber of values to read. If NULL, reads all.
indexFile index for multi-file datasets. Default: 1L.
collapseWhether to collapse result. Default: FALSE.
asyncIf TRUE, perform the variable read in a one-shot
worker and return the final array directly once complete.
No Mirai/Future object is exposed. Default: FALSE.
timeoutOptional positive number of seconds for the async
worker phase. Only supported when async = TRUE.
EsgDataset$get_time_axis()Get time axis information
indexFile index for multi-file datasets. Default: 1L.
variableOptional data-variable name whose ordered temporal
cell_methods should be parsed and validated. Default: NULL.
A list containing time values, units, calendar, canonical CF
coordinates, optional CF interval bounds, and axis length.
raw_values retains the original numeric time coordinates;
bounds$raw_values retains bounds in their declared dimension order.
Reference time zones are normalized to UTC without changing raw units.
standard and gregorian use the Julian/Gregorian reform; their
native dates before 1582-10-15 differ from proleptic Gregorian POSIX
labels. Julian and fixed-calendar POSIX values are surrogate timestamps;
canonical CF fields remain authoritative for calendar dates.
Units month and year are fixed UDUNITS durations, not calendar
increments. Fractional seconds retain available numeric precision.
kind distinguishes ordinary axes from "climatology". The latter
has a separate climatology descriptor with original boundary values,
dimension order and native endpoints, while bounds remains NULL.
climatology$dimensions records the RNetCDF array dimension names;
climatology$time_dimension gives the native time position.
Supplying variable returns its original statistics$raw cell methods
and ordered statistics$operations; the climatology interpretation
identifies annual or midnight wrapping and full-day daily intervals.
These are source-computed statistical values over possibly disjoint
subintervals, not observations spanning the enclosing endpoint range.
No climatology is recalculated or expanded into ordinary samples.
CMIP6 monthly extrema can instead use ordered within days and
over days methods on ordinary interval bounds. Their statistics are
retained without adding a climatology descriptor or inferring daily
observations from the monthly summaries.
Leap-second calendars and explicitly defined calendars are unsupported.
Ordinary equal bounds are retained as native metadata. Weather reads accept
them only for explicit time: point values at that same native time;
interval-valued variables require positive native durations.
EsgDataset$read_array()Read variable data as a list of arrays (one per file)
EsgDataset$read_array(
variable,
start = NULL,
count = NULL,
collapse = FALSE,
async = FALSE,
timeout = NULL
)variableVariable name.
startStarting indices. If NULL, starts from beginning.
countNumber of values to read. If NULL, reads all.
collapseWhether to collapse result. Default: FALSE.
asyncIf TRUE, read array values in a one-shot worker and
return the final list directly once complete. No
Mirai/Future object is exposed. Default: FALSE.
timeoutOptional positive number of seconds for the async
worker phase. Only supported when async = TRUE.
EsgDataset$read_data_table()Read variable data as a list of data.table (one per file)
EsgDataset$read_data_table(
variable,
start = NULL,
count = NULL,
rbind = FALSE,
async = FALSE,
timeout = NULL
)variableVariable name.
startStarting indices. If NULL, starts from beginning.
countNumber of values to read. If NULL, reads all.
rbindIf TRUE, return a single data.table by row-binding
the per-file results with data.table::rbindlist(..., idcol = "file_index").
Default: FALSE.
asyncIf TRUE, offload the array read phase to a one-shot
worker and still return the final data.table result directly.
No Mirai/Future object is exposed. Default: FALSE.
timeoutOptional positive number of seconds for the async
worker phase. Only supported when async = TRUE.
Climatological variables retain time_kind = "climatology",
original time_value offsets and climatology_bound_* columns, native
endpoints, ordered cell methods and cycle interpretation. No temporal
averaging is performed by this reader. A source value name that
conflicts with coordinate/statistical columns receives _value
suffixes until unique, with a warning; source values are retained.
EsgDataset$read_region()Read variable values near a target coordinate and optional time range
EsgDataset$read_region(
variable,
lon,
lat,
time = "auto",
method = "nearest",
rbind = TRUE,
async = FALSE,
timeout = NULL
)variableCharacter vector of variable names.
lonTarget longitude.
latTarget latitude.
timeTime range to read. Use "auto" to reuse the time
range recorded by EsgResultFile$filter_time() or
EsgResultAggregation$filter_time() when available; if no
recorded range exists, all times are read. Use NULL to
always read the full time axis. A length-2 character,
Date, or POSIXt range is parsed in UTC and used
explicitly. Default: "auto".
methodGrid extraction method. One of "nearest", "idw",
"bilinear", or "mean". Default: "nearest".
rbindIf TRUE, return one data.table. If FALSE, return a
list of per-file, per-variable data.tables. Default: TRUE.
asyncIf TRUE, offload each NetCDF variable read to a
one-shot worker. Default: FALSE.
timeoutOptional positive number of seconds for each async
read. Only supported when async = TRUE.
A data.table or list of data.tables with columns including
file_index, variable, time, optional time_bound_start and
time_bound_end, canonical cf_* calendar coordinates including
cf_second_of_day (seconds since native-calendar midnight),
annual_phase, lon, lat, method, and value.
Climatological statistics require the raw readers and are rejected
here instead of being treated as ordinary weather observations.
The "grid_sources" attribute records contributing grid coordinates
and weights.
EsgDataset$read_neighborhood()Read the four enclosing native grid corners for multiple sites,
without applying spatial weights or aggregating their climate values.
For remote use, open a URL ending in #noprefetch&nocache to disable
netCDF-C automatic small-variable prefetch before selecting data.
The high-level File acquisition method applies these options itself.
variableCharacter vector of climate variables to read.
sitesA data.frame with unique site_id, lon and lat columns.
timeA native-calendar range, or a list of disjoint ranges.
on_chunkOptional function receiving each bounded raw block. With a callback, values are delivered rather than accumulated.
For read_neighborhood(), a list of values, neighborhoods, native CF time coordinates and actual read_slices. A list of time ranges selects their union without fetching intervening dates. Climatological axes retain original offsets, climatology boundaries, ordered per-variable cell methods and explicit statistical-kind markers. Ranges select representative time labels, not every date enclosed by climatology bounds. No source statistic is recalculated.
EsgDataset$selection()Return file selection provenance for this dataset.
$selection() maps the current dataset file positions back to the
result rows that produced the dataset when that information is
available. It does not record intermediate filter steps.
## ------------------------------------------------
## Method `EsgDataset$new()`
## ------------------------------------------------
if (FALSE) { # \dontrun{
# Single file
ds <- EsgDataset$new("https://example.com/data.nc")
# Multiple files
ds <- EsgDataset$new(c("url1.nc", "url2.nc"))
} # }
## ------------------------------------------------
## Method `EsgDataset$open()`
## ------------------------------------------------
if (FALSE) { # \dontrun{
ds$open()
# Returns the opened dataset directly; no Mirai/Future to collect.
ds$open(async = TRUE, timeout = 10)
} # }
## ------------------------------------------------
## Method `EsgDataset$close()`
## ------------------------------------------------
if (FALSE) { # \dontrun{
ds$close()
} # }
## ------------------------------------------------
## Method `EsgDataset$file_inq()`
## ------------------------------------------------
if (FALSE) { # \dontrun{
info <- ds$file_inq()
} # }
## ------------------------------------------------
## Method `EsgDataset$var_inq()`
## ------------------------------------------------
if (FALSE) { # \dontrun{
var_info <- ds$var_inq("tas")
} # }
## ------------------------------------------------
## Method `EsgDataset$dim_inq()`
## ------------------------------------------------
if (FALSE) { # \dontrun{
dim_info <- ds$dim_inq("time")
} # }
## ------------------------------------------------
## Method `EsgDataset$att_get()`
## ------------------------------------------------
if (FALSE) { # \dontrun{
units <- ds$att_get("tas", "units")
} # }
## ------------------------------------------------
## Method `EsgDataset$var_get()`
## ------------------------------------------------
if (FALSE) { # \dontrun{
data <- ds$var_get("tas")
data_subset <- ds$var_get("tas", start = c(1, 1, 1), count = c(10, 10, 1))
# Returns the final array directly; no Mirai/Future handling required.
data_async <- ds$var_get("tas", async = TRUE, timeout = 10)
} # }
## ------------------------------------------------
## Method `EsgDataset$get_variables()`
## ------------------------------------------------
if (FALSE) { # \dontrun{
vars <- ds$get_variables()
} # }
## ------------------------------------------------
## Method `EsgDataset$get_dimensions()`
## ------------------------------------------------
if (FALSE) { # \dontrun{
dims <- ds$get_dimensions()
} # }
## ------------------------------------------------
## Method `EsgDataset$get_time_axis()`
## ------------------------------------------------
if (FALSE) { # \dontrun{
time_info <- ds$get_time_axis()
} # }
## ------------------------------------------------
## Method `EsgDataset$get_spatial_grid()`
## ------------------------------------------------
if (FALSE) { # \dontrun{
grid <- ds$get_spatial_grid()
} # }
## ------------------------------------------------
## Method `EsgDataset$read_array()`
## ------------------------------------------------
if (FALSE) { # \dontrun{
data_list <- ds$read_array("tas")
data <- data_list[[1]]
# Returns the final list directly; no Mirai/Future handling required.
data_list_async <- ds$read_array("tas", async = TRUE, timeout = 10)
} # }
## ------------------------------------------------
## Method `EsgDataset$read_data_table()`
## ------------------------------------------------
if (FALSE) { # \dontrun{
dt_list <- ds$read_data_table("tas")
dt <- dt_list[[1]]
dt_all <- ds$read_data_table("tas", rbind = TRUE)
# Returns the final data.table directly; no Mirai/Future handling required.
dt_async <- ds$read_data_table("tas", async = TRUE, timeout = 10)
} # }
## ------------------------------------------------
## Method `EsgDataset$read_region()`
## ------------------------------------------------
if (FALSE) { # \dontrun{
dt <- ds$read_region(
variable = c("tas", "hurs"),
lon = 103.98,
lat = 1.37,
time = c("2050-01-01", "2050-12-31")
)
} # }