ESGF Query results for File type — EsgResultFile • epwshiftr

EsgResultFile is a class that represents query results for File type from ESGF search RESTful API.

In general, there is no need to create an EsgResultDataset manually. Usually, it is created by calling EsgResultDataset$collect().

Author

Hongyuan Jia

Super class

EsgResult -> EsgResultFile

Active bindings

filename: A character vector indicating file names on the sever.
url_opendap: A character vector of the OPeNDAP URLs of the files.
url_download: A character vector of the download URLs of the files.
fields: A character vector indicating all response fields, followed by derived fields such as filename, url_opendap and url_download when their source fields are available.

Methods

Inherited methods

`EsgResultFile$to_data_table()`

Convert the results into a data.table

Usage

EsgResultFile$to_data_table(fields = NULL, formatted = FALSE)

Arguments

fields: A non-empty character vector indicating the fields to put into the data.table. If NULL, all fields in the query result will be used. Default: NULL.
formatted: Whether to use formatted values for special fields, including url and size. Default: FALSE.

Returns

A data.table.

`EsgResultFile$filter_time()`

Filter file records by the time range covered by each file

method = "drs" parses the time range from CMIP/DRS-style NetCDF filenames in title, URL basenames, or id. This is fast and does not open files, but it depends on the ESGF filename convention. Records whose time range cannot be parsed are kept and reported with a warning.

method = "opendap" opens each OPeNDAP URL and reads the NetCDF time axis to determine the file range. This is more exact, but much slower and requires OPeNDAP access.

The requested time filter is recorded on the returned result and is carried into EsgDataset$read_region() when a dataset is opened from the filtered result.

Usage

EsgResultFile$filter_time(start, stop, method = c("drs", "opendap", "auto"))

Arguments

start, stop: Time range boundaries. Character, Date, and POSIXt inputs are accepted and parsed in UTC.
method: How to determine file time ranges. "auto" prefers ESGF metadata and fills absent ranges from DRS filenames; "drs" always parses filenames, and "opendap" reads the remote time axis. Default: "drs".

Returns

A new EsgResultFile object.

`EsgResultFile$access_plan()`

List every compatible OPeNDAP candidate without a mandatory speed check.

Usage

EsgResultFile$access_plan(expand = TRUE)

Arguments

expand: Whether to query same-version master and Replica records.

Returns

For access_plan(), a candidate table with immutable source keys and a directory_complete attribute for the originating File query. Replica expansion does not certify that the original query's File pages were fully collected.

`EsgResultFile$fetch_subset()`

Acquire raw 2x2 climate neighborhoods with Replica switching and resume.

Usage

EsgResultFile$fetch_subset(
  requests,
  dest,
  workers = 4L,
  resume = TRUE,
  timeout = 180,
  retries = 2L,
  discover = TRUE,
  network_id = "default",
  node_workers = 2L,
  max_values = 8192L,
  replica_cooldown = 0
)

Arguments

requests: Table with request_id, site_id, lon, lat, time_start, time_stop and optional source filters and case_id labels.
dest: Local acquisition directory.
workers: Number of persistent source readers.
resume: Reuse locally verified native blocks.
timeout: Maximum seconds without saved-block progress.
retries: Number of later rounds after the first candidate round.
discover: Complete an original unsliced partial File query while known Files begin reading, and query compatible replicas after known candidates fail. Explicit selections and complete directories stay fixed.
network_id: User-defined network identity for local timing history.
node_workers: Maximum simultaneous readers for one data node.
max_values: Maximum native weather values per OPeNDAP request. Larger budgets combine only contiguous required times and corners, without changing saved native identity or existing API defaults.
replica_cooldown: Seconds to defer a failed endpoint, from 0 to 3600; 0 preserves immediate bounded retries. Explicit HTTP 404/410 failures cool for eight times this interval. Repeated failures use finite backoff up to eight times the interval. A node cools only after three distinct Files report network/service, generic DAP or native I/O failures. Evidence spans the greater of five minutes, eight cooldown intervals and three times the longest observed failed-job duration, capped at one day and computed independently per node. Known File absences stay endpoint-local; successful native data reads reset the corresponding failure streak. After corroborated cooling, only one recovery read is admitted until real native values succeed; metadata-only success does not reopen all node slots. If every candidate is cooling, the source remains incomplete for a later call.

Returns

For fetch_subset(), a lightweight acquisition summary. complete describes native coverage of the selected Files; the independent directory_complete flag is false for a partially collected File directory. An intentional slice of a fully collected directory retains true directory completeness. No missing pages are implied to have been collected by successful reads or cache reuse. A verified completed original query is saved locally and reused on repeat calls. Older saved partial results without query-selection intent remain explicit directory gaps.

`EsgResultFile$download_plan()`

Build a persistent downloader plan for file records.

Usage

EsgResultFile$download_plan(
  replica = c("auto", "current"),
  service = "HTTPServer",
  probe = TRUE,
  strategy = c("fastest", "first", "stable"),
  all = TRUE,
  node_stats = NULL,
  network_policy = NULL,
  node_policy = NULL,
  probe_concurrency = 1L,
  probe_cache_seconds = 3600L
)

Arguments

replica: Whether to use current records or expand known replicas.
service: ESGF URL service to download from. Default: "HTTPServer".
probe: Whether to lightly probe URLs before ranking them.
strategy: Candidate ranking strategy.
all: Whether replica expansion should retrieve all matching records.
node_stats: Optional data node history from Downloader$data_nodes().
network_policy: Optional network options from Downloader$network_policy.
node_policy: Optional data-node cooldown policy from Downloader$node_policy.
probe_concurrency: Maximum concurrent URL probes. Default: 1.
probe_cache_seconds: Seconds to reuse fresh data-node probe history before probing a URL again. Default: 3600.

Returns

A data.table download plan.

`EsgResultFile$expand_replicas()`

Query ESGF for same-version master and replica records.

Usage

EsgResultFile$expand_replicas(service = "HTTPServer", all = TRUE)

Arguments

service: ESGF URL service to keep in the method contract. Default: "HTTPServer".
all: Whether to retrieve all matching records. Default: TRUE.

Returns

A new EsgResultFile object with expanded replica records when instance_id, or master_id plus version, is available; otherwise self.

`EsgResultFile$repair_urls()`

Replace unreachable service URLs from compatible replicas.

$repair_urls() checks the current records for the selected service, queries compatible ESGF replicas for records whose URL is missing or unreachable, checks candidate replica URLs, and returns a new result with only that service URL repaired in the original row order. The logical record and its other service URLs are retained. The original result is not modified. The returned result records the original query URL plus the replica lookup query URL in $query_url("all").

Usage

EsgResultFile$repair_urls(
  service = c("OPENDAP", "HTTPServer"),
  index_node = NULL,
  probe = NULL
)

Arguments

service: Service URL to repair. One of "OPENDAP" or "HTTPServer". Default: "OPENDAP".
index_node: Optional ESGF search index node used to look up replicas. If NULL, the current result index node is used.
probe: Optional named list of probe settings. Supported fields are level, timeout, concurrency, network_policy, cache_seconds, and cache_failures_seconds. Default level is "data_node".

Returns

A new EsgResultFile object.

`EsgResultFile$select_replica()`

Select the preferred candidate URL per logical file.

Usage

EsgResultFile$select_replica(
  strategy = c("fastest", "first", "stable"),
  probe = TRUE,
  service = "HTTPServer",
  node_stats = NULL,
  network_policy = NULL,
  node_policy = NULL,
  probe_concurrency = 1L,
  probe_cache_seconds = 3600L
)

Arguments

strategy: Candidate ranking strategy.
probe: Whether to lightly probe URLs before ranking them.
service: ESGF URL service to download from. Default: "HTTPServer".
node_stats: Optional data node history from Downloader$data_nodes().
network_policy: Optional network options from Downloader$network_policy.
node_policy: Optional data-node cooldown policy from Downloader$node_policy.
probe_concurrency: Maximum concurrent URL probes. Default: 1.
probe_cache_seconds: Seconds to reuse fresh data-node probe history before probing a URL again. Default: 3600.

Returns

A data.table with one selected candidate per logical file.

`EsgResultFile$download()`

Enqueue and run file downloads using a Downloader.

Usage

EsgResultFile$download(
  downloader = NULL,
  store = NULL,
  replica = c("auto", "current"),
  service = "HTTPServer",
  probe = TRUE,
  strategy = c("fastest", "first", "stable"),
  probe_concurrency = NULL,
  probe_cache_seconds = 3600L,
  session_label = NULL,
  run = TRUE,
  ...
)

Arguments

downloader: Optional persistent Downloader. If NULL, store$downloader() is used when store is supplied.
store: Optional EsgStore providing a bound downloader.
replica: Whether to use current records or expand known replicas before downloading.
service: ESGF URL service to download from. Default: "HTTPServer".
probe: Whether to lightly probe URLs before ranking them.
strategy: Candidate ranking strategy.
probe_concurrency: Maximum concurrent URL probes when probe = TRUE. Default comes from the downloader worker count.
probe_cache_seconds: Seconds to reuse fresh data-node probe history before probing a URL again. Default: 3600.
session_label: Optional download session label.
run: Whether to run the queued session immediately. Default: TRUE.
...: Additional arguments passed to Downloader$run().

Returns

The created downloader session ID.

`EsgResultFile$print()`

Print a summary of the current dataset

Usage

EsgResultFile$print(n = 10L)

Arguments

n: An integer indicating how many items to print. If NULL, all items will be printed. Default: 10L.

Returns

The EsgResultFile object itself, invisibly.

`EsgResultFile$open_dataset()`

Open a file as an EsgDataset for remote data access via OPeNDAP

Usage

EsgResultFile$open_dataset(
  which = NULL,
  fallback = c("ask", "auto", "error"),
  store = NULL,
  downloader = NULL,
  progress = getOption("epwshiftr.progress", interactive())
)

Arguments

which

File records to open. Use integer indices or file IDs. If NULL, all file records are opened. Default: NULL.

fallback

What to do if OPeNDAP is unavailable. One of:

"ask": Interactively ask the user (default). In a non-interactive session this raises an error.
"auto": Automatically download the file via HTTPServer.
"error": Raise an error.

store

Optional EsgStore used for recoverable HTTP fallback.

downloader

Optional persistent Downloader used for recoverable HTTP fallback.

progress

Whether to show progress bars while opening OPeNDAP records and any HTTP fallback downloads. By default the package option epwshiftr.progress is used, falling back to interactive().

Returns

An EsgDataset object with the connection already opened.

`EsgResultFile$clone()`

The objects of this class are cloneable with this method.

Usage

EsgResultFile$clone(deep = FALSE)

Arguments

deep: Whether to make a deep clone.