ESGF Query results for File type

EsgResultFile is a class that represents query results for File type from ESGF search RESTful API.

In general, there is no need to create an EsgResultDataset manually. Usually, it is created by calling EsgResultDataset$collect().

Author

Hongyuan Jia

Super class

epwshiftr::EsgResult -> EsgResultFile

Active bindings

filename

A character vector indicating file names on the sever.

url_opendap

A character vector of the OPeNDAP URLs of the files.

url_download

A character vector of the download URLs of the files.

fields

A character vector indicating all response fields, followed by derived fields such as filename, url_opendap and url_download when their source fields are available.

Methods

Public methods

• EsgResultFile$to_data_table()

• EsgResultFile$filter_time()

• EsgResultFile$download_plan()

• EsgResultFile$expand_replicas()

• EsgResultFile$repair_urls()

• EsgResultFile$select_replica()

• EsgResultFile$download()

• EsgResultFile$print()

• EsgResultFile$open_dataset()

• EsgResultFile$clone()

Inherited methods

• epwshiftr::EsgResult$count()
• epwshiftr::EsgResult$filter()
• epwshiftr::EsgResult$initialize()
• epwshiftr::EsgResult$load()
• epwshiftr::EsgResult$query_url()
• epwshiftr::EsgResult$reachable()
• epwshiftr::EsgResult$save()
• epwshiftr::EsgResult$selection()
• epwshiftr::EsgResult$slice()
• epwshiftr::EsgResult$to_dt()

---

Method to_data_table()

Convert the results into a data.table

Usage

EsgResultFile$to_data_table(fields = NULL, formatted = FALSE)

Arguments

fields

A non-empty character vector indicating the fields to put into the data.table. If NULL, all fields in the query result will be used. Default: NULL.

formatted

Whether to use formatted values for special fields, including url and size. Default: FALSE.

Returns

A data.table.

---

Method filter_time()

Filter file records by the time range covered by each file

method = "drs" parses the time range from CMIP/DRS-style NetCDF filenames in title, URL basenames, or id. This is fast and does not open files, but it depends on the ESGF filename convention. Records whose time range cannot be parsed are kept and reported with a warning.

method = "opendap" opens each OPeNDAP URL and reads the NetCDF time axis to determine the file range. This is more exact, but much slower and requires OPeNDAP access.

The requested time filter is recorded on the returned result and is carried into EsgDataset$read_region() when a dataset is opened from the filtered result.

Usage

EsgResultFile$filter_time(start, stop, method = c("drs", "opendap"))

Arguments

start, stop

Time range boundaries. Character, Date, and POSIXt inputs are accepted and parsed in UTC.

method

How to determine file time ranges. One of "drs" or "opendap". Default: "drs".

Returns

A new EsgResultFile object.

---

Method download_plan()

Build a persistent downloader plan for file records.

Usage

EsgResultFile$download_plan(
  replica = c("auto", "current"),
  service = "HTTPServer",
  probe = TRUE,
  strategy = c("fastest", "first", "stable"),
  all = TRUE,
  node_stats = NULL,
  network_policy = NULL,
  node_policy = NULL,
  probe_concurrency = 1L,
  probe_cache_seconds = 3600L
)

Arguments

replica

Whether to use current records or expand known replicas.

service

ESGF URL service to download from. Default: "HTTPServer".

probe

Whether to lightly probe URLs before ranking them.

strategy

Candidate ranking strategy.

all

Whether replica expansion should retrieve all matching records.

node_stats

Optional data node history from Downloader$data_nodes().

network_policy

Optional network options from Downloader$network_policy.

node_policy

Optional data-node cooldown policy from Downloader$node_policy.

probe_concurrency

Maximum concurrent URL probes. Default: 1.

probe_cache_seconds

Seconds to reuse fresh data-node probe history before probing a URL again. Default: 3600.

Returns

A data.table download plan.

---

Method expand_replicas()

Query ESGF for same-version master and replica records.

Usage

EsgResultFile$expand_replicas(service = "HTTPServer", all = TRUE)

Arguments

service

ESGF URL service to keep in the method contract. Default: "HTTPServer".

all

Whether to retrieve all matching records. Default: TRUE.

Returns

A new EsgResultFile object with expanded replica records when instance_id, or master_id plus version, is available; otherwise self.

---

Method repair_urls()

Replace unreachable service URLs with reachable replica records.

$repair_urls() probes the current records for the selected service, queries ESGF replicas by master_id for records whose URL is missing or unreachable, probes candidate replica URLs, and returns a new result with repaired records in the original row order. The original result is not modified. The returned result records the original query URL plus the replica lookup query URL in $query_url("all").

Usage

EsgResultFile$repair_urls(
  service = c("OPENDAP", "HTTPServer"),
  index_node = NULL,
  probe = NULL
)

Arguments

service

Service URL to repair. One of "OPENDAP" or "HTTPServer". Default: "OPENDAP".

index_node

Optional ESGF search index node used to look up replicas. If NULL, the current result index node is used.

probe

Optional named list of probe settings. Supported fields are level, timeout, concurrency, network_policy, cache_seconds, and cache_failures_seconds. Default level is "data_node".

Returns

A new EsgResultFile object.

---

Method select_replica()

Select the preferred candidate URL per logical file.

Usage

EsgResultFile$select_replica(
  strategy = c("fastest", "first", "stable"),
  probe = TRUE,
  service = "HTTPServer",
  node_stats = NULL,
  network_policy = NULL,
  node_policy = NULL,
  probe_concurrency = 1L,
  probe_cache_seconds = 3600L
)

Arguments

strategy

Candidate ranking strategy.

probe

Whether to lightly probe URLs before ranking them.

service

ESGF URL service to download from. Default: "HTTPServer".

node_stats

Optional data node history from Downloader$data_nodes().

network_policy

Optional network options from Downloader$network_policy.

node_policy

Optional data-node cooldown policy from Downloader$node_policy.

probe_concurrency

Maximum concurrent URL probes. Default: 1.

probe_cache_seconds

Seconds to reuse fresh data-node probe history before probing a URL again. Default: 3600.

Returns

A data.table with one selected candidate per logical file.

---

Method download()

Enqueue and run file downloads using a Downloader.

Usage

EsgResultFile$download(
  downloader = NULL,
  store = NULL,
  replica = c("auto", "current"),
  service = "HTTPServer",
  probe = TRUE,
  strategy = c("fastest", "first", "stable"),
  probe_concurrency = NULL,
  probe_cache_seconds = 3600L,
  session_label = NULL,
  run = TRUE,
  ...
)

Arguments

downloader

Optional persistent Downloader. If NULL, store$downloader() is used when store is supplied.

store

Optional EsgStore providing a bound downloader.

replica

Whether to use current records or expand known replicas before downloading.

service

ESGF URL service to download from. Default: "HTTPServer".

probe

Whether to lightly probe URLs before ranking them.

strategy

Candidate ranking strategy.

probe_concurrency

Maximum concurrent URL probes when probe = TRUE. Default comes from the downloader worker count.

probe_cache_seconds

Seconds to reuse fresh data-node probe history before probing a URL again. Default: 3600.

session_label

Optional download session label.

run

Whether to run the queued session immediately. Default: TRUE.

...

Additional arguments passed to Downloader$run().

Returns

The created downloader session ID.

---

Method print()

Print a summary of the current dataset

Usage

EsgResultFile$print(n = 10L)

Arguments

n

An integer indicating how many items to print. If NULL, all items will be printed. Default: 10L.

Returns

The EsgResultFile object itself, invisibly.

---

Method open_dataset()

Open a file as an EsgDataset for remote data access via OPeNDAP

Usage

EsgResultFile$open_dataset(
  which = NULL,
  fallback = c("ask", "auto", "error"),
  store = NULL,
  downloader = NULL,
  progress = getOption("epwshiftr.progress", interactive())
)

Arguments

which

File records to open. Use integer indices or file IDs. If NULL, all file records are opened. Default: NULL.

fallback

What to do if OPeNDAP is unavailable. One of:

• "ask": Interactively ask the user (default). In a non-interactive session this raises an error.

• "auto": Automatically download the file via HTTPServer.

• "error": Raise an error.

store

Optional EsgStore used for recoverable HTTP fallback.

downloader

Optional persistent Downloader used for recoverable HTTP fallback.

progress

Whether to show progress bars while opening OPeNDAP records and any HTTP fallback downloads. By default the package option epwshiftr.progress is used, falling back to interactive().

Returns

An EsgDataset object with the connection already opened.

---

Method clone()

The objects of this class are cloneable with this method.

Usage

EsgResultFile$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.