Acquire Raw OPeNDAP Neighborhoods

Use EsgResultFile$fetch_subset() when you need the original values around a location for your own interpolation or analysis. On rectilinear latitude and longitude grids, it reads the enclosing 2x2 neighborhood for the requested variable and native time window through OPeNDAP. The four corners remain separate; the acquisition does not interpolate them or download the complete NetCDF file.

The examples below contact external ESGF services when you run them. They are shown without execution here, so no live results or connection speeds are implied. Start with one machine and keep its acquisition directory for resume.

Describe the Required Data

Each request identifies a location, variable and time window. request_id identifies the demand; the optional case_id records which analysis uses it. Use distinct request IDs when several cases share the same physical data.

library(epwshiftr)

requests <- data.frame(
    request_id = "singapore-tas-june-2041",
    case_id = "example-case",
    site_id = "Singapore",
    lon = 103.98,
    lat = 1.37,
    variable_id = "tas",
    time_start = "2041-06-01",
    time_stop = "2041-06-30T23:59:59"
)
dest <- "climate-neighborhoods"

Endpoints are inclusive. A date without a time denotes midnight; include a time to request the whole final day. Dates must be meaningful in the source calendar. Native CF coordinates are authoritative, including for 360_day and noleap data; do not infer their calendar dates from a POSIX time proxy.

Collect the Complete File Directory

Build an explicit query for the model, experiment, member, variable and frequency you need. Collect every matching File page with all = TRUE before narrowing the result to the requested interval. This searches metadata; it does not read the climate variable.

query <- esg_query("https://esgf-data.dkrz.de")$activity_id(
    "ScenarioMIP"
)$source_id("MPI-ESM1-2-LR")$experiment_id("ssp126")$variant_label(
    "r1i1p1f1"
)$variable_id("tas")$frequency("mon")$params(table_id = "Amon")

files <- query$collect(type = "File", all = TRUE, limit = FALSE)
files <- files$filter_time(
    requests$time_start,
    requests$time_stop,
    method = "drs"
)
candidates <- files$access_plan()
attr(candidates, "directory_complete")

access_plan() lists compatible OPeNDAP Replica candidates. Successful data reads cannot establish that missing query pages were collected. The separate directory_complete flag records that distinction; filtering or deliberately slicing an already complete directory retains its directory evidence.

Acquire the Four Corners

result <- files$fetch_subset(
    requests,
    dest,
    workers = 4L,
    node_workers = 2L,
    timeout = 180,
    retries = 2L,
    network_id = "single-machine"
)
result$directory_complete
result$complete
subset_status(dest, "nodes")

The package switches to another compatible Replica after a read failure and updates node ranking from actual demand reads. It does not require a separate speed test of every candidate. A file whose candidates fail releases its worker so other files can continue; retries bounds later attempts. Node observations belong to the current client and network_id, so another machine can have a different ranking.

result$complete concerns native coverage of the selected Files. result$directory_complete concerns File-query pagination. Neither establishes that every variable, frequency, model or year required by your study is present. Check those study requirements separately. A case label is an association, not a scientific acceptance rule.

Tune Transport Without Changing the Requested Data

max_values limits native weather values in each request. Its default remains 8192, including all four corners. For a service that benefits from fewer, larger requests, try 65536 in a separate acquisition directory before changing an existing run. Only consecutive required native time indices are combined; gaps, physical File boundaries and the four unaggregated corners remain intact. A different budget can resume previously saved blocks without reading them again.

result <- files$fetch_subset(
    requests,
    "climate-neighborhoods-timing",
    workers = 4L,
    node_workers = 4L,
    max_values = 65536L,
    replica_cooldown = 60,
    network_id = "single-machine"
)

node_workers already controls simultaneous File readers on each data node. Compare useful native values per second and failures before increasing it; a larger worker count does not create more node slots. When several machines use one service, consider their combined allowance.

Replica cooling is optional and defaults to zero, retaining immediate finite retries. With replica_cooldown = 60, recent transport failures defer that exact File URL for 60 seconds on the current client/network identity. Repeated failures increase that interval up to eightfold; known File absences, including explicit HTTP 404 or 410, use the eightfold interval immediately and remain File-local. Generic NetCDF: DAP failure errors alone do not establish a node fault. Node cooling requires at least three distinct Files with network/service, generic DAP or native I/O failures. Each node retains evidence across the greater of five minutes, eight cooldown intervals and three times its longest recorded failed-job duration, capped at the existing one-day history horizon. Invalid or missing durations do not extend this window. A slow failure on one node does not extend another node’s evidence. Its recovery spacing also increases up to eightfold with further failures. A successful native data read resets that node’s failure streak; a successful read of the same File URL also resets its endpoint backoff. Content and CF identity checks continue to apply to every Replica.

After a corroborated node cooldown, the first recovery uses one reader until actual native values succeed. Metadata-only success does not reopen every node slot. The configured node allowance returns after a native success.

Cooling never occupies a sleeping worker. Other Files and compatible candidates continue; Replica discovery remains available when known URLs are cooling. If all compatible candidates are cooling, the source stays incomplete with a saved gap. A later call retries after the deadline and reuses verified blocks. The existing attempt ledger supplies these deadlines across calls; no extra speed checks or separate health database are required.

Inspect Saved Values and Gaps

Native selection evidence is saved before the first weather block, so a later read failure can retain the exact native denominator and four-corner membership. After the acquisition writer closes, inspect checked physical coverage:

coverage <- subset_status(dest, "coverage")
timing <- subset_status(dest, "timing", source_key = unique(coverage$source_key))

complete requires every requested native time at all four original corners. The coverage view preserves request and case associations; deduplicate physical File/location/window keys for totals. Unknown axes retain NA expected counts, and an empty native selection stays incomplete. status, gap_kind, retryable and retry_after distinguish partial data, nonterminal work, transport failures, cooling and nonretryable identity/selection problems. Job labels alone do not certify raw coverage. source_key can restrict these two views before reading or hashing unrelated artifacts. These views read local data only and require the unique acquisition writer to be closed.

Timing reports recorded open/metadata phases and successful native-read phases separately from whole failed tasks. A failed task may already have committed useful blocks, so its full duration is not pure waste or an additional disjoint phase. Historical phases without evidence remain NA. Acquisition coverage also does not certify scientific quality of native missing values.

A caller can use the structured gaps for a finite recovery campaign after its first pass: retain the original identities, retry only eligible incomplete windows, respect cooldown deadlines, and stop on a fixed budget, no progress or a nonretryable blocker. Expiration alone does not launch a later acquisition. Persist the spent budget across restarts and report unresolved gaps explicitly.

After an acquisition call returns, inspect its local results:

subset_status(dest)
subset_status(dest, "attempts")
raw <- read_subset(dest, case_id = "example-case")
head(raw$values)
head(raw$neighborhoods)
raw$gaps$directory
raw$gaps$unmatched
raw$gaps$incomplete_requests
raw$gaps$invalid_blocks

raw$values contains unique native point/time values with their coordinates and CF time information. raw$neighborhoods relates them to each location’s corners and requested window; raw$links retains the case/request associations. Keep these together when comparing interpolation algorithms.

subset_status() and read_subset() use local data only. Directory gaps, unmatched requests, missing native coverage and invalid saved blocks remain explicit; the reader does not fill them through hidden network access.

Resume in the Same Directory

Repeat the acquisition with the same requests and destination:

resumed <- files$fetch_subset(
    requests,
    dest,
    resume = TRUE,
    workers = 4L,
    network_id = "single-machine"
)
raw <- read_subset(dest, case_id = "example-case")
raw$gaps$incomplete_requests

Verified saved blocks are reused and only outstanding demand is read. A fully saved request does not need another climate-data read. With the default discover = TRUE, an original, unsliced partial File query automatically collects its missing pages and saves the completed directory for local reuse on repeat calls. An explicitly sliced result is never expanded automatically, even when it contains every held row. Existing directory gaps remain visible when collection context is unknown or discovery is disabled. To resolve a partial original directory, explicitly collect the original File query with all = TRUE and supply that complete result to fetch_subset().

Keep one writer per acquisition directory. If work is later divided between machines, give each machine a fixed File list and its own destination, and retain the request/case associations when combining results.