Troubleshoot ESGF Workflows

This article collects the checks that are most useful when a future EPW workflow returns empty results, cannot open remote data, downloads slowly, or produces incomplete coverage.

Start with the Stage

For high-level workflows, inspect the stage first. Stage diagnostics are meant to point to the layer that needs attention.

shift_status(epws)
shift_diagnostics(epws)
shift_ids(epws)

Use shift_ids() only for deeper inspection. Ordinary scripts should rely on the stage helpers first.

Index Node vs Data Node

An index node answers search requests. A data node hosts files and service URLs. Confusing the two is a common source of empty results.

query <- esg_query("https://esgf-data.dkrz.de")

query$list_fields()
query$list_shards()
query$list_values(c("activity_id", "experiment_id", "table_id"))

shards controls distributed search indexes. data_node is a search facet that restricts where the matching files are hosted.

If a query is unexpectedly empty:

confirm the index node is a standard ESGF search node for the record type;
remove data_node temporarily to see whether files exist elsewhere;
verify activity_id, experiment_id, source_id, variant_label, frequency, table_id, and variable_id;
use ESG dictionaries to check request values before contacting ESGF.

Dataset, File, and Aggregation Records

Dataset records are parent records. File records identify NetCDF files. Aggregation records are optional and may be absent.

datasets <- shift_datasets(request)
datasets$count()

files <- datasets$collect(type = "File", fields = "*", all = TRUE, limit = NULL)
files$count()

aggregations <- datasets$collect(type = "Aggregation", fields = "*", all = TRUE, limit = NULL)
aggregations$count()

If Aggregation records are empty, continue with File records. The robust store workflow does not require Aggregation records.

Time Filtering

ESGF datetime search parameters and file coverage are not the same thing. For CMIP-style files, prefer collecting Dataset/File records and then filtering file coverage.

files_2060 <- files$filter_time(
    "2060-01-01T00:00:00Z",
    "2060-12-31T23:59:59Z",
    method = "drs"
)

Use method = "drs" for lightweight filename parsing. Use method = "opendap" only when you need to inspect remote time axes and can tolerate live network reads.

OPeNDAP and HTTPServer Access

The default extraction path is OPeNDAP-first. Full NetCDF downloads are optional unless you need offline work, repeated extraction, or fallback from unreliable remote access.

file_table <- shift_files(files_stage)$to_data_table(fields = c(
    "filename", "variable_id", "data_node", "url_opendap", "url_download"
))

file_table[, .(
    filename,
    variable_id,
    opendap = !is.na(url_opendap) & nzchar(url_opendap),
    http = !is.na(url_download) & nzchar(url_download)
)]

When OPeNDAP fails, try:

rerun extraction with fallback = "auto";
insert shift_download() before shift_extract() if repeated local reads are preferred;
check whether the selected files expose url_opendap;
inspect data-node history in Downloader.

Download Problems

Downloader state is persistent. Inspect sessions, tasks, jobs, and events before re-running a large download.

downloader <- shift_store(files_stage)$downloader()

downloader$sessions()
downloader$tasks(session_id = session_id)
downloader$events(session_id = session_id)
downloader$data_nodes(service = "HTTPServer")

For failed or interrupted tasks:

downloader$retry(session_id = session_id)
downloader$resume(session_id = session_id)
downloader$verify(session_id = session_id)

If using the CLI:

epwshiftr --store ~/cmip6-store download watch --query <query_id>
epwshiftr --store ~/cmip6-store download logs --session <session_id> --tail 100
epwshiftr --store ~/cmip6-store download retry --query <query_id> --run

Store Validation

Use store validation when local files, manifest records, or download layout look inconsistent.

store <- shift_store(epws)

store$storage_report(detail = TRUE)

validation <- store$validate_files(
    query_id = shift_ids(epws)$query_id,
    checksum = FALSE,
    layout = TRUE
)

validation$summary
validation$actions

Repair and cleanup methods are preview-first. Execute only after reviewing actions.

store$repair_files(validation$actions, dry_run = TRUE)
store$cleanup_downloads(scope = "tmp", older_than = 7 * 24 * 3600, dry_run = TRUE)

Extraction Coverage

Coverage must be complete before strict morphing can write future EPWs.

coverage <- shift_coverage(extracted)
coverage[, .(site_id, variable_id, complete, status, output_rows, output_file_count)]

If coverage is incomplete:

check that the requested variables match epw_morph_variables("recommended");
confirm each File record covers the requested morphing period;
inspect whether OPeNDAP access or fallback downloads failed;
rerun shift_extract() after fixing file selection or access;
reduce to a single variable or shorter period to isolate the failing record.

Multi-Table Selection and Snow Depth

Enhanced Belcher recipes can combine atmospheric variables from Amon with snow depth (snd, metres) from LImon. Keep both table and grid automatic for the first resolution attempt:

recipe <- epw_morph_recipe("belcher")
epw_morph_variables(recipe)
epw_morph_variables(recipe, include_optional = TRUE)

climate <- shift_cmip6(
    "BCC-CSM2-MR", "ssp585",
    table = NULL,
    grid = NULL
)
print(climate)

The plan receipt reports tables auto by variable. After resolution, progress, watch, and verbose receipts show the authoritative partitions, for example Amon=gn · LImon=gr. A single grid_label in a case table is only the primary compatibility summary; use the partition result when diagnosing multi-table selection.

Common causes of avoidable resolution failures are:

table = "Amon" forces every variable, including snd, into Amon;
grid = "gn" can reject an otherwise complete case when its LImon data exists only on gr;
a named table value lost its variable name in hand-written JSON. Use "table": {"snd": "LImon"}, not "table": "LImon", for an override;
future snd exists but the matching historical reference does not, or vice versa.

With snow_depth = "auto", the last case is informational: snow morphing is disabled and the EPW is still produced. Use snow_depth = "required" to make missing future/reference snd block resolution, or "off" to avoid querying it altogether.

required_snow <- belcher(
    reference = historical_reference(1995:2014),
    options = list(snow_depth = "required")
)

no_snow <- belcher(options = list(snow_depth = "off"))

If automatic mapping is correct but a particular catalog needs a pin, override only that variable with table = c(snd = "LImon"). Normal and verbose prints preserve the distinction between an automatic mapping, a named override, and a scalar table forced across all variables.

Morphing Diagnostics

Strict morphing blocks on missing recipe variables, missing baseline EPW fields, or incomplete monthly summaries.

morphed <- shift_morph(extracted, baseline = epw, strict = FALSE)
shift_diagnostics(morphed)

Use the lower-level EpwMorpher article when you need to inspect climate summaries, baseline summaries, factors, and morphing plan diagnostics directly.

Quick Triage Order

shift_status() and shift_diagnostics().
Query values and index/data-node selection.
Dataset count, then File count.
File coverage with filter_time(method = "drs").
OPeNDAP/HTTPServer URL availability.
Downloader sessions/tasks/events if full downloads are involved.
Store validation if local files or manifest state look wrong.
shift_coverage() before strict morphing.
For enhanced recipes, inspect the selected table/grid partitions and whether optional or required snd is available on both sides of the comparison.

Hongyuan Jia