Manage ESGF Stores from the Command Line

Hongyuan Jia

Source: vignettes/articles/cli-esgf-store.Rmd

The epwshiftr command line interface is a thin wrapper around the package’s R APIs. It is meant for day-to-day ESGF store operations and future-weather workflows: checking the local environment, creating and tracking queries, previewing updates, running downloads, validating local files, extracting site climate data, morphing baseline EPW files, and writing future EPW outputs.

The CLI mirrors the same layers documented in ESGF query results, ESG stores, Downloader, Create Future EPW Files, and ESGF troubleshooting. Workflow state is store backed. Commands exchange persistent query_id, plan_id, and morph_id values from the EsgStore manifest rather than writing separate stage files.

It is not a separate cross-platform binary. The launcher installed by install_cli() calls the current Rscript and runs epwshiftr::epwshiftr_cli(exit = TRUE). This keeps command-line behaviour close to the R package while still giving you a convenient epwshiftr command.

Install the Launcher

Install the launcher once from R:

Rscript -e 'epwshiftr::install_cli()'

On macOS and Linux, the default location is ~/.local/bin/epwshiftr. On Windows, it is %LOCALAPPDATA%/epwshiftr/bin/epwshiftr.cmd. install_cli() does not edit shell profiles or PATH; if the target directory is not on PATH, add it yourself or call the launcher by its full path.

You can also use the CLI entry point directly without installing a launcher:

Rscript -e 'epwshiftr::epwshiftr_cli(c("doctor"), exit = TRUE)'

Use doctor before longer runs. The default checks are local and read-only. Add --network only when you want to probe an ESGF index node.

epwshiftr doctor
epwshiftr doctor --network --index-node https://esgf-data.dkrz.de

Choose a Store

Most commands operate on an EsgStore. Use --store to choose the store for one command:

epwshiftr --store ~/cmip6-store query list

When --store is omitted, the CLI uses store_dir(), the same default store location used by the R API.

Run a Future EPW Workflow from JSON

Use shift run --config when you want one command to execute the main workflow: request, collect, optional download, extract, morph, and write EPW files. The configuration file is JSON and is validated before work starts. Morphing methods own their structural inputs. A matching historical reference is the recommended Belcher configuration. Omit method.reference only when no suitable reference data exist; that fallback uses the baseline EPW without creating an implicit historical query.

{
  "version": 1,
  "epw": "baseline/SIN.epw",
  "climate": {
    "provider": "cmip6",
    "model": "BCC-CSM2-MR",
    "scenarios": ["ssp126", "ssp585"],
    "member": null,
    "grid": null,
    "frequency": "mon",
    "table": null
  },
  "periods": {
    "2060s": "2055:2065"
  },
  "method": {
    "name": "belcher",
    "profile": "enhanced",
    "options": {
      "transition_hours": 72,
      "humidity_source": "auto",
      "diffuse_model": "rbl_2010",
      "illuminance_model": "perez_1990",
      "snow_depth": "auto",
      "ground_temperatures": "recalculate",
      "typical_extreme_periods": "recalculate",
      "design_conditions": "drop"
    },
    "reference": {
      "mode": "historical",
      "periods": {
        "reference": "1995:2014"
      }
    }
  },
  "dir": "outputs/future-epw",
  "control": {
    "strict": true,
    "allow_partial": false,
    "download": "auto",
    "resume": true,
    "overwrite": false
  }
}

climate.table = null enables recipe-aware table selection: atmospheric variables normally use Amon, while snd uses LImon. A scalar forces every variable into one table; a JSON object such as {"snd": "LImon"} overrides only the named variable. The method profile and resolved options are persisted with the run. Older task records that lack these fields resume as legacy rather than silently adopting enhanced defaults.

Preview the parsed request and site without writing store state:

epwshiftr --store ~/cmip6-store shift run --config shift-workflow.json --dry-run

Run the workflow. control.download owns the access policy; "auto" prefers OPeNDAP and falls back to HTTP downloads.

epwshiftr --store ~/cmip6-store shift run --config shift-workflow.json

For a detached worker, add --background. Progress presentation is a CLI concern rather than part of the JSON scientific configuration:

epwshiftr --store ~/cmip6-store shift run \
  --config shift-workflow.json \
  --background

The result includes one durable run ID. All workflow inspection and recovery commands use that ID, so they do not need users to coordinate query, extraction, and morph identifiers manually:

epwshiftr --store ~/cmip6-store shift status --run <run_id>
epwshiftr --store ~/cmip6-store shift show --run <run_id>
epwshiftr --store ~/cmip6-store shift watch --run <run_id> --follow
epwshiftr --store ~/cmip6-store shift diagnostics --run <run_id>
epwshiftr --store ~/cmip6-store shift outputs --run <run_id>
epwshiftr --store ~/cmip6-store shift data --run <run_id> --limit 5
epwshiftr --store ~/cmip6-store shift logs --run <run_id> --tail 100
epwshiftr --store ~/cmip6-store shift cancel --run <run_id>
epwshiftr --store ~/cmip6-store shift resume --run <run_id> --background

Ctrl-C during shift watch --follow stops watching without cancelling the worker. A normal shift cancel stops at the next safe workflow boundary. Use shift cancel --force only when the worker does not respond to that request.

You can generate and validate workflow JSON from the CLI:

epwshiftr shift config example --output shift-workflow.json
epwshiftr shift config validate --config shift-workflow.json

Run Extraction and Morphing in Stages

Use extract commands when a query already exists in the store and you want to plan, run, and inspect the climate extraction stage directly.

epwshiftr --store ~/cmip6-store extract plan \
  --query <query_id> \
  --site-id SIN \
  --lon 103.98 \
  --lat 1.37 \
  --time 2055-01-01T00:00:00Z,2064-12-31T23:59:59Z \
  --variable tas,hurs,psl,rsds,rlds,sfcWind,clt \
  --nearest 1

epwshiftr --store ~/cmip6-store extract run --plan <plan_id> --fallback auto
epwshiftr --store ~/cmip6-store extract retry --plan <plan_id>
epwshiftr --store ~/cmip6-store extract retry --plan <plan_id> --run
epwshiftr --store ~/cmip6-store extract coverage --plan <plan_id>
epwshiftr --store ~/cmip6-store extract artifacts --plan <plan_id>

Use morph commands to inspect available recipes/backends, run morphing from extracted plan_id values, and write EPW outputs later from the stored morph_id.

epwshiftr --store ~/cmip6-store morph variables --recipe belcher
epwshiftr --store ~/cmip6-store morph backends

epwshiftr --store ~/cmip6-store morph run \
  --plan <plan_id> \
  --epw baseline/SIN.epw \
  --period 2060s=2055:2064 \
  --recipe belcher \
  --profile enhanced \
  --option transition_hours=72 \
  --option snow_depth=auto \
  --reference historical \
  --reference-period reference=1995 \
  --strict true

epwshiftr --store ~/cmip6-store morph epw --morph <morph_id> --dir outputs/future-epw
epwshiftr --store ~/cmip6-store morph retry --morph <morph_id>
epwshiftr --store ~/cmip6-store morph retry --morph <morph_id> --run
epwshiftr --store ~/cmip6-store morph status --morph <morph_id>
epwshiftr --store ~/cmip6-store morph outputs --morph <morph_id>

Repeat --option KEY=VALUE for each override. Omitted values come from the selected profile; --profile legacy restores the pre-enhancement calculation and EPW-header policies.

Search Without Saving

query search builds an EsgQuery from key=value constraints. The command contacts ESGF unless you pass --dry-run.

epwshiftr query search --dry-run \
  --index-node https://esgf-data.dkrz.de \
  --type Dataset \
  project=CMIP6 \
  activity_id=ScenarioMIP \
  experiment_id=ssp585 \
  variable_id=tas,psl \
  table_id=Amon \
  datetime_start=2050-01-01 \
  datetime_stop=2050-12-31 \
  latest=true

Comma-separated values become multi-value query parameters. Boolean strings true and false become logical query values.

Use --fields to control fields requested from ESGF, and --columns to control the human-readable table printed by the CLI. If a display column is an ESGF result field, include it in --fields as well.

epwshiftr query search \
  --fields source_id,experiment_id,variable_id,datetime_start,datetime_stop,size \
  --columns variable_id,datetime_start,datetime_stop,size \
  project=CMIP6 \
  variable_id=tas \
  datetime_start=2050-01-01 \
  datetime_stop=2050-12-31

Use datetime_start and datetime_stop for data coverage constraints. These are mapped to EsgQuery$datetime_range(), so they are not plain ESGF facets. For a target year, set both boundaries:

epwshiftr query search --dry-run \
  project=CMIP6 \
  variable_id=tas \
  datetime_start=2050-01-01 \
  datetime_stop=2050-12-31

datetime_end is accepted as an alias for datetime_stop.

Use key!=value for a negated facet constraint:

epwshiftr query search --dry-run \
  project=CMIP6 \
  source_id!=BCC-CSM2-MR \
  variable_id=tas

A single key cannot mix positive and negated constraints in the same command. Use either source_id=A,B or source_id!=A,B, not both.

Save a Tracked Query

Use query add when a search should become part of a persistent store. This saves the query definition in the store; it does not download files.

epwshiftr --store ~/cmip6-store query add \
  --label singapore-ssp585-amon \
  --tag cmip6 \
  --tag future-weather \
  --track \
  --index-node https://esgf-data.dkrz.de \
  project=CMIP6 \
  activity_id=ScenarioMIP \
  experiment_id=ssp585 \
  variable_id=tas,psl \
  table_id=Amon \
  datetime_start=2050-01-01 \
  datetime_stop=2050-12-31 \
  latest=true

Use --dry-run to inspect the query URL, label, tags, and tracked flag before writing to the store.

epwshiftr --store ~/cmip6-store query add --dry-run \
  --label singapore-ssp585-amon \
  project=CMIP6 variable_id=tas table_id=Amon

You can also import a query JSON file saved from R:

epwshiftr --store ~/cmip6-store query add \
  --query-file query.json \
  --label imported-query \
  --track

Preview and Record Query Updates

Tracked queries support a two-step update path. First preview the diff without changing the store:

epwshiftr --store ~/cmip6-store query preview <query_id>
epwshiftr --store ~/cmip6-store query preview <query_id> --detail

Then persist the update when the diff is acceptable:

epwshiftr --store ~/cmip6-store query update <query_id>

Inspect query metadata, files, update batches, and non-current changes:

epwshiftr --store ~/cmip6-store query show <query_id> --files --updates --changes
epwshiftr --store ~/cmip6-store query files <query_id>
epwshiftr --store ~/cmip6-store query changes <query_id> --latest

Plan and Run Downloads

download preflight refreshes query results, computes the update diff, expands replica candidates, and estimates the download set without enqueueing tasks or downloading data.

epwshiftr --store ~/cmip6-store download preflight <query_id> \
  --replica auto \
  --service HTTPServer \
  --strategy fastest

For a conservative dry check, disable network probing:

epwshiftr --store ~/cmip6-store download preflight <query_id> --no-probe

download run performs the full store-aware path: refresh, plan, enqueue, run, and sync completed files back into the store catalog.

epwshiftr --store ~/cmip6-store download run <query_id> \
  --session-label singapore-ssp585-amon \
  --replica auto \
  --service HTTPServer

For long downloads, run in the background. --mode process starts a detached R process for this job. --mode daemon submits the job to a running downloader daemon.

epwshiftr --store ~/cmip6-store download run <query_id> \
  --session-label singapore-ssp585-amon \
  --background \
  --mode process

The downloader keeps persistent sessions, tasks, candidate URLs, data-node statistics, jobs, logs, and events. You can stop R and later inspect or resume the same session.

epwshiftr --store ~/cmip6-store download sessions
epwshiftr --store ~/cmip6-store download tasks --session <session_id>
epwshiftr --store ~/cmip6-store download watch --query <query_id>
epwshiftr --store ~/cmip6-store download logs --session <session_id> --tail 50
epwshiftr --store ~/cmip6-store download jobs
epwshiftr --store ~/cmip6-store download logs --job <job_id> --tail 50
epwshiftr --store ~/cmip6-store download stop --job <job_id>

Use daemon mode when you want one persistent process to run queued jobs over time:

epwshiftr --store ~/cmip6-store download daemon start
epwshiftr --store ~/cmip6-store download run <query_id> --background --mode daemon
epwshiftr --store ~/cmip6-store download daemon status
epwshiftr --store ~/cmip6-store download daemon stop

Retry failed or cancelled tasks:

epwshiftr --store ~/cmip6-store download retry --query <query_id>
epwshiftr --store ~/cmip6-store download retry --query <query_id> --run

Resume interrupted tasks or verify completed files:

epwshiftr --store ~/cmip6-store download resume --session <session_id>
epwshiftr --store ~/cmip6-store download verify --session <session_id>

Tune Download Behaviour

Show the current persistent downloader configuration:

epwshiftr --store ~/cmip6-store download config show

Set operational limits when running large batches:

epwshiftr --store ~/cmip6-store download config set \
  --workers 4 \
  --host-concurrency 2 \
  --timeout 120 \
  --chunk-size 1048576 \
  --bandwidth-limit none

Inspect data-node performance records and reset stale node statistics if needed:

epwshiftr --store ~/cmip6-store download nodes
epwshiftr --store ~/cmip6-store download reset-nodes --node esgf.ceda.ac.uk
epwshiftr --store ~/cmip6-store download reset-nodes --node esgf.ceda.ac.uk --execute

Maintain Local Files

The storage commands expose the store’s file-system maintenance tools.

Choose a download layout policy:

epwshiftr --store ~/cmip6-store storage layout show
epwshiftr --store ~/cmip6-store storage layout set --layout drs

Summarise disk usage and validate cataloged files:

epwshiftr --store ~/cmip6-store storage report
epwshiftr --store ~/cmip6-store storage report --detail
epwshiftr --store ~/cmip6-store storage validate --query <query_id>

Checksum validation can be expensive on large NetCDF collections, so it is opt-in:

epwshiftr --store ~/cmip6-store storage validate --query <query_id> --checksum

Repairs and cleanup commands default to preview mode. Add --execute only after reviewing the proposed actions.

epwshiftr --store ~/cmip6-store storage repair
epwshiftr --store ~/cmip6-store storage repair --execute

epwshiftr --store ~/cmip6-store storage cleanup --scope tmp --older-than 604800
epwshiftr --store ~/cmip6-store storage cleanup --scope tmp --older-than 604800 --execute

Use JSON and JSONL for Automation

Human-readable output is the default. Add --json when another script should consume a complete result. Add --jsonl with download watch --follow when a script should consume streaming progress events one object per line.

epwshiftr --store ~/cmip6-store --json query status <query_id>
epwshiftr --store ~/cmip6-store --json download watch --query <query_id>
epwshiftr --store ~/cmip6-store --json storage validate --query <query_id>
epwshiftr --store ~/cmip6-store --jsonl download watch --query <query_id> --follow

Use --quiet when you only need the process status. CLI status codes are:

  • 0: success;
  • 1: runtime error;
  • 2: usage error.

Command Groups

Use built-in help for the complete command list:

epwshiftr help
epwshiftr help query add
epwshiftr help download run
epwshiftr help shift run
epwshiftr help extract plan
epwshiftr help morph run
epwshiftr help storage validate

The command groups intentionally mirror the package boundaries:

  • doctor: local and optional network diagnostics;
  • query: create, track, update, inspect, tag, and remove store queries;
  • download: preflight, run, resume, retry, verify, and inspect downloads;
  • shift: config-driven request, collect, extract, morph, and EPW workflow runs;
  • extract: store-ID extraction planning, execution, coverage, and artifacts;
  • morph: morphing variables, backends, runs, EPW writing, status, and outputs;
  • storage: layout, report, validate, repair, and cleanup local files;
  • esgf: compact ESGF query and download health reports.