vignettes/articles/future-epw-workflow.Rmd
future-epw-workflow.Rmd
library(epwshiftr)
if (!exists("shift_request", mode = "function")) {
stop(
"This raw article must be rendered from the package source with ",
"`Rscript tools/raw-vignettes.R render`.",
call. = FALSE
)
}This article is the recommended main workflow for epwshiftr. It runs
a real store-native shift workflow from ESGF File records to generated
EPW files. It uses monthly Amon data so the live remote
reads are much smaller than an equivalent daily workflow.
The first example uses the task-oriented API. The staged walkthrough
that follows exposes the lower-level shift_*() facade for
inspection and teaching. When a step touches a lower-level engine, this
article links to the companion article for that layer: ESGF query results, ESG dictionaries, ESG stores, Downloader, EpwMorpher, CLI operations, and ESGF troubleshooting.
Use a temporary store for this article run. For a real project, replace this with a persistent project cache path.
workflow_root <- file.path(tempdir(), "epwshiftr-future-epw-workflow")
if (dir.exists(workflow_root)) {
unlink(workflow_root, recursive = TRUE)
}
dir.create(workflow_root, recursive = TRUE, showWarnings = FALSE)
options(
epwshiftr.dir_cache = file.path(workflow_root, "cache")
)The shortest workflow names one or more registered weather methods directly. The package discovers CMIP6 models that can satisfy every selected method, uses the same model/member/grid identities for each method, obtains matching historical model periods where required, and routes observational calibration only to methods that declare that role.
The following call uses the San Francisco weather file shipped with
the latest locally installed EnergyPlus version. It requests SSP1-2.6
and SSP5-8.5, selects three complete r1i1p1f1 GCMs, and
produces 2050 and 2080 weather files with original morphing, BWS+BTWS,
and ISIMIP3BASD.
energyplus_version <- tail(eplusr::avail_eplus(), 1L)
san_francisco_epw <- eplusr::path_eplus_weather(
energyplus_version,
"USA_CA_San.Francisco.Intl.AP.724940_TMY3.epw",
strict = TRUE
)
run <- shift_future_epw(
epw = san_francisco_epw,
climate = shift_cmip6(
model = 3L,
scenarios = c("ssp126", "ssp585"),
member = "r1i1p1f1"
),
periods = list(`2050` = 2041:2060, `2080` = 2071:2090),
methods = c("original_morphing", "bws_btws", "isimip3basd"),
calibration = shift_era5(years = 1995:2014),
dir = file.path(workflow_root, "future-epw")
)
shift_status(run)
shift_outputs(run)
shift_diagnostics(run)bws_btws is the complete Eames method: BTWS reconstructs
hourly temperature, while BWS transforms bounded global solar radiation
and total sky cover. ERA5 is used only by ISIMIP3BASD in this selection
because that method declares an observed_reference role.
Each method/model child remains independently resumable. Output and
diagnostic tables include method, model, member, and grid columns for
downstream analysis. shift_summary() groups output counts
and optional local weather statistics by scientific identity; it does
not rank methods.
For method/model batches, save shift_ids(run)$batch_id
to reopen the result in a later R session using the same store:
batch <- shift_batch_get(batch_id, store = store_dir())
shift_watch(batch)
shift_outputs(batch)
shift_diagnostics(batch)
shift_history(store = store_dir(), type = "batch")
shift_summary(batch, weather = TRUE)shift_resume(batch) continues unfinished children and
reuses verified completed outputs. Case counts describe scientific
combinations of site, scenario, and period within each child; physical
EPW file counts can be larger when a method produces several weather
years. The batch receipt shows method, reconstruction, model, maturity
status, warnings, and reuse for each call. It shares the single-workflow
box style, separating the batch overview, Workflows, and
Results with labelled rules. Narrow terminals preserve the
same information without borders.
Dynamic batch views fit their terminal height and show failed and
active children before completed work.
shift_watch(batch, follow = FALSE, ui = shift_ui(detail = "detail"))
prints every child, live activity, update time, diagnostic, and recovery
action. Errors appear before warnings, cancelled children have a
separate count, and a completed watch leaves its final receipt visible
even when its return value is assigned.
shift_history() lists root runs, batch parents, and
child runs with exact store paths. Filter by type = "run"
or "batch" and
status = c("failed", "partial"). This inspection is local
and does not create a store; unreadable records remain visible with
status unavailable and an explanatory error.
shift_summary(batch) separates completed cases, recorded
EPWs, and available files for each method/model/scenario/period and
member/grid. Add weather = TRUE to read existing local EPWs
and report temperature, humidity, wind speed, and global horizontal
radiation means. Each mean is weighted by valid hourly rows, excludes
EPW missing codes, and reports its own valid-hour count. Unreadable
files are counted with their errors. Compare output types, years, and
valid-hour coverage before comparing the means; these summaries do not
establish scientific equivalence between a representative year and a
multi-year sequence.
For a standalone collect/extract/morph/EPW workflow, pass its saved
ShiftRun to shift_summary(). The summary
recovers the method from the result’s morphing stage and groups the
output manifest by scientific identity. Case counts cover distinct cases
with saved outputs, so multiple weather-year files for one case do not
inflate the case count.
With access = "auto", shift_era5() uses the
official point time-series service when all required variables are
available there and otherwise uses the complete CDS subset service. CDS
access reads a personal access token from
ECMWF_DATASTORES_KEY, ~/.ecmwfdatastoresrc, or
~/.cdsapirc; credentials are never stored in a shift plan.
shift_era6() reserves the same provider-neutral boundary
but currently raises an explicit unavailable error, because no stable
public ERA6 dataset endpoint exists yet. It never substitutes ERA5 under
an ERA6 label.
The source object deliberately contains no credentials, and creating it does not contact CDS. Before the first retrieval:
ECMWF_DATASTORES_KEY,
~/.ecmwfdatastoresrc, or ~/.cdsapirc.The local check validates that configuration without making a network request. The explicit network check verifies the token through the CDS account endpoint without submitting or downloading a climate dataset:
era5 <- shift_era5(years = 1995:2014)
shift_check(era5)
shift_check(era5, network = TRUE)An empty diagnostic table means the requested check passed. A successful token check cannot guarantee access to every dataset because CDS terms are dataset-specific. The first retrieval remains the authoritative check and, if terms are missing, returns a dedicated error with the exact dataset page. The package never accepts terms on the user’s behalf.
Maintainers can exercise the complete remote request, download, NetCDF normalization, persistence, and reuse workflow after configuring a token:
EPWSHIFTR_RUN_LIVE_ERA5=true Rscript -e \
'devtools::test(filter = "era5-live", stop_on_failure = TRUE)'The repository runs the same test in its separate manual and weekly
workflow. It requires the ECMWF_DATASTORES_KEY repository
secret and is intentionally excluded from ordinary pull-request checks,
where secrets are unavailable.
Use an explicit transform when method-specific options or one exact model are needed. The following form selects original morphing as a reusable monthly transformation and gives the run an explicit matching historical CMIP6 reference. A transform contains calculation settings only; the site, model, periods, reference, and output directory remain execution inputs.
epw <- system.file(
"extdata/examples/SGP_Singapore.486980_IWEC.epw",
package = "epwshiftr",
mustWork = TRUE
)
run <- shift_future_epw(
epw = epw,
climate = shift_cmip6(
model = "BCC-CSM2-MR",
scenarios = c("ssp126", "ssp585")
),
periods = list(`2060s` = 2055:2065),
transform = monthly_transform("original_morphing"),
reference = historical_reference(1995:2014),
dir = file.path(workflow_root, "future-epw")
)
shift_status(run)
shift_outputs(run)
shift_missing(run)Use weather_transforms() to inspect the available
methods, their source frequencies, required input roles, and hourly
reconstruction. Original morphing, introduced by Belcher et al., is one
canonical published method. The enhanced package method is selected
separately with monthly_transform("epwshiftr"); neither
method exposes backend profiles or physical-policy switches.
Scientific options belong to the transform rather than the UI.
Configure them on the package method and keep table = NULL
unless a dataset requires an explicit override:
transform <- monthly_transform(
"epwshiftr",
transition_hours = 72L,
humidity_source = "auto",
snow_depth = "auto"
)
# Named values override individual variables; unnamed scalars force every
# variable into one table and are rarely suitable when snd is enabled.
climate <- shift_cmip6(
"BCC-CSM2-MR", "ssp585",
table = c(snd = "LImon")
)See Inspect EPW Morphing for the complete transform and execution contracts.
Use the same call with dry_run = TRUE to obtain a
ShiftPlan, inspect it with shift_explain(),
and execute it later with shift_run(). The persisted run ID
supports shift_status(), shift_resume(), and
the corresponding CLI commands.
Every public Shift object prints as a compact semantic receipt.
Configuration objects summarize scientific intent without expanding
stored rules, function environments, yearly vectors, or every ESGF node.
Stage objects add a preview of the records that matter at that point in
the workflow. ShiftRun refreshes its persisted state and
prints the same static dashboard used by shift_watch().
climate <- shift_cmip6(
model = "BCC-CSM2-MR",
scenarios = c("ssp126", "ssp585")
)
transform <- monthly_transform("original_morphing")
plan <- shift_future_epw(
epw = epw,
climate = climate,
periods = list(`2060s` = 2055:2065),
transform = transform,
reference = historical_reference(1995:2014),
dir = file.path(workflow_root, "future-epw"),
dry_run = TRUE
)
print(climate)
print(transform)
print(plan, n = 3L, width = 80L)Data previews show at most 10 rows by default. Use
n = Inf for every row, set width when
rendering into a constrained console or report, and use
verbose = TRUE for store paths, stable IDs, filters, nodes,
transform options, and additional diagnostics. Normal receipts identify
the selected transform and whether CMIP tables are automatic or forced.
After resolution, progress and watch views report exact partitions such
as Amon=gn · LImon=gr instead of collapsing them to one
representative grid. These controls affect presentation only and do not
change the workflow specification or its spec_hash.
The foreground runner starts reporting before its first ESGF request.
Its shared stage vocabulary is resolve, optional
download, extract_future, optional
extract_reference, coverage,
morph, and write_epw. Within a stage it
identifies the node, future/reference role, variable, scenario, period,
reuse/fallback outcome, and output count.
ERA5 preparation appears as a Calibration operation with
variable counts and CDS request status. Morphing counts cases, while
writing/export counts physical EPW files. A started unit does not count
as completed until its outcome arrives. Batch discovery identifies the
current method, node, and required variables.
shift_ui() controls only presentation. In a capable
terminal, the foreground dashboard updates in place with plan context, a
stage rail, the current operation, measured progress, and recent
outcomes. Wide terminals group these rows under Workflow
and Activity; narrow terminals keep the same content with
less decoration. A final Results receipt stays in
scrollback with completed cases, EPW file counts, warnings, the output
directory, and filenames.
Dynamic height uses options(epwshiftr.ui_height = 24L)
when set; otherwise it reads native POSIX terminal dimensions, then
LINES, with a 24-row fallback. One row is reserved for the
prompt. Set the option for a smaller IDE pane; oversized generic frames
fall back to a compact status row, and static detail receipts remain
complete.
| Setting | Use |
|---|---|
progress = "auto" |
Choose a live dashboard or append-only logs to match the terminal. |
progress = "log" |
Use stable logs for redirected output and screen readers. |
progress = "none" |
Suppress progress while retaining persisted events. |
detail = "detail" |
Include selection, reuse, fallback, and all output paths. |
detail = "debug" |
Also include full URLs and internal paths. |
motion = "reduced" or "none"
|
Reduce or remove animation while keeping status visible. |
Resolver attempts and elapsed time are reported separately from
measured workflow progress. On failure, Diagnosis
summarizes repeated mirror failures, the cause, and any missing
coverage. Transient failures can be retried; incomplete scientific
coverage requires changing the selection or reference.
shift_ui(refresh = ...) controls visual update
frequency, while shift_control(refresh = TRUE) explicitly
refreshes remote catalog and service information. Ordinary resume
retains the resolved selection. The heartbeat setting
controls durable worker liveness and is independent of animation.
run <- shift_future_epw(
epw = epw,
climate = shift_cmip6("BCC-CSM2-MR", c("ssp126", "ssp585")),
periods = list(`2060s` = 2055:2065),
transform = monthly_transform("original_morphing"),
reference = historical_reference(1995:2014),
dir = file.path(workflow_root, "future-epw"),
ui = shift_ui(progress = "log", detail = "detail", heartbeat = 10)
)Use a detached Rscript worker when the run should
outlive the current R session. The call returns a queued
ShiftRun immediately. Inspectors refresh that handle from
the store, and shift_watch() follows the same structured
events shown by the foreground reporter.
run <- shift_future_epw(
epw = epw,
climate = shift_cmip6("BCC-CSM2-MR", c("ssp126", "ssp585")),
periods = list(`2060s` = 2055:2065),
transform = monthly_transform("original_morphing"),
reference = historical_reference(1995:2014),
dir = file.path(workflow_root, "future-epw"),
background = TRUE
)
shift_status(run)
shift_watch(run)
shift_logs(run, tail = 50)
# Request cancellation at the next workflow boundary:
shift_cancel(run)
# Terminate the recorded worker PID after persisting the request:
shift_cancel(run, force = TRUE)Interrupting shift_watch() stops only the monitor. It
does not cancel the worker. A failed, cancelled, or partial run can
create a new attempt with
shift_resume(run, background = TRUE); the resolved node,
member, and exact per-table grid partitions remain pinned.
shift_site() describes the location that will be
extracted from climate projection data. The id is the
stable site key used in extraction plans, manifest rows, and output
naming, so use a short value that will still make sense when you process
several sites.
This article writes a deterministic Singapore baseline EPW into the temporary workflow directory so the raw render is self-contained. The climate query and remote data reads below still use live ESGF services.
You can provide the site metadata directly:
epw <- write_vignette_epw(
file.path(workflow_root, "baseline", "SGP_Singapore.486980_IWEC.epw")
)
site <- shift_site(
id = "SIN",
lon = 103.98,
lat = 1.37,
label = "Singapore"
)
site
#> ══ EPW Site ════════════════════════════════════════════════════════════════════
#> • ID: SIN
#> • Label: Singapore
#> • Coordinates: 103.980000, 1.370000Or you can read the same information directly from the EPW LOCATION header:
epw_site <- shift_site(epw)
epw_site
#> ══ EPW Site ════════════════════════════════════════════════════════════════════
#> • ID: SGP_Singapore.486980_IWEC
#> • Label: Singapore
#> • Coordinates: 103.980000, 1.370000
#> • EPW: SGP_Singapore.486980_IWEC.epwThis article keeps using the explicit site object so the
site ID is short and predictable, while the EPW file itself is passed
later as the morphing baseline.
The selected transform exposes its semantic input contract. This
staged walkthrough reads the required future-model variable set from
that contract so the same IDs are used in the ESGF request, extraction,
coverage check, and shift_morph() call. Original morphing
requires matching historical and future model inputs.
transform <- monthly_transform("original_morphing")
variables <- transform@required_inputs$model_future@variable_sets[[1L]]
variables
#> [1] "tas" "tasmax" "tasmin" "hurs" "psl" "rlds" "rsds"
#> [8] "sfcWind" "clt" "pr"shift_request() describes the remote climate data you
want. The future request targets one ESGF ScenarioMIP model, scenario,
member, and monthly table. The reference request uses the matching
historical experiment so original morphing change-factor morphing can
compare future monthly fields with reference monthly fields before
applying those changes to the baseline EPW.
The values inside filters are ESGF search fields.
options is not a search filter.
options$index_node chooses the ESGF search node; if
omitted, epwshiftr uses its default ESGF index. The other
request option currently recognized by the ESGF adapter is
time_filter_method, which controls how File records are
filtered by time after Dataset collection; the default is
"drs" filename parsing. The time argument
narrows File records after Dataset collection; it does not mean the
whole period is downloaded. A numeric year such as 2060L is
expanded to that whole calendar year.
Read more about the lower-level query object in ESGF query results, and about request-value validation in ESG dictionaries.
request <- shift_request(
project = "CMIP6",
time = 2060L,
filters = list(
activity_id = "ScenarioMIP",
source_id = "MPI-ESM1-2-LR",
experiment_id = "ssp585",
variant_label = "r1i1p1f1",
frequency = "mon",
variable_id = variables,
data_node = "esgf.ceda.ac.uk",
table_id = "Amon"
),
options = list(index_node = "https://esgf-data.dkrz.de")
)
request
#> ══ ESGF request ════════════════════════════════════════════════════════════════
#> • Index node: https://esgf-data.dkrz.de
#> ── Query parameters ────────────────────────────────────────────────────────────
#> • project = CMIP6
#> • activity_id = ScenarioMIP
#> • experiment_id = ssp585
#> • source_id = MPI-ESM1-2-LR
#> • variable_id = tas, tasmax, tasmin, hurs, psl, rlds, rsds, sfcWind, clt, pr
#> • frequency = mon
#> • variant_label = r1i1p1f1
#> • data_node = esgf.ceda.ac.uk
#> • fields = *
#> • type = Dataset
#> • offset = 0
#> • distrib = true
#> • limit = 10
#> • format = application/solr+json
#> • table_id = Amon
#> • datetime_start: [* TO 2060-01-01T00:00:00Z]
#> • datetime_stop: [2060-12-31T23:59:59Z TO *]
reference_request <- shift_request(
project = "CMIP6",
time = 1995L,
filters = list(
activity_id = "CMIP",
source_id = "MPI-ESM1-2-LR",
experiment_id = "historical",
variant_label = "r1i1p1f1",
frequency = "mon",
variable_id = variables,
data_node = "esgf.ceda.ac.uk",
table_id = "Amon"
),
options = list(index_node = "https://esgf-data.dkrz.de")
)
reference_request
#> ══ ESGF request ════════════════════════════════════════════════════════════════
#> • Index node: https://esgf-data.dkrz.de
#> ── Query parameters ────────────────────────────────────────────────────────────
#> • project = CMIP6
#> • activity_id = CMIP
#> • experiment_id = historical
#> • source_id = MPI-ESM1-2-LR
#> • variable_id = tas, tasmax, tasmin, hurs, psl, rlds, rsds, sfcWind, clt, pr
#> • frequency = mon
#> • variant_label = r1i1p1f1
#> • data_node = esgf.ceda.ac.uk
#> • fields = *
#> • type = Dataset
#> • offset = 0
#> • distrib = true
#> • limit = 10
#> • format = application/solr+json
#> • table_id = Amon
#> • datetime_start: [* TO 1995-01-01T00:00:00Z]
#> • datetime_stop: [1995-12-31T23:59:59Z TO *]The diagram below is the map for the rest of the article. Each
shift_*() call returns a stage object that can be printed,
checked, and passed to the next step. The stage also carries
run_id and step_id, so the next
shift_*() call automatically continues the same persisted
run. There is no separate session object to create or pass.
ShiftRequest
EsgResultDataset
ShiftFiles
ShiftClimate
ShiftMorphed
ShiftOutputs
shift_download() is optional. Use it when you want to
prefetch full NetCDF files for offline work, repeated extraction, or
unstable OPeNDAP access.
The ordinary path goes directly from collected File records to extraction. Full NetCDF downloads are not required unless you intentionally want a local source-file cache.
Before collecting File records, inspect the Dataset matches. If this
table is broader or narrower than intended, change the request filters
before continuing. shift_datasets() runs the Dataset-level
ESGF search described by shift_request(). The returned
EsgResultDataset object is not the data to download yet; it
is the list of Dataset records that will later be expanded into File
records. This standalone query uses the same dashboard, persisted run,
and query heartbeat reporting as the other shift_*() tasks.
Use ui = shift_ui(...) to control its presentation;
low-level EsgQuery$collect() continues to expose its native
progress control.
For a deeper look at Dataset, File, and Aggregation results, see ESGF query results.
The remaining chunks are a live ESGF walkthrough. They are displayed but not executed while the package documentation is precompiled, which keeps installed vignettes deterministic and avoids turning documentation builds into remote network jobs. Run them interactively to reproduce the workflow.
datasets <- shift_datasets(request)
datasetsUse $to_data_table() when you want row-level details for
decisions such as whether the request matched the expected variables,
model, variant, and data node.
dataset_table <- datasets$to_data_table(fields = c(
"id", "source_id", "experiment_id", "variant_label",
"variable_id", "data_node", "number_of_files"
), formatted = TRUE)
dataset_tableSummarise the table before moving on. In this request, each variable should have one matching Dataset record on the selected data node.
dataset_table[, .(
datasets = .N,
files = sum(number_of_files, na.rm = TRUE),
variables = paste(sort(unique(variable_id)), collapse = ", ")
), by = .(source_id, experiment_id, variant_label, data_node)]The Dataset result can also be filtered locally before collecting child File records. This is useful when a broad request intentionally returns several models, variants, data nodes, or variables and you want to inspect or keep only part of the match. The example below keeps only two variables so the effect is easy to see.
selected_datasets <- datasets$filter(function(x) {
x$variable_id %in% c("tas", "hurs")
})
selected_datasets$to_data_table(fields = c(
"id", "variable_id", "data_node", "number_of_files"
))For a lower-level workflow, you can collect File records from that
subset directly. The staged shift_collect() call below
performs the same Dataset-to-File expansion for the original request and
stores the result in an EsgStore, so the main workflow
continues with shift_collect().
selected_files <- selected_datasets$collect(
type = "File",
fields = "*",
all = TRUE,
limit = NULL
)store is the local directory where
epwshiftr records ESGF File metadata, download tasks,
extraction outputs, morphing factors, and generated EPWs.
shift_collect() first collects Dataset records, then uses
Dataset$collect(type = "File") to collect the concrete
files needed by the rest of the workflow. The returned
ShiftFiles object is the workflow stage: it remembers the
store path and internal query ID so later steps do not need the user to
pass file paths or manifest IDs.
For store internals such as query snapshots, file catalogs, artifacts, and tracked updates, see ESG stores.
files <- shift_collect(
request,
store = file.path(workflow_root, "singapore-store")
)
files
reference_files <- shift_collect(
reference_request,
store = file.path(workflow_root, "singapore-store")
)
reference_filesUse shift_files() when you want to inspect the
underlying EsgResultFile object that was saved into the
store. Printing it gives the same high-level summary as a direct ESGF
File query result.
file_result <- shift_files(files)
file_resultConvert the File result to a table when you want to inspect exactly which files were found. The URL columns are long, so this view shows whether each file has OPeNDAP and HTTPServer access instead of printing the full URLs.
file_table <- file_result$to_data_table(fields = c(
"filename", "variable_id", "data_node", "size",
"url_opendap", "url_download"
), formatted = TRUE)
file_table[, .(
filename,
variable_id,
data_node,
size,
opendap = !is.na(url_opendap) & nzchar(url_opendap),
http = !is.na(url_download) & nzchar(url_download)
)]For the normal single-site workflow, you can skip
shift_download() and go directly to
shift_extract(). Extraction opens the OPeNDAP URL first and
reads only the requested site, variables, and time range before storing
the extracted result as Parquet.
shift_download() is useful when you deliberately want a
complete local copy of the original ESGF NetCDF files before extraction.
It downloads full source files through selected HTTPServer
URLs into the store’s downloads/ directory. This is
different from OPeNDAP, which lets shift_extract() read
only the requested site, variables, and time range.
Use this optional prefetch step when you plan to reuse the same source files for many sites or periods, need offline extraction later, or expect OPeNDAP to be unavailable or unstable.
By default, shift_download() runs in the foreground
(run = TRUE, background = FALSE). In an
interactive session, keep progress = TRUE to see per-file
progress bars. This article sets progress = FALSE only to
keep the precompiled output compact.
If the network drops, the downloader keeps partial .part
files and resume = TRUE lets the next run continue where
possible. If the final file is already present and complete, it is
reused. Use overwrite = TRUE only when you want to discard
an existing completed file and download it again.
If a data node becomes unstable, rerun shift_download()
with the same stage. The store keeps the File records and download
session metadata, while the downloader records task status and data-node
history. If you run the optional chunk below, inspect the result with
shift_status(downloads),
shift_check(downloads), and
data.table::as.data.table(downloads).
For persistent sessions, background jobs, daemon mode, retries, and node history, see Downloader. For the same operations from a terminal, see CLI operations.
downloads <- shift_download(
files,
replica = "current",
service = "HTTPServer",
strategy = "stable",
probe = FALSE,
progress = FALSE
)
downloadsThis is where the remote climate data are actually read in the
default workflow. shift_extract() opens OPeNDAP when
possible, extracts only the requested site and period, and stores the
extracted rows as Parquet artifacts in the store. In the code below the
result is named extracted because it is the extracted
site-level climate stage. Its class is ShiftClimate,
because that stage is the climate data that shift_morph()
will summarise and compare with the baseline EPW.
epw_morph_periods() maps user-facing period labels to
one or more years. The name, such as 2060s, becomes the
period label in summaries, morphing cases, and output paths. The numeric
value is the year or years used to calculate that period. This article
uses one year so the remote extraction stays small:
epw_morph_periods(`2060s` = 2060L)A wider period is also valid, for example:
epw_morph_periods(`2060s` = 2055:2064)The collected files must cover every year used by the period.
fallback = "auto" means extraction tries OPeNDAP first
and may fall back to HTTP file downloads when remote OPeNDAP access is
unavailable. Use "error" when you want remote access
failures to stop the extraction instead.
Extraction is recorded in the local EsgStore; see ESG stores for the lower-level API. If
OPeNDAP, data-node, or coverage problems appear, see ESGF troubleshooting.
periods <- epw_morph_periods(`2060s` = 2060L)
reference_periods <- epw_morph_periods(reference = 1995L)
extracted <- shift_extract(
files,
site = site,
periods = periods,
variables = variables,
fallback = "auto"
)
extracted
reference <- shift_extract(
reference_files,
site = site,
periods = reference_periods,
variables = variables,
fallback = "auto"
)
referenceshift_coverage() checks whether every requested variable
has extracted rows for the selected site and period. This is the main
sanity check before morphing.
coverage <- shift_coverage(extracted)
coverage[, .(variable_id, complete, status, output_rows, output_file_count)]The extracted values are not stored inside the small stage object.
They are written as partitioned Parquet files under the store and
registered in the store manifest. shift_artifacts() shows
those registered files:
extract_artifacts <- shift_artifacts(extracted)
extract_artifacts[kind == "extract", .(kind, role, relative_path)]Use shift_data() when you want to inspect the actual
extracted table without manually finding or reading those Parquet files.
By default it returns a preview instead of loading everything into
memory.
extracted_data <- shift_data(
extracted,
n = 20L,
columns = c("site_id", "variable_id", "time", "lon", "lat", "value", "units")
)
extracted_datashift_morph() summarises the extracted monthly climate,
compares it with the baseline EPW, creates morphing factors, and writes
morphed hourly results back to the store. With
strict = TRUE, missing required variables or incomplete
coverage are blocking errors instead of warnings.
shift_morph() wraps the lower-level
EpwMorpher planning and execution API. See EpwMorpher when you need to inspect monthly
summaries, factor diagnostics, case grouping, or transform input
contracts.
Matching historical CMIP6 data are required so original morphing can
compute future-versus-historical change factors. Pass either an
extracted historical ShiftClimate stage or an explicit
reference spec such as historical_reference(1995:2014). The
same transform can be reused across sites, models, scenarios, and
periods.
shift_morph(
extracted,
reference = reference,
baseline = epw,
transform = transform,
strict = TRUE
)
morphed <- shift_morph(
extracted,
reference = reference,
baseline = epw,
transform = transform,
strict = TRUE
)
morphedThe morphed stage is still store-native. It contains
hourly future weather data as Parquet artifacts, not EPW text files yet.
Inspect the artifact rows when you want to see where those intermediate
results live:
morph_artifacts <- shift_artifacts(morphed)
morph_artifacts[, .(kind, role, relative_path)]Use the same shift_data() helper to preview the hourly
morphed weather table. The metadata columns identify the morphing case;
the weather columns are the hourly EPW-style values that will be written
to the final EPW. The preview below omits long IDs to keep the table
readable; include case_id in columns when you
need to join rows back to a specific morphing case.
morphed_data <- shift_data(
morphed,
n = 24L,
columns = c(
"period", "year", "month", "day", "hour",
"dry_bulb_temperature", "relative_humidity", "wind_speed"
)
)
morphed_datashift_epw() writes EnergyPlus Weather files from the
morphed hourly results. It returns a ShiftOutputs stage.
The first chunk assigns the result while hiding verbose writer output;
the second prints the stage object.
For the lower-level write path and output registry, see EpwMorpher.
epws <- shift_epw(morphed)
epwsshift_outputs() lists the EPW files written by
shift_epw(). These are the files you can pass directly to
EnergyPlus.
outputs <- shift_outputs(epws)
outputs[, .(path, source_id, experiment_id, variant_label, period)]shift_data(epws) reads the written EPW file back and
returns its hourly weather data with output metadata attached. This is
useful for confirming that the final file contains the same kind of
hourly weather values you inspected in the store-native morphed Parquet
step. Output metadata such as output_id,
case_id, and path are available in
shift_data(epws); they are omitted here so the weather
values stay visible.
epw_data <- shift_data(
epws,
n = 24L,
columns = c(
"period", "year", "month", "day", "hour",
"dry_bulb_temperature", "relative_humidity", "wind_speed"
)
)
epw_dataAt this point the workflow has produced EPW files, but there are still a few checks worth doing before using them in EnergyPlus or passing them to someone else. These checks answer three practical questions:
Use shift_status() when you want a compact stage-level
check. It returns a single status string so it can be used in scripts,
reports, or simple guards. For a successful run, the sequence should end
with an EPW stage marked written.
data.table::data.table(
stage = c("request", "collect", "extract", "reference", "morph", "epw"),
status = c(
shift_status(request),
shift_status(files),
shift_status(extracted),
shift_status(reference),
shift_status(morphed),
shift_status(epws)
)
)Stage status and run status answer different questions.
shift_status(epws) reports that the EPW artifact is
written; shift_status(shift_run_get(epws)) is
waiting because a store-local EPW can still be exported.
Calling shift_export_epw(epws, dir) completes the run. If
the store-local file is the intentional endpoint, use
shift_complete(epws) instead.
Diagnostics are the first place to look when a stage is blocked, failed, or returns fewer outputs than expected. An empty diagnostics table is the normal successful result. Non-empty rows are intended to explain the stage, severity, and action rather than expose internal manifest IDs first.
For common causes and the first place to look for each class of failure, see ESGF troubleshooting.
shift_diagnostics(epws)The final EPW depends on the extracted climate table. Even after the EPW file is written, it is useful to confirm that each morphing variable has complete monthly coverage for the requested period. Missing or incomplete rows here usually mean the original ESGF query, Dataset selection, or extraction period needs to be adjusted.
Coverage problems usually originate in query selection, time filtering, or remote access. The ESGF troubleshooting article collects those checks in one place.
coverage <- shift_coverage(epws)
coverage[, .(variable_id, complete, status, output_rows, output_file_count)]shift_outputs() lists the written EPW files and their
case metadata. These are the files to pass to EnergyPlus, archive with a
simulation run, or reopen with the package’s internal EPW reader. The
store also keeps the intermediate Parquet artifacts so the workflow can
be inspected or reused without repeating the remote query.
outputs <- shift_outputs(epws)
outputs[, .(
file = basename(path),
source_id,
experiment_id,
variant_label,
period
)]