vignettes/articles/future-epw-workflow.Rmd
future-epw-workflow.Rmd
library(epwshiftr)
if (!exists("shift_request", mode = "function")) {
stop(
"This raw article must be rendered from the package source with ",
"`tools/render-raw-vignettes.R`.",
call. = FALSE
)
}This article is the recommended main workflow for epwshiftr. It runs
a real store-native shift workflow from ESGF File records to generated
EPW files. It uses monthly Amon data so the live remote
reads are much smaller than an equivalent daily workflow.
The examples below use the high-level shift_*() facade.
When a step touches a lower-level engine, this article links to the
companion article for that layer: ESGF
query results, ESG dictionaries,
ESG stores, Downloader, EpwMorpher, CLI operations, and ESGF troubleshooting.
Use a temporary store for this article run. For a real project, replace this with a persistent project cache path.
workflow_root <- file.path(tempdir(), "epwshiftr-future-epw-workflow")
if (dir.exists(workflow_root)) {
unlink(workflow_root, recursive = TRUE)
}
dir.create(workflow_root, recursive = TRUE, showWarnings = FALSE)
options(
epwshiftr.dir_cache = file.path(workflow_root, "cache")
)shift_site() describes the location that will be
extracted from climate projection data. The id is the
stable site key used in extraction plans, manifest rows, and output
naming, so use a short value that will still make sense when you process
several sites.
This article writes a deterministic Singapore baseline EPW into the temporary workflow directory so the raw render is self-contained. The climate query and remote data reads below still use live ESGF services.
You can provide the site metadata directly:
epw <- write_vignette_epw(
file.path(workflow_root, "baseline", "SGP_Singapore.486980_IWEC.epw")
)
site <- shift_site(
id = "SIN",
lon = 103.98,
lat = 1.37,
label = "Singapore"
)
site
#> <ShiftSite>
#> id: SIN
#> lonlat: 103.980000, 1.370000
#> label: SingaporeOr you can read the same information from an eplusr::Epw
object:
epw_site <- shift_site(eplusr::read_epw(epw))
epw_site
#> <ShiftSite>
#> id: 486980
#> lonlat: 103.980000, 1.370000
#> label: Singapore
#> epw: EpwThis article keeps using the explicit site object so the
site ID is short and predictable, while the EPW file itself is passed
later as the morphing baseline.
epw_morph_variables() returns the CMIP variable IDs
needed by the selected morphing recipe. The result is a plain character
vector because the same variable IDs are used at several workflow
stages: first as the ESGF variable_id filter in
shift_request(), then as the extraction variable list in
shift_extract(), and finally as the coverage check used by
shift_morph().
The helper provides three named variable sets:
"minimal": air temperature and relative humidity,
useful for relaxed demonstrations."recommended": the current strict Belcher recipe set,
including precipitation."extended": the recommended set plus related max/min
variables for future recipes.This article uses the default Belcher recipe. It is a change-factor backend, so the morphing step needs both a future climate extraction and a reference climate extraction.
shift_recipe <- epw_morph_recipe("belcher")
variables <- epw_morph_variables(shift_recipe)
variables
#> [1] "tas" "hurs" "psl" "rlds" "rsds" "sfcWind" "clt"
#> [8] "pr"shift_request() describes the remote climate data you
want. The future request targets one ESGF ScenarioMIP model, scenario,
member, and monthly table. The reference request uses the matching
historical experiment so Belcher change-factor morphing can compare
future monthly fields with reference monthly fields before applying
those changes to the baseline EPW.
The values inside filters are ESGF search fields.
options is not a search filter.
options$index_node chooses the ESGF search node; if
omitted, epwshiftr uses its default ESGF index. The other
request option currently recognized by the ESGF adapter is
time_filter_method, which controls how File records are
filtered by time after Dataset collection; the default is
"drs" filename parsing. The time argument
narrows File records after Dataset collection; it does not mean the
whole period is downloaded. A numeric year such as 2060L is
expanded to that whole calendar year.
Read more about the lower-level query object in ESGF query results, and about request-value validation in ESG dictionaries.
request <- shift_request(
project = "CMIP6",
time = 2060L,
filters = list(
activity_id = "ScenarioMIP",
source_id = "MPI-ESM1-2-LR",
experiment_id = "ssp585",
variant_label = "r1i1p1f1",
frequency = "mon",
variable_id = variables,
data_node = "esgf.ceda.ac.uk",
table_id = "Amon"
),
options = list(index_node = "https://esgf-data.dkrz.de")
)
request
#> <epwshiftr::ShiftRequest>
#> stage: request
#> status: new
#> project: CMIP6
#> source: MPI-ESM1-2-LR
#> experiment: ssp585
#> variant: r1i1p1f1
#> frequency: mon
#> variables: tas, hurs, psl, rlds, rsds, sfcWind, clt, ... (8 total)
#> time: 2060-01-01T00:00:00Z -> 2060-12-31T23:59:59Z
reference_request <- shift_request(
project = "CMIP6",
time = 1995L,
filters = list(
activity_id = "CMIP",
source_id = "MPI-ESM1-2-LR",
experiment_id = "historical",
variant_label = "r1i1p1f1",
frequency = "mon",
variable_id = variables,
data_node = "esgf.ceda.ac.uk",
table_id = "Amon"
),
options = list(index_node = "https://esgf-data.dkrz.de")
)
reference_requestThe diagram below is the map for the rest of the article. Each
shift_*() call returns a stage object that can be printed,
checked, and passed to the next step.
ShiftRequest
EsgResultDataset
ShiftFiles
ShiftClimate
ShiftMorphed
ShiftOutputs
shift_download() is optional. Use it when you want to
prefetch full NetCDF files for offline work, repeated extraction, or
unstable OPeNDAP access.
The ordinary path goes directly from collected File records to extraction. Full NetCDF downloads are not required unless you intentionally want a local source-file cache.
Before collecting File records, inspect the Dataset matches. If this
table is broader or narrower than intended, change the request filters
before continuing. shift_datasets() runs the Dataset-level
ESGF search described by shift_request(). The returned
EsgResultDataset object is not the data to download yet; it
is the list of Dataset records that will later be expanded into File
records.
For a deeper look at Dataset, File, and Aggregation results, see ESGF query results.
datasets <- shift_datasets(request)
datasets
#> [1] CMIP6.ScenarioMIP.MPI-M.MPI-ESM1-2-LR.ssp585.r1i1p1f1.Amon.clt.gn.v20190710|esgf.ceda.ac.uk
#> [ 5 Files, 50.15 MiB | No Aggregations ]
#> [ Access: <HTTPServer, OPENDAP> ]
#> [2] CMIP6.ScenarioMIP.MPI-M.MPI-ESM1-2-LR.ssp585.r1i1p1f1.Amon.hurs.gn.v20190815|esgf.ceda.ac.uk
#> [ 5 Files, 45.55 MiB | No Aggregations ]
#> [ Access: <HTTPServer, OPENDAP> ]
#> [3] CMIP6.ScenarioMIP.MPI-M.MPI-ESM1-2-LR.ssp585.r1i1p1f1.Amon.psl.gn.v20190710|esgf.ceda.ac.uk
#> [ 5 Files, 33.85 MiB | No Aggregations ]
#> [ Access: <HTTPServer, OPENDAP> ]
#> [4] CMIP6.ScenarioMIP.MPI-M.MPI-ESM1-2-LR.ssp585.r1i1p1f1.Amon.rlds.gn.v20190710|esgf.ceda.ac.uk
#> [ 5 Files, 37.41 MiB | No Aggregations ]
#> [ Access: <HTTPServer, OPENDAP> ]
#> [5] CMIP6.ScenarioMIP.MPI-M.MPI-ESM1-2-LR.ssp585.r1i1p1f1.Amon.rsds.gn.v20190710|esgf.ceda.ac.uk
#> [ 5 Files, 36.4 MiB | No Aggregations ]
#> [ Access: <HTTPServer, OPENDAP> ]
#> [6] CMIP6.ScenarioMIP.MPI-M.MPI-ESM1-2-LR.ssp585.r1i1p1f1.Amon.sfcWind.gn.v20190710|esgf.ceda.ac.uk
#> [ 5 Files, 40.27 MiB | No Aggregations ]
#> [ Access: <HTTPServer, OPENDAP> ]
#> [7] CMIP6.ScenarioMIP.MPI-M.MPI-ESM1-2-LR.ssp585.r1i1p1f1.Amon.tas.gn.v20190710|esgf.ceda.ac.uk
#> [ 5 Files, 32.91 MiB | No Aggregations ]
#> [ Access: <HTTPServer, OPENDAP> ]Use $to_data_table() when you want row-level details for
decisions such as whether the request matched the expected variables,
model, variant, and data node.
dataset_table <- datasets$to_data_table(fields = c(
"id", "source_id", "experiment_id", "variant_label",
"variable_id", "data_node", "number_of_files"
), formatted = TRUE)
dataset_table
#> id
#> <char>
#> 1: CMIP6.ScenarioMIP.MPI-M.MPI-ESM1-2-LR.ssp585.r1i1p1f1.Amon.clt.gn.v20190710|esgf.ceda.ac.uk
#> 2: CMIP6.ScenarioMIP.MPI-M.MPI-ESM1-2-LR.ssp585.r1i1p1f1.Amon.hurs.gn.v20190815|esgf.ceda.ac.uk
#> 3: CMIP6.ScenarioMIP.MPI-M.MPI-ESM1-2-LR.ssp585.r1i1p1f1.Amon.psl.gn.v20190710|esgf.ceda.ac.uk
#> 4: CMIP6.ScenarioMIP.MPI-M.MPI-ESM1-2-LR.ssp585.r1i1p1f1.Amon.rlds.gn.v20190710|esgf.ceda.ac.uk
#> 5: CMIP6.ScenarioMIP.MPI-M.MPI-ESM1-2-LR.ssp585.r1i1p1f1.Amon.rsds.gn.v20190710|esgf.ceda.ac.uk
#> 6: CMIP6.ScenarioMIP.MPI-M.MPI-ESM1-2-LR.ssp585.r1i1p1f1.Amon.sfcWind.gn.v20190710|esgf.ceda.ac.uk
#> 7: CMIP6.ScenarioMIP.MPI-M.MPI-ESM1-2-LR.ssp585.r1i1p1f1.Amon.tas.gn.v20190710|esgf.ceda.ac.uk
#> source_id experiment_id variant_label variable_id data_node
#> <char> <char> <char> <char> <char>
#> 1: MPI-ESM1-2-LR ssp585 r1i1p1f1 clt esgf.ceda.ac.uk
#> 2: MPI-ESM1-2-LR ssp585 r1i1p1f1 hurs esgf.ceda.ac.uk
#> 3: MPI-ESM1-2-LR ssp585 r1i1p1f1 psl esgf.ceda.ac.uk
#> 4: MPI-ESM1-2-LR ssp585 r1i1p1f1 rlds esgf.ceda.ac.uk
#> 5: MPI-ESM1-2-LR ssp585 r1i1p1f1 rsds esgf.ceda.ac.uk
#> 6: MPI-ESM1-2-LR ssp585 r1i1p1f1 sfcWind esgf.ceda.ac.uk
#> 7: MPI-ESM1-2-LR ssp585 r1i1p1f1 tas esgf.ceda.ac.uk
#> number_of_files
#> <int>
#> 1: 5
#> 2: 5
#> 3: 5
#> 4: 5
#> 5: 5
#> 6: 5
#> 7: 5Summarise the table before moving on. In this request, each variable should have one matching Dataset record on the selected data node.
dataset_table[, .(
datasets = .N,
files = sum(number_of_files, na.rm = TRUE),
variables = paste(sort(unique(variable_id)), collapse = ", ")
), by = .(source_id, experiment_id, variant_label, data_node)]
#> source_id experiment_id variant_label data_node datasets files
#> <char> <char> <char> <char> <int> <int>
#> 1: MPI-ESM1-2-LR ssp585 r1i1p1f1 esgf.ceda.ac.uk 7 35
#> variables
#> <char>
#> 1: clt, hurs, psl, rlds, rsds, sfcWind, tasThe Dataset result can also be filtered locally before collecting child File records. This is useful when a broad request intentionally returns several models, variants, data nodes, or variables and you want to inspect or keep only part of the match. The example below keeps only two variables so the effect is easy to see.
selected_datasets <- datasets$filter(function(x) {
x$variable_id %in% c("tas", "hurs")
})
selected_datasets$to_data_table(fields = c(
"id", "variable_id", "data_node", "number_of_files"
))
#> id
#> <char>
#> 1: CMIP6.ScenarioMIP.MPI-M.MPI-ESM1-2-LR.ssp585.r1i1p1f1.Amon.hurs.gn.v20190815|esgf.ceda.ac.uk
#> 2: CMIP6.ScenarioMIP.MPI-M.MPI-ESM1-2-LR.ssp585.r1i1p1f1.Amon.tas.gn.v20190710|esgf.ceda.ac.uk
#> variable_id data_node number_of_files
#> <char> <char> <int>
#> 1: hurs esgf.ceda.ac.uk 5
#> 2: tas esgf.ceda.ac.uk 5For a lower-level workflow, you can collect File records from that
subset directly. The staged shift_collect() call below
performs the same Dataset-to-File expansion for the original request and
stores the result in an EsgStore, so the main workflow
continues with shift_collect().
selected_files <- selected_datasets$collect(
type = "File",
fields = "*",
all = TRUE,
limit = NULL
)store is the local directory where
epwshiftr records ESGF File metadata, download tasks,
extraction outputs, morphing factors, and generated EPWs.
shift_collect() first collects Dataset records, then uses
Dataset$collect(type = "File") to collect the concrete
files needed by the rest of the workflow. The returned
ShiftFiles object is the workflow stage: it remembers the
store path and internal query ID so later steps do not need the user to
pass file paths or manifest IDs.
For store internals such as query snapshots, file catalogs, artifacts, and tracked updates, see ESG stores.
files <- shift_collect(
request,
store = file.path(workflow_root, "singapore-store")
)
files
#> <epwshiftr::ShiftFiles>
#> stage: files
#> status: collected
#> store: <tempdir>/epwshiftr-future-epw-workflow/singapore-store
#> files: 7
#> variables: clt, hurs, psl, rlds, rsds, sfcWind, tas
#> ids: query_id
reference_files <- shift_collect(
reference_request,
store = file.path(workflow_root, "singapore-store")
)
reference_filesUse shift_files() when you want to inspect the
underlying EsgResultFile object that was saved into the
store. Printing it gives the same high-level summary as a direct ESGF
File query result.
file_result <- shift_files(files)
file_result
#> [1] CMIP6.ScenarioMIP.MPI-M.MPI-ESM1-2-LR.ssp585.r1i1p1f1.Amon.clt.gn.v20190710.clt_Amon_MPI-ESM1-2-LR_ssp585_r1i1p1f1_gn_205501-207412.nc|esgf.ceda.ac.uk
#> [ 11.65 MiB | Access: <HTTPServer, OPENDAP> ]
#> [2] CMIP6.ScenarioMIP.MPI-M.MPI-ESM1-2-LR.ssp585.r1i1p1f1.Amon.hurs.gn.v20190815.hurs_Amon_MPI-ESM1-2-LR_ssp585_r1i1p1f1_gn_205501-207412.nc|esgf.ceda.ac.uk
#> [ 10.58 MiB | Access: <HTTPServer, OPENDAP> ]
#> [3] CMIP6.ScenarioMIP.MPI-M.MPI-ESM1-2-LR.ssp585.r1i1p1f1.Amon.psl.gn.v20190710.psl_Amon_MPI-ESM1-2-LR_ssp585_r1i1p1f1_gn_205501-207412.nc|esgf.ceda.ac.uk
#> [ 7.86 MiB | Access: <HTTPServer, OPENDAP> ]
#> [4] CMIP6.ScenarioMIP.MPI-M.MPI-ESM1-2-LR.ssp585.r1i1p1f1.Amon.rlds.gn.v20190710.rlds_Amon_MPI-ESM1-2-LR_ssp585_r1i1p1f1_gn_205501-207412.nc|esgf.ceda.ac.uk
#> [ 8.68 MiB | Access: <HTTPServer, OPENDAP> ]
#> [5] CMIP6.ScenarioMIP.MPI-M.MPI-ESM1-2-LR.ssp585.r1i1p1f1.Amon.rsds.gn.v20190710.rsds_Amon_MPI-ESM1-2-LR_ssp585_r1i1p1f1_gn_205501-207412.nc|esgf.ceda.ac.uk
#> [ 8.44 MiB | Access: <HTTPServer, OPENDAP> ]
#> [6] CMIP6.ScenarioMIP.MPI-M.MPI-ESM1-2-LR.ssp585.r1i1p1f1.Amon.sfcWind.gn.v20190710.sfcWind_Amon_MPI-ESM1-2-LR_ssp585_r1i1p1f1_gn_205501-207412.nc|esgf.ceda.ac.uk
#> [ 9.35 MiB | Access: <HTTPServer, OPENDAP> ]
#> [7] CMIP6.ScenarioMIP.MPI-M.MPI-ESM1-2-LR.ssp585.r1i1p1f1.Amon.tas.gn.v20190710.tas_Amon_MPI-ESM1-2-LR_ssp585_r1i1p1f1_gn_205501-207412.nc|esgf.ceda.ac.uk
#> [ 7.64 MiB | Access: <HTTPServer, OPENDAP> ]Convert the File result to a table when you want to inspect exactly which files were found. The URL columns are long, so this view shows whether each file has OPeNDAP and HTTPServer access instead of printing the full URLs.
file_table <- file_result$to_data_table(fields = c(
"filename", "variable_id", "data_node", "size",
"url_opendap", "url_download"
), formatted = TRUE)
file_table[, .(
filename,
variable_id,
data_node,
size,
opendap = !is.na(url_opendap) & nzchar(url_opendap),
http = !is.na(url_download) & nzchar(url_download)
)]
#> filename variable_id
#> <char> <char>
#> 1: clt_Amon_MPI-ESM1-2-LR_ssp585_r1i1p1f1_gn_205501-207412.nc clt
#> 2: hurs_Amon_MPI-ESM1-2-LR_ssp585_r1i1p1f1_gn_205501-207412.nc hurs
#> 3: psl_Amon_MPI-ESM1-2-LR_ssp585_r1i1p1f1_gn_205501-207412.nc psl
#> 4: rlds_Amon_MPI-ESM1-2-LR_ssp585_r1i1p1f1_gn_205501-207412.nc rlds
#> 5: rsds_Amon_MPI-ESM1-2-LR_ssp585_r1i1p1f1_gn_205501-207412.nc rsds
#> 6: sfcWind_Amon_MPI-ESM1-2-LR_ssp585_r1i1p1f1_gn_205501-207412.nc sfcWind
#> 7: tas_Amon_MPI-ESM1-2-LR_ssp585_r1i1p1f1_gn_205501-207412.nc tas
#> data_node size opendap http
#> <char> <units> <lgcl> <lgcl>
#> 1: esgf.ceda.ac.uk 11.652859 [MiB] TRUE TRUE
#> 2: esgf.ceda.ac.uk 10.582312 [MiB] TRUE TRUE
#> 3: esgf.ceda.ac.uk 7.863685 [MiB] TRUE TRUE
#> 4: esgf.ceda.ac.uk 8.684435 [MiB] TRUE TRUE
#> 5: esgf.ceda.ac.uk 8.441223 [MiB] TRUE TRUE
#> 6: esgf.ceda.ac.uk 9.353412 [MiB] TRUE TRUE
#> 7: esgf.ceda.ac.uk 7.640529 [MiB] TRUE TRUEFor the normal single-site workflow, you can skip
shift_download() and go directly to
shift_extract(). Extraction opens the OPeNDAP URL first and
reads only the requested site, variables, and time range before storing
the extracted result as Parquet.
shift_download() is useful when you deliberately want a
complete local copy of the original ESGF NetCDF files before extraction.
It downloads full source files through selected HTTPServer
URLs into the store’s downloads/ directory. This is
different from OPeNDAP, which lets shift_extract() read
only the requested site, variables, and time range.
Use this optional prefetch step when you plan to reuse the same source files for many sites or periods, need offline extraction later, or expect OPeNDAP to be unavailable or unstable.
By default, shift_download() runs in the foreground
(run = TRUE, background = FALSE). In an
interactive session, keep progress = TRUE to see per-file
progress bars. This article sets progress = FALSE only to
keep the precompiled output compact.
If the network drops, the downloader keeps partial .part
files and resume = TRUE lets the next run continue where
possible. If the final file is already present and complete, it is
reused. Use overwrite = TRUE only when you want to discard
an existing completed file and download it again.
If a data node becomes unstable, rerun shift_download()
with the same stage. The store keeps the File records and download
session metadata, while the downloader records task status and data-node
history. If you run the optional chunk below, inspect the result with
shift_status(downloads),
shift_check(downloads), and
data.table::as.data.table(downloads).
For persistent sessions, background jobs, daemon mode, retries, and node history, see Downloader. For the same operations from a terminal, see CLI operations.
downloads <- shift_download(
files,
replica = "current",
service = "HTTPServer",
strategy = "stable",
probe = FALSE,
progress = FALSE
)
downloadsThis is where the remote climate data are actually read in the
default workflow. shift_extract() opens OPeNDAP when
possible, extracts only the requested site and period, and stores the
extracted rows as Parquet artifacts in the store. In the code below the
result is named extracted because it is the extracted
site-level climate stage. Its class is ShiftClimate,
because that stage is the climate data that shift_morph()
will summarise and compare with the baseline EPW.
epw_morph_periods() maps user-facing period labels to
one or more years. The name, such as 2060s, becomes the
period label in summaries, morphing cases, and output paths. The numeric
value is the year or years used to calculate that period. This article
uses one year so the remote extraction stays small:
epw_morph_periods(`2060s` = 2060L)A wider period is also valid, for example:
epw_morph_periods(`2060s` = 2055:2064)The collected files must cover every year used by the period.
fallback = "auto" means extraction tries OPeNDAP first
and may fall back to HTTP file downloads when remote OPeNDAP access is
unavailable. Use "error" when you want remote access
failures to stop the extraction instead.
Extraction is recorded in the local EsgStore; see ESG stores for the lower-level API. If
OPeNDAP, data-node, or coverage problems appear, see ESGF troubleshooting.
periods <- epw_morph_periods(`2060s` = 2060L)
reference_periods <- epw_morph_periods(reference = 1995L)
extracted <- shift_extract(
files,
site = site,
periods = periods,
variables = variables,
fallback = "auto"
)
extracted
#> <epwshiftr::ShiftClimate>
#> stage: climate
#> status: extracted
#> store: <tempdir>/epwshiftr-future-epw-workflow/singapore-store
#> coverage: 7/7 complete
#> ids: query_id, plan_id
reference <- shift_extract(
reference_files,
site = site,
periods = reference_periods,
variables = variables,
fallback = "auto"
)
referenceshift_coverage() checks whether every requested variable
has extracted rows for the selected site and period. This is the main
sanity check before morphing.
coverage <- shift_coverage(extracted)
coverage[, .(variable_id, complete, status, output_rows, output_file_count)]
#> variable_id complete status output_rows output_file_count
#> <char> <lgcl> <char> <int> <int>
#> 1: rlds TRUE done 12 1
#> 2: rsds TRUE done 12 1
#> 3: psl TRUE done 12 1
#> 4: tas TRUE done 12 1
#> 5: clt TRUE done 12 1
#> 6: sfcWind TRUE done 12 1
#> 7: hurs TRUE done 12 1The extracted values are not stored inside the small stage object.
They are written as partitioned Parquet files under the store and
registered in the store manifest. shift_artifacts() shows
those registered files:
extract_artifacts <- shift_artifacts(extracted)
extract_artifacts[kind == "extract", .(kind, role, relative_path)]
#> kind role
#> <char> <char>
#> 1: extract derived
#> 2: extract derived
#> 3: extract derived
#> 4: extract derived
#> 5: extract derived
#> 6: extract derived
#> 7: extract derived
#> relative_path
#> <char>
#> 1: extracts/project=CMIP6/source_id=MPI-ESM1-2-LR/experiment_id=ssp585/variant_label=r1i1p1f1/frequency=mon/variable_id=clt/year=2060/part-d442dc702cdbdbb9df13f618bef50a49fe8cbdb6391357d637cbbfcc4e1d02ad.parquet
#> 2: extracts/project=CMIP6/source_id=MPI-ESM1-2-LR/experiment_id=ssp585/variant_label=r1i1p1f1/frequency=mon/variable_id=hurs/year=2060/part-d8f84ee054ee73ba2c20a6bbcd155510eea876556d4c1a08a1324d496b72b7ab.parquet
#> 3: extracts/project=CMIP6/source_id=MPI-ESM1-2-LR/experiment_id=ssp585/variant_label=r1i1p1f1/frequency=mon/variable_id=psl/year=2060/part-53c9ac1d43c5aff19881432f5380a5d10dba8fb0952df8fd7f30a7d027ad0922.parquet
#> 4: extracts/project=CMIP6/source_id=MPI-ESM1-2-LR/experiment_id=ssp585/variant_label=r1i1p1f1/frequency=mon/variable_id=rlds/year=2060/part-0529c672b2743d596b4ce567f20bc7da8679879d78416f84bbfa6a1c9d833bb3.parquet
#> 5: extracts/project=CMIP6/source_id=MPI-ESM1-2-LR/experiment_id=ssp585/variant_label=r1i1p1f1/frequency=mon/variable_id=rsds/year=2060/part-47485cadca0ce1317db7c1cf9db32e49dfa14fe931ee7b9aaa7a295f2167f38e.parquet
#> 6: extracts/project=CMIP6/source_id=MPI-ESM1-2-LR/experiment_id=ssp585/variant_label=r1i1p1f1/frequency=mon/variable_id=sfcWind/year=2060/part-d7e5fe8f877b0fd8ce0ec12d3c7fec7e446804f67338fdbbdf6acd7b08a93aef.parquet
#> 7: extracts/project=CMIP6/source_id=MPI-ESM1-2-LR/experiment_id=ssp585/variant_label=r1i1p1f1/frequency=mon/variable_id=tas/year=2060/part-7d32f39b9de114229f4abf5d8255ec18fdcbc882887b22ccb212ed4444dd303f.parquetUse shift_data() when you want to inspect the actual
extracted table without manually finding or reading those Parquet files.
By default it returns a preview instead of loading everything into
memory.
extracted_data <- shift_data(
extracted,
n = 20L,
columns = c("site_id", "variable_id", "time", "lon", "lat", "value", "units")
)
extracted_data
#> site_id variable_id time lon lat value units
#> <char> <char> <POSc> <num> <num> <num> <char>
#> 1: SIN clt 2060-01-16 12:00:00 103.125 0.93263 82.30624 %
#> 2: SIN clt 2060-02-15 12:00:00 103.125 0.93263 64.81476 %
#> 3: SIN clt 2060-03-16 12:00:00 103.125 0.93263 57.06557 %
#> 4: SIN clt 2060-04-16 00:00:00 103.125 0.93263 45.10655 %
#> 5: SIN clt 2060-05-16 12:00:00 103.125 0.93263 58.98287 %
#> 6: SIN clt 2060-06-16 00:00:00 103.125 0.93263 87.22534 %
#> 7: SIN clt 2060-07-16 12:00:00 103.125 0.93263 92.17834 %
#> 8: SIN clt 2060-08-16 12:00:00 103.125 0.93263 87.00104 %
#> 9: SIN clt 2060-09-16 00:00:00 103.125 0.93263 93.51807 %
#> 10: SIN clt 2060-10-16 12:00:00 103.125 0.93263 80.29469 %
#> 11: SIN clt 2060-11-16 00:00:00 103.125 0.93263 63.98064 %
#> 12: SIN clt 2060-12-16 12:00:00 103.125 0.93263 88.22657 %
#> 13: SIN hurs 2060-01-16 12:00:00 103.125 0.93263 87.63216 %
#> 14: SIN hurs 2060-02-15 12:00:00 103.125 0.93263 79.58945 %
#> 15: SIN hurs 2060-03-16 12:00:00 103.125 0.93263 76.78159 %
#> 16: SIN hurs 2060-04-16 00:00:00 103.125 0.93263 81.37238 %
#> 17: SIN hurs 2060-05-16 12:00:00 103.125 0.93263 76.57287 %
#> 18: SIN hurs 2060-06-16 00:00:00 103.125 0.93263 80.83648 %
#> 19: SIN hurs 2060-07-16 12:00:00 103.125 0.93263 79.76086 %
#> 20: SIN hurs 2060-08-16 12:00:00 103.125 0.93263 81.98067 %
#> site_id variable_id time lon lat value units
#> <char> <char> <POSc> <num> <num> <num> <char>shift_morph() summarises the extracted monthly climate,
compares it with the baseline EPW, creates morphing factors, and writes
morphed hourly results back to the store. With
strict = TRUE, missing required variables or incomplete
coverage are blocking errors instead of warnings.
shift_morph() wraps the lower-level
EpwMorpher planning and execution API. See EpwMorpher when you need to inspect monthly
summaries, factor diagnostics, case grouping, or custom backend
registration.
For Belcher change-factor morphing, pass either an extracted
historical ShiftClimate stage or an automatic reference
spec such as
shift_reference_historical(reference_periods).
The same shift_recipe used to choose request variables
can be passed into shift_morph(). Adjust the recipe when
you want to change Belcher methods or select another registered
backend:
shift_recipe <- epw_morph_recipe(
"belcher",
methods = c(tdb = "shift", rh = "shift")
)
shift_morph(
extracted,
reference = reference,
baseline = epw,
recipe = shift_recipe,
strict = TRUE
)
morphed <- shift_morph(
extracted,
reference = reference,
baseline = epw,
recipe = shift_recipe,
strict = TRUE
)
morphed
#> <epwshiftr::ShiftMorphed>
#> stage: morphed
#> status: morphed
#> store: <tempdir>/epwshiftr-future-epw-workflow/singapore-store
#> morph: planned
#> cases: 1
#> ids: query_id, plan_id, summary_id, baseline_id, morph_idThe morphed stage is still store-native. It contains
hourly future weather data as Parquet artifacts, not EPW text files yet.
Inspect the artifact rows when you want to see where those intermediate
results live:
morph_artifacts <- shift_artifacts(morphed)
morph_artifacts[, .(kind, role, relative_path)]
#> kind role
#> <char> <char>
#> 1: output derived
#> relative_path
#> <char>
#> 1: outputs/epw-morph/7151ba639a04006fccc4809f2754db0c5111217b4b2083cb7cc8c02bb90875ca/case=449d8e0e3db38f5e39681257d5f5c78d802a31e5bdcaeb40f9045f798938b76e.parquetUse the same shift_data() helper to preview the hourly
morphed weather table. The metadata columns identify the morphing case;
the weather columns are the hourly EPW-style values that will be written
to the final EPW. The preview below omits long IDs to keep the table
readable; include case_id in columns when you
need to join rows back to a specific morphing case.
morphed_data <- shift_data(
morphed,
n = 24L,
columns = c(
"period", "year", "month", "day", "hour",
"dry_bulb_temperature", "relative_humidity", "wind_speed"
)
)
morphed_data
#> period year month day hour dry_bulb_temperature relative_humidity
#> <char> <int> <int> <int> <int> <num> <num>
#> 1: 2060s 2001 1 1 1 23.91930 98.86706
#> 2: 2060s 2001 1 1 2 23.81837 100.00000
#> 3: 2060s 2001 1 1 3 23.71744 100.00000
#> 4: 2060s 2001 1 1 4 23.81837 100.00000
#> 5: 2060s 2001 1 1 5 23.91930 98.86706
#> 6: 2060s 2001 1 1 6 24.22207 96.62008
#> 7: 2060s 2001 1 1 7 24.52485 94.37310
#> 8: 2060s 2001 1 1 8 24.82762 91.00263
#> 9: 2060s 2001 1 1 9 25.23132 87.63216
#> 10: 2060s 2001 1 1 10 25.63503 84.26170
#> 11: 2060s 2001 1 1 11 26.03873 80.89123
#> 12: 2060s 2001 1 1 12 26.34150 78.64425
#> 13: 2060s 2001 1 1 13 26.54335 76.39727
#> 14: 2060s 2001 1 1 14 26.74520 74.15029
#> 15: 2060s 2001 1 1 15 26.74520 74.15029
#> 16: 2060s 2001 1 1 16 26.74520 74.15029
#> 17: 2060s 2001 1 1 17 26.54335 76.39727
#> 18: 2060s 2001 1 1 18 26.34150 78.64425
#> 19: 2060s 2001 1 1 19 26.03873 80.89123
#> 20: 2060s 2001 1 1 20 25.63503 84.26170
#> 21: 2060s 2001 1 1 21 25.23132 87.63216
#> 22: 2060s 2001 1 1 22 24.82762 91.00263
#> 23: 2060s 2001 1 1 23 24.52485 94.37310
#> 24: 2060s 2001 1 1 24 24.22207 96.62008
#> period year month day hour dry_bulb_temperature relative_humidity
#> <char> <int> <int> <int> <int> <num> <num>
#> wind_speed
#> <num>
#> 1: 1.0653853
#> 2: 1.1506162
#> 3: 1.2358470
#> 4: 1.3210778
#> 5: 1.3636932
#> 6: 1.4063087
#> 7: 1.4063087
#> 8: 1.4063087
#> 9: 1.3636932
#> 10: 1.3210778
#> 11: 1.2358470
#> 12: 1.1506162
#> 13: 1.0653853
#> 14: 0.9801545
#> 15: 0.8949237
#> 16: 0.8096929
#> 17: 0.7670774
#> 18: 0.7244620
#> 19: 0.7244620
#> 20: 0.7244620
#> 21: 0.7670774
#> 22: 0.8096929
#> 23: 0.8949237
#> 24: 0.9801545
#> wind_speed
#> <num>shift_epw() writes EnergyPlus Weather files from the
morphed hourly results. It returns a ShiftOutputs stage.
The first chunk assigns the result while hiding verbose writer output;
the second prints the stage object.
For the lower-level write path and output registry, see EpwMorpher.
epws <- shift_epw(morphed)
epws
#> <epwshiftr::ShiftOutputs>
#> stage: outputs
#> status: written
#> store: <tempdir>/epwshiftr-future-epw-workflow/singapore-store
#> outputs: 1
#> ids: query_id, plan_id, summary_id, baseline_id, morph_idshift_outputs() lists the EPW files written by
shift_epw(). These are the files you can pass to EnergyPlus
or open with eplusr::read_epw().
outputs <- shift_outputs(epws)
outputs[, .(path, source_id, experiment_id, variant_label, period)]
#> path
#> <char>
#> 1: outputs/future-epw/MPI-ESM1-2-LR/ssp585/r1i1p1f1/2060s/ae36a1702acd-SGP_Singapore.486980_IWEC.MPI-ESM1-2-LR.ssp585.r1i1p1f1.2060s.epw
#> source_id experiment_id variant_label period
#> <char> <char> <char> <char>
#> 1: MPI-ESM1-2-LR ssp585 r1i1p1f1 2060sshift_data(epws) reads the written EPW file back and
returns its hourly weather data with output metadata attached. This is
useful for confirming that the final file contains the same kind of
hourly weather values you inspected in the store-native morphed Parquet
step. Output metadata such as output_id,
case_id, and path are available in
shift_data(epws); they are omitted here so the weather
values stay visible.
epw_data <- shift_data(
epws,
n = 24L,
columns = c(
"period", "year", "month", "day", "hour",
"dry_bulb_temperature", "relative_humidity", "wind_speed"
)
)
epw_data
#> period year month day hour dry_bulb_temperature relative_humidity
#> <char> <int> <int> <int> <int> <num> <num>
#> 1: 2060s 2001 1 1 1 23.9 98
#> 2: 2060s 2001 1 1 2 23.8 100
#> 3: 2060s 2001 1 1 3 23.7 100
#> 4: 2060s 2001 1 1 4 23.8 100
#> 5: 2060s 2001 1 1 5 23.9 98
#> 6: 2060s 2001 1 1 6 24.2 96
#> 7: 2060s 2001 1 1 7 24.5 94
#> 8: 2060s 2001 1 1 8 24.8 91
#> 9: 2060s 2001 1 1 9 25.2 87
#> 10: 2060s 2001 1 1 10 25.6 84
#> 11: 2060s 2001 1 1 11 26.0 80
#> 12: 2060s 2001 1 1 12 26.3 78
#> 13: 2060s 2001 1 1 13 26.5 76
#> 14: 2060s 2001 1 1 14 26.7 74
#> 15: 2060s 2001 1 1 15 26.7 74
#> 16: 2060s 2001 1 1 16 26.7 74
#> 17: 2060s 2001 1 1 17 26.5 76
#> 18: 2060s 2001 1 1 18 26.3 78
#> 19: 2060s 2001 1 1 19 26.0 80
#> 20: 2060s 2001 1 1 20 25.6 84
#> 21: 2060s 2001 1 1 21 25.2 87
#> 22: 2060s 2001 1 1 22 24.8 91
#> 23: 2060s 2001 1 1 23 24.5 94
#> 24: 2060s 2001 1 1 24 24.2 96
#> period year month day hour dry_bulb_temperature relative_humidity
#> <char> <int> <int> <int> <int> <num> <num>
#> wind_speed
#> <num>
#> 1: 1.1
#> 2: 1.2
#> 3: 1.2
#> 4: 1.3
#> 5: 1.4
#> 6: 1.4
#> 7: 1.4
#> 8: 1.4
#> 9: 1.4
#> 10: 1.3
#> 11: 1.2
#> 12: 1.2
#> 13: 1.1
#> 14: 1.0
#> 15: 0.9
#> 16: 0.8
#> 17: 0.8
#> 18: 0.7
#> 19: 0.7
#> 20: 0.7
#> 21: 0.8
#> 22: 0.8
#> 23: 0.9
#> 24: 1.0
#> wind_speed
#> <num>At this point the workflow has produced EPW files, but there are still a few checks worth doing before using them in EnergyPlus or passing them to someone else. These checks answer three practical questions:
Use shift_status() when you want a compact stage-level
check. It returns a single status string so it can be used in scripts,
reports, or simple guards. For a successful run, the sequence should end
with an EPW stage marked written.
data.table::data.table(
stage = c("request", "collect", "extract", "reference", "morph", "epw"),
status = c(
shift_status(request),
shift_status(files),
shift_status(extracted),
shift_status(reference),
shift_status(morphed),
shift_status(epws)
)
)
#> stage status
#> <char> <char>
#> 1: request new
#> 2: collect collected
#> 3: extract extracted
#> 4: reference extracted
#> 5: morph morphed
#> 6: epw writtenDiagnostics are the first place to look when a stage is blocked, failed, or returns fewer outputs than expected. An empty diagnostics table is the normal successful result. Non-empty rows are intended to explain the stage, severity, and action rather than expose internal manifest IDs first.
For common causes and the first place to look for each class of failure, see ESGF troubleshooting.
shift_diagnostics(epws)
#> Empty data.table (0 rows and 16 cols): stage,severity,code,message,query_id,session_id...The final EPW depends on the extracted climate table. Even after the EPW file is written, it is useful to confirm that each morphing variable has complete monthly coverage for the requested period. Missing or incomplete rows here usually mean the original ESGF query, Dataset selection, or extraction period needs to be adjusted.
Coverage problems usually originate in query selection, time filtering, or remote access. The ESGF troubleshooting article collects those checks in one place.
coverage <- shift_coverage(epws)
coverage[, .(variable_id, complete, status, output_rows, output_file_count)]
#> variable_id complete status output_rows output_file_count
#> <char> <lgcl> <char> <int> <int>
#> 1: rlds TRUE done 12 1
#> 2: rsds TRUE done 12 1
#> 3: psl TRUE done 12 1
#> 4: tas TRUE done 12 1
#> 5: clt TRUE done 12 1
#> 6: sfcWind TRUE done 12 1
#> 7: hurs TRUE done 12 1shift_outputs() lists the written EPW files and their
case metadata. These are the files to pass to EnergyPlus, archive with a
simulation run, or reopen with eplusr::read_epw(). The
store also keeps the intermediate Parquet artifacts so the workflow can
be inspected or reused without repeating the remote query.
outputs <- shift_outputs(epws)
outputs[, .(
file = basename(path),
source_id,
experiment_id,
variant_label,
period
)]
#> file
#> <char>
#> 1: ae36a1702acd-SGP_Singapore.486980_IWEC.MPI-ESM1-2-LR.ssp585.r1i1p1f1.2060s.epw
#> source_id experiment_id variant_label period
#> <char> <char> <char> <char>
#> 1: MPI-ESM1-2-LR ssp585 r1i1p1f1 2060s