The Earth System Grid Federation (ESGF) is an international collaboration for the software that powers most global climate change research, notably assessments by the Intergovernmental Panel on Climate Change (IPCC).
The ESGF search service exposes RESTful APIs that can be used by clients to query the contents of the underlying search index, and return results matching the given constraints. The documentation of the APIs can be found using this link.
EsgQuery is the workhorse for dealing with ESGF search services.
Start with esg_query() / EsgQuery for new workflow code. The legacy
data.table-oriented API is available from the legacy branch or v0.1.4.
esg_query(index_node = "https://esgf-node.ornl.gov")The URL to the ESGF Index Node. Default is to use the ORNL (Oak Ridge National Laboratory) Index Node. Current possible values could be:
ORNL (Oak Ridge National Laboratory), USA:
https://esgf-node.ornl.gov. The default value.
LLNL (Lawrence Livermore National Laboratory), USA:
https://esgf-node.llnl.gov
NCI (National Computational Infrastructure), Australia:
https://esgf.nci.org.au
IPSL (Institut Pierre-Simon Laplace), France:
https://esgf-node.ipsl.upmc.fr
DKRZ (Deutsches Klimarechenzentrum), Germany:
https://esgf-data.dkrz.de
LIU (National Academic Infrastructure for Supercomputing), Sweden:
https://esg-dn1.nsc.liu.se
CEDA (Centre for Environmental Data Analysis), UK:
https://esgf.ceda.ac.uk
For bridge index nodes, only predefined common facets are returned.
$list_fields() can be used to get all available fields,
including facets.
EsgQuery objectesg_query() returns an EsgQuery object, which is an R6
object with quite a few methods that can be classified into 3 categories:
Value listing: methods to list all possible values of facets, fields, shards, and values.
Parameter getter & setter: methods to get the query parameter values or set them before sending the actual query to the ESGF search services.
Query responses: methods to collect results for the query response.
EsgQuery object provides the following value-listing methods to query
available facets, fields, shards, and values from the ESGF index node:
EsgQuery$list_facets():
List all available facet names. When called, a
facet listing query
is sent to the index node to get all available facets for the current
project (default: CMIP6).
EsgQuery$list_fields():
List all available field names. This is useful for bridge index nodes
where facet listing is not available.
EsgQuery$list_shards():
List all available shards (ESGF index nodes) that can be queried in
distributed searches.
EsgQuery$list_values():
List all available values of specific facets.
The ESGF search services support a lot of parameters. The EsgQuery
contains dedicated methods to set values for most of them, including:
Most common keywords:
facets,
offset,
limit,
fields,
replica,
latest,
distrib
and
shards.
Most common facets:
project,
activity_id,
experiment_id,
source_id,
variable_id,
frequency,
variant_label,
nominal_resolution,
datetime_range,
timestamp_range,
version_range
and
data_node.
All methods act in a similar way:
If input is given, the corresponding parameter is set and the updated
EsgQuery object is returned.
This makes it possible to chain different parameter setters, e.g.
EsgQuery$project("CMIP6")$frequency("day")$limit(1) sets the parameter
project, frequency and limit sequentially.
For parameters that want character inputs, you can put a preceding ! to
negate the constraints, e.g. EsgQuery$project(!"CMIP6") searches for
all projects except for CMIP6.
If no input is given, the current parameter value is returned. For example,
directly calling EsgQuery$project() returns the current value of the
project parameter. The returned value can be two types:
NULL, i.e. there is no constraint on the corresponding parameter
A QueryParam object. Use query_param__value() and
query_param__negate() to inspect it.
Despite methods for specific keywords and facets, you can specify arbitrary
query parameters using
EsgQuery$params() method. For
details on the usage, please see the
documentation.
The query is not sent unless related methods are called:
EsgQuery$count(): Count the total
number of records that match the query.
You can return only the total number of matched record by calling
EsgQuery$count(facets = FALSE)
You can also count the matched records for specified facets, e.g.
EsgQuery$count(facets = c("source_id", "activity_id"))
EsgQuery$collect(): Collect the
query results and format it into an EsgResultDataset object.
Some ESGF index nodes are "bridge" nodes that have certain limitations
compared to standard index nodes. When using a bridge index node (e.g.,
https://esgf-node.ornl.gov/esgf-1-5-bridge), the following restrictions
apply:
The fields parameter is not supported. All available fields are always
returned.
Only Dataset and File queries are supported. Aggregation queries
should use a standard ESGF search index node, such as
https://esgf-data.dkrz.de or https://esgf.ceda.ac.uk.
The retracted parameter is not supported and will be ignored.
Wget script generation is not supported. Calling $url(wget = TRUE) will
result in an error.
Facet listing is not available. $list_facets() will return a predefined
set of common facets instead. Use $list_fields() to get all available
fields.
EsgQuery object also provides several other helper functions:
Query URL generation:
EsgQuery$url(): Returns the actual
query URL or the wget script URL which can be used to download all files
matching the given constraints.
State persistence:
EsgQuery$save(): Save the query
state to a JSON file for later use.
EsgQuery$load(): Restore the
query state from a JSON file created by $save().
Display:
EsgQuery$print(): Print a
summary of the current EsgQuery object including the index node URL
and all query parameters.
new()Create a new EsgQuery object
EsgQuery$new(index_node = "https://esgf-node.ornl.gov")index_nodeThe URL to the ESGF Index Node. Default is to use the ORNL (Oak Ridge National Laboratory) Index Node. Current possible values could be:
ORNL (Oak Ridge National Laboratory), USA:
https://esgf-node.ornl.gov. The default value.
LLNL (Lawrence Livermore National Laboratory), USA:
https://esgf-node.llnl.gov
NCI (National Computational Infrastructure), Australia:
https://esgf.nci.org.au
IPSL (Institut Pierre-Simon Laplace), France:
https://esgf-node.ipsl.upmc.fr
DKRZ (Deutsches Klimarechenzentrum), Germany:
https://esgf-data.dkrz.de
LIU (National Academic Infrastructure for Supercomputing), Sweden:
https://esg-dn1.nsc.liu.se
CEDA (Centre for Environmental Data Analysis), UK:
https://esgf.ceda.ac.uk
index_node()Get or set the ESGF index node.
$index_node() returns the current normalized index node URL.
$index_node(value) updates the index node after applying the same
normalization used by EsgQuery$new().
Existing query parameters are kept unchanged.
list_facets()List all available facet names
list_fields()List all available field names
list_shards()List all available shards.
list_values()List all available values of specific facets.
facetsA character vector giving the facet names.
forceBy default, every value listing query is cached and
reused when possible. If TRUE, the previous cache is
abandoned and a new query is re-sent and cached. Default:
FALSE.
project()Get or set the project facet parameter.
activity_id()Get or set the activity_id facet parameter.
experiment_id()Get or set the experiment_id facet parameter.
source_id()Get or set the source_id facet parameter.
variable_id()Get or set the variable_id facet parameter.
frequency()Get or set the frequency facet parameter.
variant_label()Get or set the variant_label facet parameter.
nominal_resolution()Get or set the nominal_resolution facet parameter.
data_node()Get or set the data_node facet parameter.
fields()Get or set the fields parameter.
datetime_range()Get or set temporal coverage overlap constraints.
start, stopTemporal boundary strings accepted by solr_date(),
complete Solr range expressions, "*", or NULL. If both are
omitted, the current range state is returned. The helper renders
Solr constraints for the ESGF REST start/end temporal
coverage keyword semantics.
timestamp_range()Get or set Solr index timestamp range constraints.
from, toTimestamp boundary strings accepted by solr_date(),
"*", or NULL. Complete Solr range expressions are not
accepted here. If both are omitted, the current range state is
returned.
version_range()Get or set version range constraints.
min, maxVersion boundaries such as 20200101, "20200101",
simplified dates, "*", or NULL. ESGF version is queried
as a numeric field; simplified date inputs are normalized to
comparable YYYYMMDD integer boundaries before rendering.
Solr Date Math and complete range expressions are not accepted
here. If both are omitted, the current range state is returned.
latest()Get or set the latest parameter.
limit()Get or set the limit parameter.
offset()Get or set the offset parameter.
distrib()Get or set the distrib parameter.
params()Get or set ad hoc query parameters.
$params() handles parameters without dedicated methods and can also
update supported dedicated parameters by name.
The type and format control parameters cannot be changed here:
EsgQuery always performs Dataset queries and always parses JSON
responses. Use EsgResultDataset$collect() to collect File or
Aggregation records from Dataset results.
url()Get the URL of actual query or wget script
The wget script URL can be used to download a bash script that contains wget commands for downloading all files matching the query constraints. This is useful for batch downloading large amounts of data.
wgetWhether to return the URL of the wget script that can be
used to download all files matching the given constraints.
Default: FALSE.
\dontrun{
q$url()
# get the wget script URL
q$url(wget = TRUE)
# You can download the wget script using the URL directly. For
# example, the code below downloads the script and save it as
# 'wget.sh' in R's temporary folder:
download.file(q$url(TRUE), file.path(tempdir(), "wget.sh"), mode = "wb")
}
count()Send a query of facet counting and fetch the results
facetsNULL, a flag or a character vector. There are three
options:
If NULL or FALSE, only the total number of matched records is
returned.
If TRUE, the value of $facets()
is used to limit the facets. If $facets() returns NULL, only the
total count is returned. This is the default value.
If a character vector, it is used to limit the facets.
collect()Send the actual query and fetch the results
$collect() sends the actual query to the ESGF search services.
By default it collects type=Dataset results and returns an
EsgResultDataset object. If type is "File" or
"Aggregation", it first collects matching Dataset results and then
collects child File or Aggregation results for those datasets.
The fields included depend on fields parameter.
However, the following fields are always included in the results:
access, data_node, id, index_node, instance_id, latest, master_id, number_of_aggregations, number_of_files, replica, size, url, version.
When a local EsgDict is available for the query project, $collect()
also performs a warning-only dictionary check before sending the query.
Missing local dictionaries are ignored and never downloaded.
EsgQuery$collect(
all = FALSE,
limit = TRUE,
params = TRUE,
type = "Dataset",
fields = NULL,
progress = getOption("epwshiftr.progress", interactive()),
...
)allWhether to collect all results despite of the value of
offset. Default: FALSE.
limitIf all = FALSE, the maximum number of records to
collect in this request. If all = TRUE, the page size used
for each paginated request, not a total cap. When all = TRUE
and limit = TRUE, the current query limit value is used;
if limit = FALSE, the allowed maximum limit number
10000 is used. It can also be a positive
integer used as a temporary page size. Default: TRUE.
paramsWhether to include facet fields that have parameter
constraints explicitly set using EsgQuery$project(),
EsgQuery$activity_id(), EsgQuery$params() and etc. in the
returned fields. For example, if you set $experiment_id("ssp585"),
the experiment_id field will be included in the results when
params = TRUE. Default: TRUE.
typeResult type to collect. One of "Dataset", "File",
or "Aggregation". Default: "Dataset".
fieldsOptional fields used only when type is "File" or
"Aggregation". Dataset fields should be configured with
$fields() before collecting.
progressWhether to show a progress bar while collecting ESGF
JSON search pages. By default, the value of option
epwshiftr.progress is used, falling back to interactive().
...Arguments passed to EsgResultDataset child collection
when type is "File" or "Aggregation", including the
data_node scope filter and child-query controls.
File/Aggregation collection does not use ESGF datetime search
parameters; use $filter_time() on the returned result for
time filtering.
An EsgResultDataset, EsgResultFile, or EsgResultAggregation object.
\dontrun{
# by default, all fields with constrains are included in the results
query <- esg_query()$experiment_id("ssp585")$frequency("1hr")$fields("source_id")
res1 <- query$collect()
res1$fields
# set `params` to `FALSE` to exclude them
query$collect(params = FALSE)$fields
# collect all matched records with `query$limit()` records per query
res2 <- query$collect(all = TRUE, limit = TRUE)
identical(query$count(), res2$count())
# same as above, but collect all matched records with max allowed
# record limit per query
res3 <- query$collect(all = TRUE, limit = FALSE)
identical(res2$count(), res3$count())
# same as above, but collect all matched records with specified limit
# per query
res4 <- query$collect(all = TRUE, limit = 30)
identical(res2$count(), res4$count())
}
state()Get the current query state.
$state() returns a read-only snapshot containing the current index
node and the current parameter state.
reset()Reset query parameters to their defaults.
$reset() clears the current parameter store and restores the default
query parameters. The current index node is kept unchanged.
save()Save the query into a JSON file
$save() puts main data of an EsgQuery object into a JSON file
which can be loaded to restore the current state of query using
EsgQuery$load().
fileA string indicating the JSON file path to save the data to.
prettyWhether to add indentation whitespace to JSON output.
For details, please see jsonlite::toJSON(). Default: TRUE.
load()Restore the query state from an JSON file
$load() reads data of an EsgQuery object from a JSON file
created using
EsgQuery$save().
print()Print a summary of the current EsgQuery object
$print() gives the summary of current EsgQuery object including
the index node URL and all query parameters.
## ------------------------------------------------
## Method `EsgQuery$new`
## ------------------------------------------------
if (FALSE) { # \dontrun{
q <- EsgQuery$new(index_node = "https://esgf-node.ornl.gov")
q
} # }
## ------------------------------------------------
## Method `EsgQuery$index_node`
## ------------------------------------------------
if (FALSE) { # \dontrun{
q$index_node()
q$index_node("https://esgf.ceda.ac.uk")
} # }
## ------------------------------------------------
## Method `EsgQuery$list_facets`
## ------------------------------------------------
if (FALSE) { # \dontrun{
q$list_facets()
} # }
## ------------------------------------------------
## Method `EsgQuery$list_fields`
## ------------------------------------------------
if (FALSE) { # \dontrun{
q$list_fields()
} # }
## ------------------------------------------------
## Method `EsgQuery$list_shards`
## ------------------------------------------------
if (FALSE) { # \dontrun{
q$list_shards()
} # }
## ------------------------------------------------
## Method `EsgQuery$list_values`
## ------------------------------------------------
if (FALSE) { # \dontrun{
q$list_values(c("activity_id", "experiment_id"))
} # }
## ------------------------------------------------
## Method `EsgQuery$url`
## ------------------------------------------------
if (FALSE) { # \dontrun{
q$url()
# get the wget script URL
q$url(wget = TRUE)
# You can download the wget script using the URL directly. For
# example, the code below downloads the script and save it as
# 'wget.sh' in R's temporary folder:
download.file(q$url(TRUE), file.path(tempdir(), "wget.sh"), mode = "wb")
} # }
## ------------------------------------------------
## Method `EsgQuery$count`
## ------------------------------------------------
if (FALSE) { # \dontrun{
# get the total number of matched records
q$count(NULL) # or q$count(facets = FALSE)
# count records for specific facets
q$facets(c("activity_id", "source_id"))$count()
# same as above
q$count(facets = c("activity_id", "source_id"))
} # }
## ------------------------------------------------
## Method `EsgQuery$collect`
## ------------------------------------------------
if (FALSE) { # \dontrun{
# by default, all fields with constrains are included in the results
query <- esg_query()$experiment_id("ssp585")$frequency("1hr")$fields("source_id")
res1 <- query$collect()
res1$fields
# set `params` to `FALSE` to exclude them
query$collect(params = FALSE)$fields
# collect all matched records with `query$limit()` records per query
res2 <- query$collect(all = TRUE, limit = TRUE)
identical(query$count(), res2$count())
# same as above, but collect all matched records with max allowed
# record limit per query
res3 <- query$collect(all = TRUE, limit = FALSE)
identical(res2$count(), res3$count())
# same as above, but collect all matched records with specified limit
# per query
res4 <- query$collect(all = TRUE, limit = 30)
identical(res2$count(), res4$count())
} # }
## ------------------------------------------------
## Method `EsgQuery$state`
## ------------------------------------------------
if (FALSE) { # \dontrun{
q$state()
q$state(null = TRUE)
} # }
## ------------------------------------------------
## Method `EsgQuery$reset`
## ------------------------------------------------
if (FALSE) { # \dontrun{
q$experiment_id("ssp585")$reset()
} # }
## ------------------------------------------------
## Method `EsgQuery$save`
## ------------------------------------------------
if (FALSE) { # \dontrun{
q$save(tempfile(fileext = ".json"))
} # }
## ------------------------------------------------
## Method `EsgQuery$load`
## ------------------------------------------------
if (FALSE) { # \dontrun{
f <- tempfile(fileext = "json")
q <- esg_query()
json <- q$save(f)
q$load(f)
} # }
## ------------------------------------------------
## Method `EsgQuery$print`
## ------------------------------------------------
if (FALSE) { # \dontrun{
q$print()
} # }