NEWS.md
nearest count option with explicit grid extraction methods: "nearest", "idw", "bilinear", and "mean" (#123).v0.1.4 behavior should use the legacy branch on GitHub or install epwshiftr v0.1.4.init_cmip6_index(), summary_database(), match_coord(), extract_data(), morphing_epw(), future_epw(), EsgfQuery, esgf_query(), and related helpers. Use shift_*, EsgStore, EpwMorpher, EsgQuery, and data_node_status() instead.fst, future.apply, progressr, pingr, PCICt, and rappdirs-based implementation paths; the new implementation uses DuckDB, mirai, S7, and store-managed manifests.shift_request() -> shift_collect() -> shift_download() -> shift_extract() -> shift_morph() -> shift_epw() workflow, with inspection helpers such as shift_status(), shift_diagnostics(), shift_coverage(), shift_outputs(), and shift_data().EsgStore, a DuckDB-backed local store for query snapshots, dictionaries, source files, downloads, Parquet extracts, and generated EPW outputs.esg_query() / EsgQuery and typed EsgResult* objects for Dataset, File, and Aggregation records.EsgDataset for remote OPeNDAP NetCDF access without downloading full files.EsgDict objects with option discovery and legality checks across CMIP6 and related ESG projects.EpwMorpher, morphing recipes, backend registration, historical reference handling, resumable morphing, and manifest-backed EPW output registration (#111, #115).pr) and includes it in the recommended morphing variable set (#122).epwshiftr CLI with doctor, query, download, storage, shift, extract, morph, and esgf command groups (#114).dist extraction column after grid extraction methods became explicit (#124).EsgStore now keeps ESGF query collection and downloader operations outside the store manifest lock, reducing lock hold time during query update and download workflows (#120).EsgQuery$collect(all = TRUE) now warns and returns partial results when ESGF pagination stops making progress, instead of repeatedly requesting the same offset (#116).shift_epw() and EpwMorpher$write_epw() now fill missing, out-of-range, and special EPW values before saving generated weather files (#87).shift_* workflow and the store-native migration path.limit handling and progress labels (#119).EsgResultFile and EsgResultAggregation now share their internal download and OPeNDAP fallback helpers while keeping the public methods unchanged (#118).match_coord() has been refactored to correct the calculation method of geographical distance. Previously, epwshiftr assumes that distance on longitude and latitude is the same which is not true. Now it uses a spheroid formula to calculate the tunnel distance (#39). For details, please see Tunnel Distance . The structure of the returned epw_cmip6_coord object has also changed. The coord column in the coord data.table is also a data.table which contains 6 columns describing the matched coordinates:
index: the indices of matched coordinatesind_lon, ind_lat: The value indices of longitude or latitude in the NetCDF coordinate grids. These values are used to extract the corresponding variable valueslon, lat: the actual longitude or latitude in the NetCDF coordinate gridsdist: the distance in km between the coordinate values in NetCDF and input EPWBesides, the usage of the input threshold and max_num has been changed a little bit:
threshold: Due to the change in distance calculation, the meaning of the threshold input has been changed. Instead of directly being used to get the ‘closest’ grid points in NetCDF, the longitude and latitude threshold is only used to help exclude grid points that are definitely too far away from the target location. The default threshold, which is 1 degree for both longitude and latitude, is still reasonable for common use cases and is kept unchanged. Also threshold now can be set to NULL. In this case, the distances between the target location and all grid points will be calculated. But this may be only useful for rare cases.max_num: Now the value max_num is the key input to control how many grid points are to be matched. The points will always be ordered in descending order in terms of the distances.The data in the returned value of extract_data() has been updated to include a new column dist which gives the spherical distance in km between EPW location and grid coordinates (#39).
The document on the return value structure for extract_data() and morphing_epw() has been fixed (#29). And the column order for all metadata in the returned data.table from extract_data() and morphing_epw() are not consistent. The columns will always be in the order below (#45):
activity_drsinstitution_idsource_idexperiment_idmember_idtable_idlonlatdistfull is added to future_epw(). When setting to TRUE, a data.table containing information about how the data are split by the by argument and also the generated future EPWs and their paths are returned (#18).summary_database() can append results to the previous scan and detect if any previously matched NetCDF files do not exist. It stores the metadata of those missing files as a new attribute not_found in the results. Warnings are issued if warning is set to TRUE. Also, a new parameter miss has been added to control how to manage those invalid entries. You can set it to "keep", which is the default, to do nothing about it or "overwrite" to overwrite those entries based on the newly matched NetCDF files if possible (#40).extract_data() now supports non-standard calendar, e.g. 360_day (#32).warning has been added in morphing_epw(). If set to TRUE, warnings will be issued for cases with input data less than a decade (10 years) . This is because using data that only covers a short period of time may not be able to capture the average of future climate (#41).EsgDict class. Please see ?EsgDict for details (#53).epwshiftr.threshold_alpha has been added to set the threshold of the absolute value for alpha, i.e. monthly-mean fractional change when performing morphing operations. The default value is set to 3. If the morphing methods are set "stretch" or "combined", and the absolute alpha exceeds the threshold value, warnings are issued and the morphing method is fallbacked to "shift" to avoid unrealistic morphed values (#54).replica can be NULL in esgf_query() and init_cmip6_index(). In this case, both the master record and replicas are all returned (#61).EsgfQuery is added to support more flexible APIs for ESGF search services. Please see ?EsgfQuery for details (#63, #69).summary_database() when no NetCDF files are found in the input directory (#25).epw in match_coord() is a search string (#25).morphing_epw() can correctly fall back to use “Shift” method when any missing values are detected in maximum and minimum prediction values of climate variables (#25).combined method is used in morphing_epw() (#25).get_data_node() works again (#80)LazyData in DESCRIPTION (#16).