22-3 Import from Delft-FEWS Open Archive

Introduction

Data can be imported back into FEWS in two different ways: 

• On an Operator Client, using seamless integration - in this case the archived data is automatically displayed in the client and visible for a user
• On a Stand Alone Application using a module configuration file -  the ImportArchiveModule

This page describes how data is imported and what configuration files need to be set up. Once downloaded, the data can be ingested by a Delft-FEWS stand alone with the aim to bring the local datastore back in the 'original state', i.e. with a similar time series set definition as used when Delft-FEWS ran the workflow to produce the data. What is different compared to a normal Delft-FEWS import, is that importArchiveModule reads the timeseries definition from the Archive NetCDF files during the import. Every archived FEWS NetCDF file, contains the exact timeseries definition of the data. How it was stored in the FEWS database before exported. The importArchiveModule has been implemented to uses this definition included in the netCDF files to put the data back in exactly the same way in a Delft-FEWS local datastore

IdMapping

By default it is not necessary to configure an idMap for the archive import. The variables containing time series in the netCDF files in the open archive have attributes describing the timeSeriesSet definition, which will allow Delft-FEWS to automatically import the data back into the datastore.

It is possible to define an idMapping for the archive import. This id mapping can only be used for mapping location ids and parameters ids. Note that the location id or parameter id defined in the xml doesn't have to be available anymore in FEWS. 

Note that for versions older than 2025.02, the variables in the netCDF file contained an attribute called timeseries_sets_xml. This attribute contained an xml file with the original timeseries definition. When the netCDF file is imported the time series in the variable will be mapped to the definition defined here. 

Configuration of historical events import (FSS)

Historic events are a special datatype to Delft-FEWS as they can be used to overlay on an existing time series graph. Historic events are the only datasets which can be ingested in a Delft-FEWS client server system. Normally, the archive server has a process running (the HistoricEventsExporter) which extracts historic events data from the archive and pushes them to the Forecasting Shell Server import directories for ingest in the operational database.

This import process has its own data administration process, and does not use the timeseries definition as embedded in the netcdf files. Hence an idMap may be needed to translate netcdf variables into Delft-FEWS parameters and locations. In addition, backup and failure folders can be defined to prevent loss of data in the automated process.

Figure 4.17 and Table 4.11 describe the configuration for importing historic events. This module should be executed in a separate workflow compared to the other archived datasets.

Figure 4.17 Configuration of historical events import (importArchiveModule.xsd)

Table 4.11 Historical events import configuration (importArchiveModule)

Import data from the Open Archive in a workflow

It is possible to schedule a workflow which imports data from the archive based on a list of data sets which is provided in a file. Below an example of the configuration of such a workflow.

<importArchiveModule xsi:schemaLocation="http://www.wldelft.nl/fews https://fewsdocs.deltares.nl/schemas/version1.0/importArchiveModule.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
					 xmlns="http://www.wldelft.nl/fews">
	<importRequestedDataSets>
		<importFolder>$IMPORT_FOLDER$</importFolder>
		<dataFolderArchive>$ARCHIVE_DOWNLOAD_FOLDER$</dataFolderArchive>
	</importRequestedDataSets>
</importArchiveModule>

The importFolder tag should define the directory which contains the files which define which datasets should be imported. The import folder is by default always the subfolder dataImport of the root of the archive.

The dataFolderArchive tag defines the location of the root of the data folder of the archive. 

Below an example of how the files which define which datasets should be imported should look like

<dataSets xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.wldelft.nl/fews/archive" xsi:schemaLocation="http://www.wldelft.nl/fews/archive http://fews.wldelft.nl/schemas//version1.0/archive-schemas/dataImport.xsd">
    <dataSet>dataSet1\metaData.xml</dataSet>
    <dataSet>dataSet2\metaData.xml</dataSet>
</dataSets>

In the Open Archive each dataset has its own metaData.xml file. The import task will import each dataset which is defined in the list entirely. The import folder can contain multiple files. If that is the case the files will be imported one by one in a random order.

The files can be provided by writing them directly into the defined import folder. Another option is to upload the files. For this a dedicated webservice is available in the archive. 

The webservice is available at the following relative URL at the archive web application /dataSetImportRequest/upload/. The file can be uploaded by using a post request.

Use data from the Open Archive in a Workflow

Seamless integration while running a workflow, see

• 06 Workflow Configuration for the use (and limitations) of <downloadMissingDataFromArchive> and
• Archives for the features and limitations of seamless integration

Using the Open Archive in Delft-FEWS

The Archive Catalogue, is a display that is used to view and download data from Deltares Open Archive to Delft-FEWS. The following page describes how to use the open archive in Delft-FEWS: Delft-FEWS Archive Display. This page contains a lot of information with practical examples on how to use the open archive. Below are a few other use-cases where data is imported.

• Searching for archived forecasts in the Data Display
• Viewing archived historical events in the Data Display
• Viewing data with Seamless Integration in the Data Display