This memo describes application of the module adapter to allow integration with Delft-Fews through use of the published interface format interchange specification.
The objective of the memo is twofold:
- to provide a description of the preferred approach in developing module adapters, and,
- to illustrate the approach through an example taken from the Northeast region.
The document SIS Module Adapter Specification, Version 2.4 describes the use of the general adapter in allowing modules to be run from within Delft-Fews. The general adapter (this module is a part of Delft-Fews) is configured to provide the required data (both dynamic and static) in the published interface format. A module adapter (supplied by the same supplier as the module itself) is used to translate the data from the published interface format to the native module format.
Figure: Schematic interaction between the General Adapter and the Module adapter through the published interface
The general adapter is as stated an integral part of Delft-Fews. This adapter is used to link all third party modules required to make a forecast in a configured forecast system. To allow this, each instance of the general adapter linking Delft-Fews with a forecasting module is configured as required to provide the data inputs to the module, run the module and retrieve the data outputs from the module.
The preferred approach in running a module within Delft-Fews is in three steps:
Step 1: Export of data required by the module through the published interface from the Delft-Fews database to the module native format. This data can cover dynamic time series data, parameter value sets, module states etc. A full description of the data types supported is given in SIS Module Adapter Specification, Version 2.0. To determine what data is exported to a given module, a configuration file (XML formatted) is passed to the General Adapter as an argument. This file is a part of the Delft-Fews and is configured during set-up of a forecasting system. After the general adapter has run the input files required by the module are available in the published interface format. This can entail time series, module states etc. Figure 3 shows a schematic description of this first step. The module adapter must provide on completion an XML formatted diagnostic file giving the general adapter information on the status of the translation process. For the format of this file and associated enumeration see the published interface specification.
Figure: Detailed view of export of data from Delft-Fews to module native format through the published Interface (PI)
Step 2: In the second step the module itself is run. This is achieved by the general adapter executing a call to the module executable, with the possible addition of command line arguments. This module executable must be able to run in batch mode, without any user interaction required. The module executable reads the required input data, parameters and states in its native format, performs the required calculation and establishes a set of output files and states, again in the native module format. This set of output files must include a file giving diagnostic information on the module run. This file can be in the native module format. Figure 4 gives a detailed schematic overview of concept of running a module executable within Delft-Fews.
Figure: Detailed view of running a module executable from Delft-Fews
Step 3: The third step comprises the import of module output data into Delft-Fews. Again this is using the published interface format. The module adapter is called to transform native module output formats to the published interface format. Once completed the output data is retrieved through the published interface format for insertion into the Delft-Fews central database. As with the module inputs an XML formatted configuration file is used to determine what data are imported.
Figure Detailed view of import of data to Delft-Fews from module native format through the published Interface (PI)_
The three steps make it clear that there are three functional elements to be provided by a module supplier to allow coupling of a module with Delft-Fews. This functionality may be encompassed in either two or three separate executable.
Element |
Function |
Configuration |
Comment |
Module Adapter |
Import data from Published Interface format to Native module format |
Configuration file specifying data to import. Preferred format of file: XML |
|
Module executable |
Run module using data in native module input format. Write output in native module output format |
|
|
Module Adapter |
Exports data from native module format to published interface format |
Configuration file specifying data to export. Preferred format of file: XML |
|
Example: Kinematic wave routing at HEBDEN bridge
As an example a comparison is made in this memo of the ICA model component file for the forecast at Hebden Bridge on the Calder River in Northeast region. Although some understanding of the structure of the ICA is helpful in this comparison, it seems that developing an example from the ICA is most appropriate given the complexity of the ICA. The full ICA module component file is included in the Appendix for reference.
Description of current approach as implemented in ICA:
Two data series are forecast for the site at Hebden Bridge. These are the water level and the discharge (described in the ICA in the forecast requirement files lu-hebdbr1.rffs and qx-hebdbr1.rffs ). The discharge at the site is first calculated using the Kinematic Wave model (supplied by CEH) for the reach upstream of Hebden Bridge.
There are two inputs to the kinematic wave model, being the discharges from the Walsden sub-catchment and the Todmorden sub-catchment (ICA forecast requirements QX-WALSDN1 and QX-TODMDN1).
These two inputs are routed using the KW model to form a temporary output discharge series at Hebden Bridge (Series name in ICA Model Component File: 2Q-HEBDBR1).
In the next step, the level series available in the database for the historic period (ICA code LU-HEBDBR1) is transformed to a discharge series using the applicable stage discharge curve (ICA: RATING algorithm). This supplies for the historic period an "observed" discharge series.
Using this "observed" discharge series, an ARMA error modelling step is carried out. This uses as input the temporary discharge calculated by the KW model and the "observed" discharge series. Using the ARMA procedure the corrected output series for Hebden Bridge is derived (ICA: QX-HEBDBR1) for the new forecast period.
In the final step levels for the forecast period are derived using this forecast discharge series again through the applicable rating curve (ICA: RATING algorithm).
Data series naming conventions in Delft-Fews
In Delft-Fews, the naming convention is a little different from that applied in the ICA. The ID's with which series are identified are constructed on the basis of the (unique) station code. Data series available at that station are then identified by the station code plus a data type suffix. Clear distinction is made through parameter types of series available for the historic period and for the forecast period. The series ID is used as a unique identifier in locating data series. Each series does have a "name" that is used in all displays etc to enhance readability.
Description |
ICA Series |
Delft-Fews series |
Delft-Fews series |
Input discharge series |
QX-WALSDN1 |
H-27392-Q.hc |
H-27392-Q.fc |
Input discharge series |
QX-TODMDN1 |
H-27931-Q.hc |
H-27931-Q.fc |
Output discharge at Hebden Bridge |
2Q-HEBDBR1 |
H-27932-Q.hr |
H-27932-Q.fr |
Output discharge at Hebden Bridge |
QX-HEBDBR1 |
H-27932-Q.uhr |
H-27932-Q.ufr |
Observed Level at Hebden Bridge |
LU-HEBDBR1 |
H-27932-h.m |
Not Applicable |
Forecast Level at Hebden Bridge |
LU-HEBDBR1 |
H-27932-h.hr |
H-27932-h.fr |
Note: in the ICA the observed and modelled series for the historical period seem to occupy the same database location. These two are indeed the same value in case an ARMA error correction procedure is used with a parameterisation including the auto-regressive component.
Delft-Fews Workflow for forecast at Hebden Bridge
In Delft-Fews a series of tasks such as that required in creating the forecast at Hebden Bridge are configured in a Task File. Each of the tasks required is performed through either a standard Delft-Fews utility (e.g. validation, interpolation, error modelling) or an external module. The external module is called through the general adapter.
The example below illustrates the workflow definition for the forecast at Hebden Bridge (all file names are as an example, and the XML structure is illustrative but not definite). The sequence of steps to deliver the forecast requirement at Hebden Bridge is the same as specified in the ICA.
Workflow files such as this are configured as a part of Delft-Fews, not by any third party module suppliers. The first element uses the General Adapter to call the Kinematic Wave module. The configuration file is used by the General Adapter to direct how this module is used. An example of the configuration file for this element is given in the Appendix. Again this configuration file is configured during Delft-Fews configuration, not by the third party module supplier.
<?xml version="1.0" encoding="UTF-8"?> \\ <workflow xmlns="http://www.wldelft.nl/fews" xmlns:target="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews workflow.xsd" version="1.0"> <sequence> <!-- Run KW for reach to Hebden Bridge (using General Adapter)--> <element name="kw.hebdbr"> <caption>KW run to Hebden Bridge"</caption> <class>nl.wldelft.fews.util.GeneralAdapter</class> <configuration>kw_hebdbr_historical.xml</configuration> <auto>"true"</auto> <output>"rating.hebdbr.observed</output> </element> <!-- Transform observed levels to discharges at Hebden Bridge (using ConversionServer)--> <element name="rating.hebdbr.observed"> <caption>Apply rating curve at Hebden Bridge"</caption> <class>nl.wldelft.fews.util.ConversionServer</class> <configuration>rating_hebdbr_observed.xml</configuration> <auto>"true"</auto> <input>"kw.hebdbr</input> <output>"rating.hebdbr.observed</output> </element> <!-- Run ARMA_FILL through General Adapter--> <element name="arma.hebdbr.observed"> <caption>ARMA correction at Hebden Bridge"</caption> <class>nl.wldelft.fews.util.GeneralAdapter</class> <configuration>arma_hebdbr_historical.xml</configuration> <auto>"true"</auto> <input>"rating.hebdbr.observed</input> <output>"arma.hebdbr.observed</output> </element> <!-- Transform predicted flows to levels at Hebden Bridge--> <element name="rating.hebdbr.predicted"> <caption>Apply rating curve at Hebden Bridge"</caption> <class>nl.wldelft.fews.util.ConversionServer</class> <configuration>rating_hebdbr_predicted.xml</configuration> <auto>"true"</auto> <input>"arma.hebdbr.observed</input> </element> </sequence> </workflow>
Appendix I:
ICA Model Component file for Hebden bridge (file: CALD3-HEBDBR.RFFS)
CALD3-HEBDBR :MOD_COMP_ID
! Version: 1.13 Date: 9-OCT-1997 11:41
! Revision: 2 Date: 9-OCT-1997 14:12
! Origin : IH RFFS Model Network Set-up (Hand)
! File-type : Model Component Description
1 :MC_FILE_FORMAT
1 :COMP_STORE_IND
20961 :COMP_DATA_ID
HEBDEN BRIDGE :Channel flow- kinematic wave :COMP_NAME
1 :COMP_CLASS_IND
4 :COMP_POSITN_IND
6 :COMP_TYPE_IND
1 1 0 :COMP_PURPOSE
0 :DATA_NUM_PROFI
:PROFILE_REQ_ID_PROFI
0 :PROFILE_SUB_ID
0 :SCALE_TYPE_PROFI
0.00000 :SCALE_VALUE_PROFI
2 :DATA_NUM_INPUT
QX-WALSDN1 QX-TODMDN1 :FORECAST_REQ_ID_INPUT
0 0 :SCALE_TYPE_INPUT
0.00000 0.00000 :SCALE_VALUE_INPUT
0 :DATA_NUM_CONTR
:FORECAST_REQ_ID_CONTR
0 :DATA_NUM_SPECI
:FORECAST_REQ_ID_SPECI
2 :DATA_NUM_OUTPU
QX-HEBDBR1 LU-HEBDBR1 :FORECAST_REQ_ID_OUTPU
0 0 :SCALE_TYPE_OUTPUT
0.00000 0.00000 :SCALE_VALUE_OUTPUT
1 :DATA_NUM_DUMMY
2Q-HEBDBR1 :DUMMY_REQ_ID
0 :TIME_DELAY_IND
3 :COMP_DECOMP_TYPE
4 :COMP_DECOMP_NUM
FA_KW :MODEL_ALGOR_ID (1)
2 2 0 0 0 0 0 0 0 0 0 0 :ALGOR_MSIZE_INPUT (1)
0 0 0 0 0 0 0 0 0 0 0 0 :ALGOR_MSIZE_CONTR (1)
0 0 0 0 0 0 0 0 0 0 0 0 :ALGOR_MSIZE_SPECI (1)
1 1 0 0 0 0 0 0 0 0 0 0 :ALGOR_MSIZE_OUTPU (1)
3 0 4 0 3 0 0 0 0 0 0 0 :ALGOR_MSIZE_STATE (1)
3 0 1 0 3 0 0 0 0 0 0 0 :ALGOR_MSIZE_TRANS (1)
38 1 1 3 1 4 4 2 4 4 3 0 :ALGOR_MSIZE_PARAM (1)
29 9 2 1 1 3 4 4 0 0 0 0 :ALGOR_MSIZE_IPARA (1)
9.84000 2.90000 0.00000 1.35900 :ALGOR_VALS_PARAM (1)
0.00000 0.00000 0.00000 1.35900 :ALGOR_VALS_PARAM (1)
0.00000 0.00000 0.00000 0.73300 :ALGOR_VALS_PARAM (1)
0.00000 0.00000 0.00000 0.73300 :ALGOR_VALS_PARAM (1)
0.00000 0.00000 0.00000 1.00000 :ALGOR_VALS_PARAM (1)
0.00000 1.00000 0.00000 1.00000 :ALGOR_VALS_PARAM (1)
0.00000 1.00000 0.00000 1.00000 :ALGOR_VALS_PARAM (1)
0.00000 0.00000 0.00000 1.00000 :ALGOR_VALS_PARAM (1)
0.00000 0.00000 0.00000 1.00000 :ALGOR_VALS_PARAM (1)
0.00000 0.00000 :ALGOR_VALS_PARAM (1)
3 2 4 1 1 0 0 1 1 3 0 0 :ALGOR_VALS_IPARA (1)
0 1 1 0 1 2 2 0 1 3 1 0 :ALGOR_VALS_IPARA (1)
1 3 2 0 1 :ALGOR_VALS_IPARA (1)
QX-WALSDN1 QX-TODMDN1 :DATA_STREAM_INPUT (1)
:DATA_STREAM_CONTR (1)
:DATA_STREAM_SPECI (1)
2Q-HEBDBR1 :DATA_STREAM_OUTPU (1)
0 :TEST_SNOWFLAG_IND (1)
0 :SIMULAT_SKIP_IND (1)
0 :EXECUT_SPECI_IND (1)
6 :EMERG_WARM_UP (1)
RATING :MODEL_ALGOR_ID (2)
1 1 0 0 0 0 0 0 0 0 0 0 :ALGOR_MSIZE_INPUT (2)
0 0 0 0 0 0 0 0 0 0 0 0 :ALGOR_MSIZE_CONTR (2)
0 0 0 0 0 0 0 0 0 0 0 0 :ALGOR_MSIZE_SPECI (2)
1 1 0 0 0 0 0 0 0 0 0 0 :ALGOR_MSIZE_OUTPU (2)
0 0 0 0 0 0 0 0 0 0 0 0 :ALGOR_MSIZE_STATE (2)
0 0 0 0 0 0 0 0 0 0 0 0 :ALGOR_MSIZE_TRANS (2)
5 0 0 1 5 1 0 0 0 0 0 0 :ALGOR_MSIZE_PARAM (2)
2 2 0 0 0 0 0 0 0 0 0 0 :ALGOR_MSIZE_IPARA (2)
13.95000 0.06600 1.47800 9999.00000 :ALGOR_VALS_PARAM (2)
9999.00000 :ALGOR_VALS_PARAM (2)
1 1 :ALGOR_VALS_IPARA (2)
LU-HEBDBR1 :DATA_STREAM_INPUT (2)
:DATA_STREAM_CONTR (2)
:DATA_STREAM_SPECI (2)
QX-HEBDBR1 :DATA_STREAM_OUTPU (2)
0 :TEST_SNOWFLAG_IND (2)
0 :SIMULAT_SKIP_IND (2)
0 :EXECUT_SPECI_IND (2)
0 :EMERG_WARM_UP (2)
ARMA_FILL :MODEL_ALGOR_ID (3)
1 1 0 0 0 0 0 0 0 0 0 0 :ALGOR_MSIZE_INPUT (3)
0 0 0 0 0 0 0 0 0 0 0 0 :ALGOR_MSIZE_CONTR (3)
0 0 0 0 0 0 0 0 0 0 0 0 :ALGOR_MSIZE_SPECI (3)
1 1 0 0 0 0 0 0 0 0 0 0 :ALGOR_MSIZE_OUTPU (3)
4 0 0 1 2 2 0 0 0 0 0 0 :ALGOR_MSIZE_STATE (3)
0 0 0 0 0 0 0 0 0 0 0 0 :ALGOR_MSIZE_TRANS (3)
2 0 2 0 2 0 0 0 0 0 0 0 :ALGOR_MSIZE_PARAM (3)
3 3 0 0 0 0 0 0 0 0 0 0 :ALGOR_MSIZE_IPARA (3)
-1.50830 0.61721 :ALGOR_VALS_PARAM (3)
2 0 1 :ALGOR_VALS_IPARA (3)
2Q-HEBDBR1 :DATA_STREAM_INPUT (3)
:DATA_STREAM_CONTR (3)
:DATA_STREAM_SPECI (3)
QX-HEBDBR1 :DATA_STREAM_OUTPU (3)
0 :TEST_SNOWFLAG_IND (3)
0 :SIMULAT_SKIP_IND (3)
0 :EXECUT_SPECI_IND (3)
4 :EMERG_WARM_UP (3)
RATING :MODEL_ALGOR_ID (4)
1 1 0 0 0 0 0 0 0 0 0 0 :ALGOR_MSIZE_INPUT (4)
0 0 0 0 0 0 0 0 0 0 0 0 :ALGOR_MSIZE_CONTR (4)
0 0 0 0 0 0 0 0 0 0 0 0 :ALGOR_MSIZE_SPECI (4)
1 1 0 0 0 0 0 0 0 0 0 0 :ALGOR_MSIZE_OUTPU (4)
0 0 0 0 0 0 0 0 0 0 0 0 :ALGOR_MSIZE_STATE (4)
0 0 0 0 0 0 0 0 0 0 0 0 :ALGOR_MSIZE_TRANS (4)
5 0 0 1 5 1 0 0 0 0 0 0 :ALGOR_MSIZE_PARAM (4)
2 2 0 0 0 0 0 0 0 0 0 0 :ALGOR_MSIZE_IPARA (4)
13.95000 0.06600 1.47800 9999.00000 :ALGOR_VALS_PARAM (4)
9999.00000 :ALGOR_VALS_PARAM (4)
2 1 :ALGOR_VALS_IPARA (4)
QX-HEBDBR1 :DATA_STREAM_INPUT (4)
:DATA_STREAM_CONTR (4)
:DATA_STREAM_SPECI (4)
LU-HEBDBR1 :DATA_STREAM_OUTPU (4)
0 :TEST_SNOWFLAG_IND (4)
0 :SIMULAT_SKIP_IND (4)
0 :EXECUT_SPECI_IND (4)
0 :EMERG_WARM_UP (4)
1 :CALIBRATION_TYPE
0 :CALIB_EVENT_NUM
1 1 0 0 0 :CALIB_EVENT_BEG (1)
1 19999 0 0 :CALIB_EVENT_END (1)
0 :CALIB_EVENT_WARMUP (1)
0 :CALIB_EVENT_CARRY (1)
1 :CALIB_SCALE_TYPE
1 :CALIB_ORDER_TYPE
1 :CALIB_WEIGHTING_TYPE
1 :UNCERTAINTY_TYPE
0 :UNCERT_PARAM_NUM
0.00000 :UNCERT_VAL_PARAM
UPPER CALDER RECALIBRATION OCT 1997 :CALIB_NOTES (1)
sd fitted in model :CALIB_NOTES (2)
:CALIB_NOTES (3)
:CALIB_NOTES (4)
:CALIB_NOTES (5)
recalib R.AUSTIN IH :COMP_GEN_COMMENT (1)
:COMP_GEN_COMMENT (2)
:COMP_GEN_COMMENT (3)
:COMP_GEN_COMMENT (4)
:COMP_GEN_COMMENT (5)
Appendix II:
Configuration file for running KW module at Hebden Bridge using General Adapter
<?xml version="1.0" encoding="UTF-8"?>
\\
<dyntasks
xmlns="http://www.wldelft.nl/fews"
xmlns:target="http://www.wldelft.nl/fews"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.wldelft.nl/fews dyntasks.xsd" version="1.0">
<general exportpath="file: nffs/kw/input" importpath="file: nffs/kw/output"/>
<sequence>
<!-- export time series for discharge for Todmorden-->
<export seriesid="H-27931-Q.hc" file="file:todmdn.xml">
<xml format="fews"/>
</export>
<!-- export time series for discharge for Walsdn-->
<export seriesid="H-27392-Q.hc" file="file:walsdn.xml">
<xml format="fews"/>
</export>
<!-- export parameters for KW model to Hebden Bridge-->
<exportparams>
<moduleid>kw.hebdbr</moduleid>
<paramsetid>kw.hebdbr.par</paramsetid>
<filename>file:./kw_hebdbr_par.xml</filename>
</exportparams>
<!-- export module initial state -->
<writestate stateid="kw.hebdbr.state"/>
<!-- start kw adapter to transfrom XML data to KW format -->
<!-- note:all settings are examples of filenames. it is assumed the kw module adapter requires one argument. This is an XML file with the relevant configuration! -->
<task diagfile="file:./kwadapter_diag.xml" moduleid="kw.hebdbr">
<deletefile>file:kw_adapter.rtn</deletefile>
<exe>file:nffs/bin/kw_adapter.exe</exe>
<workdir>nffs/kw/workdir</workdir>
<arg>nffs/kw/config/kw_hebdbr_input.xml</arg>
<taskreturn>file:kw_adapter.rtn</taskreturn>
<taskfail>NONEXISTS</taskfail>
</task>
<!-- start kw module -->
<task diagfile="file:kwmodule_diag.xml" moduleid="kw.hebdbr">
<deletefile>file:kw_exe.rtn</deletefile>
<exe>file:nffs/bin/kw.exe</exe>
<workdir>nffs/kw/workdir</workdir>
<arg>nffs/kw/config/kw_hebdbr.rffs</arg>
<taskreturn>file:kw_exe.rtn</taskreturn>
<taskfail>NONEXISTS</taskfail>
</task>
<!-- Start kw adapter to import data form native to XML -->
<task diagfile="file:kwadapter_diag.xml" moduleid="kw.hebdbr">
<deletefile>file:kw_adapter.rtn</deletefile>
<exe>file:nffs/bin/kw_adapter.exe</exe>
<workdir>nffs/kw/workdir</workdir>
<arg>nffs/kw/config/kw_hebdbr_output.xml</arg>
<taskreturn>file:kw_adapter.rtn</taskreturn>
<taskfail>NONEXISTS</taskfail>
</task>
<!-- importing of timeseries -->
<import seriesid="H-27932-Q.hr" file="file:kw_hebdbr_q.xml">
<xml format="fews"/>
</import>
<!-- import resulting module state -->
<readstate stateid="kw.hebdbr.state" statename="state kw to hebdbr">
<stateloc type="file">
<readlocation>nffs/kw/state</readlocation>
<writelocation>nffs/kw/state</writelocation>
</stateloc>
</readstate>
</sequence>
</dyntasks>
Appendix III:
Example of an output XML file (shown in part)
Filename: KW_HEBDBR_Q.XML
<?xml version="1.0" encoding="UTF-8"?> \\ <timeseries xmlns="http://www.wldelft.nl/fews" xmlns:target="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews timeser.xsd" version="1.0"> <header> <type>instantanious</type> <timeseriesid>kw.output.1</timeseriesid> <content>Discharge</content> <timestep>3600</timestep> <missval>-999</missval> <longname>Simulated discharge at Hebden Bridge</longname> <stationName>Hebden Bridge</stationName> <units>m3/s</units> <locationid>H-27932</locationid> <sourceOrganisation>Northeast Region</sourceOrganisation> <sourceSystem>KW module</sourceSystem> <fileDescription>XML Data</fileDescription> <creationDate>2003-05-05</creationDate> <creationTime>14:42:14</creationTime> </header> <event date="2003-01-07" time="08:00:00" value="2.480000" flag="0"/> <event date="2003-01-07" time="09:00:00" value="2.470000" flag="0"/> <event date="2003-01-07" time="10:00:00" value="2.460000" flag="0"/> <event date="2003-01-07" time="11:00:00" value="2.440000" flag="0"/> <event date="2003-01-07" time="12:00:00" value="2.420000" flag="0"/> ... ... <event date="2003-01-17" time="02:00:00" value="1.490000" flag="0"/> <event date="2003-01-17" time="03:00:00" value="1.480000" flag="0"/> <event date="2003-01-17" time="04:00:00" value="1.480000" flag="0"/> <event date="2003-01-17" time="05:00:00" value="1.480000" flag="0"/> <event date="2003-01-17" time="06:00:00" value="1.480000" flag="0"/> <event date="2003-01-17" time="07:00:00" value="1.490000" flag="0"/> </timeseries>
