Contents

Time Series

Time series data represent data collected over a given period of time at a specific location. DELFT-FEWS can read and write time series in XML format (see fileformats.xsd) described below. Time series files can contain both equidistant times series and non-equidistant series. Multiple time series can be stored in a single file. All time and date information is given in GMT unless otherwise stated. The default (and preferred) missing value definition is NaN.
Quality flags are constructed on a philosophy of two qualifiers. The first described the origin of the data and the second the quality.
Possible origins of data are:

1. Original: This entails the data value is the original value. It has not been amended by DELFT-FEWS
2. Completed: This entails the original value was missing and was replaced by a non-missing value.
3. Corrected: This entails the original value was replaced with another non-missing value.

Possible qualifiers are:

1. Reliable: Data is reliable and valid
2. Doubtful: The validity of the data value is uncertain
3. Unreliable: The data value is unreliable and cannot be used.

See Appendix 11D for a tabular presentation of the quality flags.

The XML file holds a header containing metadata for the time series. Time series may either be equidistant or non-equidistant. This is indicated in the timestep element.
Example:

<?xml version="1.0" encoding="UTF-8"?>
<TimeSeries version="1.1" xmlns="http://www.wldelft.nl/fews" xmlns:target="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews pi_timeseries.xsd">
<series>
  <header>
    <type>accumulative</type>
    <locationId>Rhine_99_1</locationId>
    <parameter>Precipitation</parameter>
    <timeStep unit="second" multiplier="3600"/>
    <startDate date="1967-08-13" time="14:00:00"/>
    <endDate date="1967-08-13" time="15:00:00"/>
    <missVal>-999</missVal>
    <longName>Rockenau</longName>
    <stationName>Rockenau</stationName>
    <units>mm</units>
    <sourceOrganisation>BfG</sourceOrganisation>
    <sourceSystem>DWD-system3</sourceSystem>
    <fileDescription>Very short series for Rockenau</fileDescription>
    <creationDate>2003-01-11</creationDate>
    <creationTime>14:00:00</creationTime>
  </header>
  <event date="1967-08-13" time="14:00:00" value="3.1" flag="0"/>
  <event date="1967-08-13" time="15:00:00" value="-999"/>
</series>
<series>
  <header>
    <type>instantaneous</type>
    <locationId>Rhine_99_3</locationId>
    <parameter>Discharges</parameter>
    <timeStep unit="nonequidistant"/>
    <startDate date="1967-08-13" time="14:00:00"/>
    <endDate date="1967-08-13" time="15:00:00"/>
    <missVal>NaN</missVal>
    <longName>Rockenau</longName>
    <stationName>Rockenau</stationName>
    <units>m3/s</units>
    <sourceOrganisation>BfG</sourceOrganisation>
    <sourceSystem>DWD-system3</sourceSystem>
    <fileDescription>Very short series </fileDescription>
    <creationDate>2003-01-11</creationDate>
    <creationTime>14:00:00</creationTime>
  </header>
  <event date="1967-08-13" time="14:00:00" value="3.1" flag="0" comment="handmatige meting"/>
  <event date="1967-08-13" time="15:00:00" value="NaN"/>
  <event date="1967-08-13" time="18:00:00" value="7.1"/>
</series>
</TimeSeries>

See for the meaning of the quality flags the next list.

The structure of the XML schema is given in the figures below. Note that attributes are not displayed in the figures. Consult the full schema (+documentation) for details.
 

To support the validation status, it is possible since PI timeseries version 1.19 to define flagSourceColumns and (custom)flagSources in the PI-XML timeseries format. For example a flag source column named 'V3' that has a custom flag source value 'OK' will result in an attribute 

fs:V3="OK"

See the example below

<TimeSeries xmlns="http://www.wldelft.nl/fews/PI" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews/PI http://fews.wldelft.nl/schemas/version1.0/pi-schemas/pi_timeseries.xsd" version="1.19" xmlns:fs="http://www.wldelft.nl/fews/fs">
	<timeZone>1.0</timeZone>
	<series>
		<header>
			<type>instantaneous</type>
			<moduleInstanceId>WerkFilter</moduleInstanceId>
			<locationId>140-w_Leidsche Rijn</locationId>
			<parameterId>HGTE_METING_15M</parameterId>
			<timeStep unit="second" multiplier="900"/>
			<startDate date="2012-06-25" time="17:45:00"/>
			<endDate date="2012-06-25" time="21:45:00"/>
			<missVal>NaN</missVal>
			<stationName>AANVOERDER_140-w_Leidsche Rijn</stationName>
			<lat>52.082416488309036</lat>
			<lon>5.075337740730613</lon>
			<x>133623.0</x>
			<y>454951.0</y>
			<z>-9999.0</z>
			<units>mNAP</units>
		</header>
		<event date="2012-06-25" time="19:30:00" value="-0.414" flag="0" fs:V3="OK" fs:V4="OK"/>
		<event date="2012-06-25" time="19:45:00" value="-0.356" flag="0" fs:V3="OK"/>
		<event date="2012-06-25" time="20:00:00" value="-0.484" flag="0" fs:V2="OK" fs:V3="OK"/>
		<event date="2012-06-25" time="20:15:00" value="-0.57" flag="0" fs:V2="OK" fs:V3="OK"/>
		<event date="2012-06-25" time="20:30:00" value="-0.574" flag="0" fs:V2="OK" fs:V3="OK"/>
		<event date="2012-06-25" time="20:45:00" value="-0.453" flag="0" fs:V2="OK" fs:V3="OK"/>
		<event date="2012-06-25" time="21:00:00" value="-0.408" flag="0" fs:V2="OK" fs:V3="OK"/>
		<event date="2012-06-25" time="21:15:00" value="-0.551" flag="0" fs:V2="OK" fs:V3="OK"/>
		<event date="2012-06-25" time="21:30:00" value="-0.436" flag="0" fs:V2="OK" fs:V3="OK"/>
		<event date="2012-06-25" time="21:45:00" value="-0.385" flag="0" fs:V2="OK" fs:V3="OK"/>
	</series>
</TimeSeries>

Details of Timeseries Header

(Time series) Location

Time series location data files identify points at which specific times series exist. A point is represented by X, Y and Z co-ordinates. If no geographical datum is supplied LOCAL is assumed.

Example:
<?xml version="1.0" encoding="UTF-8"?>
<Locations xmlns="http://www.wldelft.nl/fews" xmlns:target="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews pi_locations.xsd" version="1.1">
<geoDatum>WGS-1984</geoDatum>
<location>
<stationName>Mozel 34</stationName>
<locationId>LocationA</locationId>
<x>3.2</x>
<y>3.1</y>
<z>100</z>
</location>
<location>
<stationName>Hupselse beek</stationName>
<locationId>LocationB</locationId>
<x>5.1</x>
<y>3.9</y>
<z>100</z>
</location>
<location>
<stationName>No</stationName>
<locationId>LocationC</locationId>
<x>4.1</x>
<y>3.1</y>
<z>100</z>
</location>
</Locations>
The structure of the XML schema is given in the figures below. Note that attributes are not displayed in the figures. Consult the full schema (+documentation) for details.

Map (stacks) data (grids)

Maps stacks are time series of grid data. Map stacks that hold one time step are treated as static data. The maps themselves are stored in either binary or ASCII file(s) but the description and the information needed to import them into the DELFT-FEWS database is stored in an XML file.
Each time step is stored in a separate ASCII/PCRaster based file or in a single USGS BIL file. In the former the order of the maps is determined by the file extension and up to the last four characters of the first part of the filename:
(eg. evap0000.001, evap0000.002 .... evap0000.999, evap0001.000). This limits the size of a stack of maps to 9999999.

Example:
<?xml version="1.0" encoding="UTF-8"?>
<MapStacks xmlns="http://www.wldelft.nl/fews" xmlns:target="http://www.wldelft.nl/fews/interface-0.9" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews pi_mapstacks.xsd" version="1.1">
<geoDatum/>
<mapStack>
<locationId>test1</locationId>
<parameter>vpd</parameter>
<timeStep>
<noneq/>
</timeStep>
<startDate date="2003-09-09" time="18:00"/>
<endDate date="2003-09-09" time="20:00"/>
<file>
<usgs file="file://allvpd.bil" header="file://allvpd.hdr"/>
</file>
<!--Three maps are defined, starting at 18:00, ending at 20:00-->
<event date="2003-09-09" time="18:00"/>
<event date="2003-09-09" time="19:00"/>
<event date="2003-09-09" time="20:00"/>
</mapStack>
</MapStacks>
The structure of the XML schema is given in the figures below. Note that attributes are not displayed in the figures. Consult the full schema (+documentation) for details.

The following three formats are allowed for the actual grids:

1. ESRI .ASC ASCII file grids (see appendix B3)
2. USGS BIL, BIP, BSQ files (with a separate header, see appendix B2)
3. PCRaster maps (see appendix B1)

Netcdf for grid data

Besides the possibility to import/export grid data by map stacks DELFT-FEWS offers the possibility to import and export grid data in netcdf-format. Because the netcdf-format is self-describing the netcdf-file only is sufficient as input for DELFT-FEWS. When data is exported from DELFT-FEWS to a model the output will also only consist of a netcdf-file.
Netcdf will only be used to import/export the actual grid data. The mapping of the grid cells to real geographical coordinates will not be handled by the general adapter.
The netcdf-files must be compliant with the conventions described in this document. These conventions restrict the CF metadata conventions.

Module state data

Modules can have their state (i.e. all the data and parameters the module needs to restart) saved by DELFT-FEWS. Many different solutions are in use by current models. As such, DELFT-FEWS regards model state data as a black box, it is not parsed or interpreted by the system. Programs must specify a list of files and/or directories that DELFT-FEWS can save (and later restore).
The model data state file is an XML file in which a read and write location are specified for each file/directory. If the read location is a directory, the write location needs to be a directory also (and v.v).
Each state must include a timestamp indicating the date and time for which that state is valid. A module ID is included to identify to which module the state pertains.
The readLocation is the location the module adapter reads from, the writeLocation is the location the module adapter writes to. Consequently the general adapter will read the data from the writeLocation and write the data to the readLocation.

Example:
<?xml version="1.0" encoding="UTF-8"?>
<State xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews pi_state.xsd" version="1.1">
<stateId>sum23</stateId>
<stateName>Summer 1995</stateName>
<dateTime date="2003-06-06" time="10:00"/>
<stateLoc type="file">
<readLocation>file://d:/exp/sbek.mda</readLocation>
<writeLocation>file://d:/imp/sbek.mda</writeLocation>
</stateLoc>
<stateLoc type="directory">
<readLocation>file://d:/HBV</readLocation>
<writeLocation>file://d:/HBV</writeLocation>
</stateLoc>
</State>
The structure of the XML schema is given in the figures below. Note that attributes are not displayed in the figures. Consult the full schema (+documentation) for details.

Longitudinal profile

A Longitudinal profile describes a profile of (part of) a branch. For each point in a profile a chainage and a value are defined. Multiple profiles can be specified in a single file. Each profile can hold several timesteps, i.e. several instances of the same profile at different times.

Example:
<?xml version="1.0" encoding="UTF-8"?>
<Profiles xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews pi_profiles.xsd" version="1.1">
<geoDatum/>
<profile>
<locationId>some id</locationId>
<parameter>water level</parameter>
<timeStep>
<seconds>3600</seconds>
</timeStep>
<startDate date="2003-09-09" time="13:00"/>
<endDate date="2003-09-09" time="14:00"/>
<event date="2003-09-09" time="13:00">
<xdata x="12" y="13" z="0" chainage="356" value="24.9"/>
</event>
<event date="2003-09-09" time="14:00">
<xdata x="12" y="13" z="0" chainage="356" value="24.9"/>
</event>
</profile>
</Profiles>
The structure of the XML schema is given in the figures below. Note that attributes are not displayed in the figures. Consult the full schema (+documentation) for details.

Module parameters

New Style Parameters

Since September 2008 a new type of module parameters file has been introduce to the DELFT-FEWS system. This file coexists with the previous module parameters file, described in the next section. However in the future the old file format is to be phased out and replaced by the new file format.
The module parameters are subdivided into Groups. A Group has an unique id and the following list of elements:

1. description: Used to describe this group of parameters. (Optional)
2. locationId: Location id for this group of parameters. If used externally this is the external locationId. (Optional)
3. model: Element to identify the model (algorithm) the parameters are configured for. (Optional)
4. validPeriod: Period the parameters are valid in. When missing the parameters are always valid. The idea is to have only one Group without a valid period. This Group would then be considered as the DEFAULT Group. (Optional)
   Choice between a list of:
5. parameter: A number of parameter items. Each parameter item must have an unique id. (> 0)
6. ensemble: A number of ensemble members. Each ensemble parameter item must have an unique id.(> 0)

The structure of the XML schema is given in the figure below:

Details of parameters

Details of ensembles

Old Style Parameters

DELFT-FEWS can read and write module parameters in XML format. The format is described in the schema. Each file can contain a number of parameters that are (optionally) combined in parameter sets. A parameter can be a floating point value (with a specified minimum and maximum value), an integer value (also with a specified minimum and maximum value) or a Boolean value (either 0 or 1, false or true). Double long and Boolean values can have an allow adjust parameter that indicates if DELFT-FEWS may adjust the parameter in some sort of fitting exercise. String parameters can be used to store specific module information.

Example:
<?xml version="1.0" encoding="UTF-8"?>
<Parameters xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews pi_prameters.xsd" version="1.1">
<param>
<name>Van genuchten Alpha</name>
<data>
<doubleData alowAdjust="1" maxVal="0.6" minVal="0.01">0.12</doubleData>
</data>
</param>
<param>
<name>Van genuchten n</name>
<data>
<doubleData alowAdjust="1" maxVal="1.9" minVal="0.6">1.2</doubleData>
</data>
</param>
<param>
<name>Nr itterations</name>
<data>
<intData alowAdjust="0" maxVal="500" minVal="3">300</intData>
</data>
</param>
</Parameters>
The structure of the XML schema is given in the figures below. Note that attributes are not displayed in the figures. Consult the full schema (+documentation) for details.

Implementing the Module Parameters in the Genaral Adapter

For information on how to export the Module Parameter file from the General Adapter, check the following link:
Module datasets and parameters

Lookup tables

To relate parameters in lookup tables two or three parameter lookup tables can be used (e.g. stage height to discharge tables). Each XML file can contain one or more relations. Optional information such as extrapolation type and the validity (time) of the relation can be added.

Example:
<?xml version="1.0" encoding="UTF-8"?>
<Table xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews
pi_table.xsd" version="1.1">
<rel>
<info>
<startDate date="2003-09-09" time="13:00"/>
<extrapolation type="non"/>
</info>
<!-- The following defines a 2 rows by three columns matrix -->
<rows>2</rows>
<cols>3</cols>
<data>
<!--The items are processed per row-->
<item type="double">12</item>
<item type="double">30</item>
<item type="string">Waterlooo station</item>
<item type="double">13</item>
<item type="double">40</item>
<item type="string">Westeinder</item>
</data>
<!-- The end-result, in tabular form should look like this:
col1col2 col3
==========================
1230Waterloo station
1340Westeinder
-->
</rel>
</Table>


The structure of the XML schema is given in the figures below. Note that attributes are not displayed in the figures. Consult the full schema (+documentation) for details.

Module diagnostics output

DELFT-FEWS uses the module diagnostics output format to report the results from modules to the user. The general adapter can also use these files. The diagnostics files can hold several warning levels attached to a line (message):
3 = info (information, all is well, e.g. :"SOBEK: program ended") 2 = warn (warning information. e.g. "SOBEK: high number of iterations") 1 = critical (critical problems. e.g. "SOBEK: no convergence") 0 = crash (full module crash. e.g. "SOBEK: ooops, what now?").

Example:
<?xml version="1.0" encoding="UTF-8"?>
<Diag xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews pi_diag.xsd" version="1.1">
<line level="3" description="HVB Starting at 12:00 10 Nov 2002"/>
<line level="1" description="No improvement in optimisation"/>
<line level="3" description="Run ended at 12:03 10 Nov 2002"/>
<!--Each line/mesasage contains the actual text and a warning level.
each message/line has a level attached to it
3 = info (information, all is well, e,g, :"SOBEK: program ended")
2 = warn (warning information.
e.g. "SOBEK: high number of iterations")
1 = critical (critical problems. e.g. "SOBEK: no convergence")
0 = crash (full module crash. e.g. "SOBEK: ooops, what now?")
All levels higher than 3 are regarded as non-essential (debug) information-->
</Diag>
The structure of the XML schema is given in the figures below. Note that attributes are not displayed in the figures. Consult the full schema (+documentation) for details.

Polygons

Polygon data identifies a specific area. For example, a catchment that contributes water to a specific branch. This data type consists of several elements, including a list of X and Y point locations that form the vertices of a polygon feature. The first and last points in this list are joined to form a closed polygon. Optionally a polygon centroid can be given. Usually this is the point to which precipitation input to (lumped) models must be interpolated.
Each polygon element can hold several events. As such, polygons can also be dynamic elements.

Example:
<?xml version="1.0" encoding="UTF-8"?>
<Polygons xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews
pi_polygons.xsd" version="1.1">
<geoDatum>LOCAL</geoDatum>
<timeZone>0.0</timeZone>
<polygon>
<locationId>String</locationId>
<parameter>String</parameter>
<timeStep>
<seconds>3600</seconds>
</timeStep>
<startDate date="1967-08-13" time="13:00:00"/>
<endDate date="1967-08-13" time="14:00:00"/>
<name>Flood Rhine</name>
<upstreamId>rh2</upstreamId>
<event date="1967-08-13" time="13:00:00">
<pt x="3.14159265358979" y="3.14159265358979" z="3.14159265358979" mark="1"/>
<pt x="3.14159265358979" y="3.14159265358979" z="3.14159265358979" mark="2"/>
</event>
<event date="1967-08-13" time="14:00:00">
<pt x="3.14159265358979" y="3.14159265358979" z="3.14159265358979" mark="0"/>
<pt x="3.14159265358979" y="3.14159265358979" z="3.14159265358979" mark="0"/>
</event>
</polygon>
</Polygons>

The structure of the XML schema is given in the figures below. Note that attributes are not displayed in the figures. Consult the full schema (+documentation) for details.

Run File

Since 2008.03. General information about the run executed by the general adapter that is used by the pre and post adapter
This file generated on the fly during a pi run file export activity from the General Adapter. This file is read only for the pre and post adapters.
A pre or post adapter that supports pi run files only needs this pi run file at the command line argument to find all the pi files it needs to read and write.
Other options can be passed to the pre or post adapter by adding extra properties to the generated pi run file during export.

Here is a list of information available in the run file:

1. timeZone: Time zone used in the exported pi files. This is also the default time zone used by the general adapter when the time zone is not specified in an imported file.
2. startDateTime: Start of the run. The earliest time exported by the General Adapter.
3. endDateTime: End of the run. The latest time exported by the General Adapter.
4. time0: This is the start of a forecast for a forecast run. This is the time 0 of a FEWS task run.
5. lastObservationDateTime: Last time of non missing observation based on observed time series sets configured in the general adapter.
6. workDir: Fully qualified path to the current working directory. This is also the startup directory of an non java executable launched by the general adapter.
7. inputParameterFile: Fully qualified path to the pi parameter file exported by the General Adapter before the run. An active modifier is already applied to this file
8. inputStateDescriptionFile: Fully qualified path to the pi state file that describes the state exported by the General Adapter. In this file you can also find the state time and the paths to the actual state files.
9. inputTimeSeriesFile: Fully qualified path to a pi time series file exported by the General Adapter before the run. Element may occur multiple times.
10. inputMapStackFile: Fully qualified path to a map stack file exported by the General Adapter before the run.
    A map stack is a list of raster files for each time step.
11. inputNetcdfFile: Fully qualified path to a pi netcdf file exported by the General Adapter before the run
12. inputLongitudinalProfilesFile: Fully qualified path to a pi longitudinal profiles file exported by the General before the run
13. outputDiagnosticFile: Fully qualified path to a pi diagnositic file that that will be imported by the General
    Adapter after the run . It is deleted by the general adapter before the run.
14. outputStateDescriptionFile: Fully qualified path to the pi state file that descripes the state that will be imported by the General Adapter
    adapter after the run
15. outputTimeSeriesFile: Fully qualified path to a pi time series file that will be imported by the General Adapter after the run
16. outputMapStackFile: Fully qualified path to a pi map stap file that will be imported by the General Adapter after the run. A map stack is a list of raster files for each time step.
17. outputNetcdfFile: Fully qualified path to a pi netcdf file that will be imported by the General Adapter after the run
18. outputLongitudinalProfilesFile: Fully qualified path to a pi longitudinal profiles file that will be imported by the General Adapter after the run
19. properties: Kind of extra command line arguments for the pre and post adapter. This is also a
    convinient way way to pass global properties to a pre or post adapter. An adapter is not allowed to access the
    FEWS global.properties directly. These extra options makes an additional pre or post adapter configuration file unnessesary.
    See the documentation of the used pre and post adapter for the supported property keys with their allowed values

The structure of the XML schema is given in the figure below (This file is currently under construction):

An example file is given in the figure below (under construction):

<?xml version="1.0" encoding="UTF-8"?>
<Run     xsi:schemaLocation="http://www.wldelft.nl/fews/PI http://fews.wldelft.nl/schemas/version1.0/pi_run.xsd"
    xmlns="http://www.wldelft.nl/fews/PI" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <timeZone>0.0</timeZone>
    <startDateTime date="2008-10-17" time="09:00:00"/>
    <endDateTime date="2008-10-23" time="11:00:00"/>
    <time0 date="2008-10-21" time="09:00:00"/>
    <workDir>F:\Fews\Midlands_sa\Modules\MCRM\Sev-C101</workDir>
    <inputStateDescriptionFile>F:\Fews\Midlands_sa\Modules\MCRM\Sev-C101\InputFromGA\state.xml</inputStateDescriptionFile>
    <inputTimeSeriesFile>F:\Fews\Midlands_sa\Modules\MCRM\Sev-C101\InputFromGA\inp_hindcast.xml</inputTimeSeriesFile>
    <inputTimeSeriesFile>F:\Fews\Midlands_sa\Modules\MCRM\Sev-C101\InputFromGA\inp_forecast.xml</inputTimeSeriesFile>
    <inputTimeSeriesFile>F:\Fews\Midlands_sa\Modules\MCRM\Sev-C101\InputFromGA\inp_stateupdate.xml</inputTimeSeriesFile>
    <outputTimeSeriesFile>F:\Fews\Midlands_SA\Modules\MCRM\SEV-C101\OutputToGA\out_hindcast.xml</outputTimeSeriesFile>
    <outputTimeSeriesFile>F:\Fews\Midlands_SA\Modules\MCRM\SEV-C101\OutputToGA\out_forecast.xml</outputTimeSeriesFile>
    <properties>
        <float key="factor" value="12.0"/>
        <string key="mode" value="G"/>
    </properties>
</Run>

Implementing the Pi_Run option in the Genaral Adapter

To generate a pi_run file for pre or post adapter you can configure a export run file activity
The export fun file activity in the General Adapter XML schema is given in the figure below: