Page History

...

...

The table below lists the various XML Schemas. Page number refers to the page in Appendix E where the schema is defined.

...

PCRaster is a unique raster based GIS package ideally suited for dynamic modelling. Information on the PCRaster native format (and a free version, including tools for conversion) can be obtained from http://www.pcraster.nl^{Image Removed}.

...

These files consist of an ASCII header file (with content information) and a binary data file. The ASCII description files support the following formats:

1. 1. Band sequential (BSQ) multiband images
   2. Band interleaved by line (BIL) multiband images
   3. Band interleaved by pixel (BIP) multiband images

...

1. 1. Add a line to the header file with the number of time blocks: for example nBlocks 3.

1. 1. The nBands property in the header file represents the number of available parameters.

...

The binary image file for the BIL/BIP/BSQ image format is merely a bit stream of the image data. How the image data is arranged in that bit stream defines whether it is a BIL, BIP, or BSQ image.
Band interleaved by line data stores pixel information band by band for each line, or row, of the image. For example, given a three-band image, all three bands of data are written for row one, all three bands of data are written for row two, and so on, until the total number of rows in the image is reached.
Band interleaved by pixel data is similar to BIL data, except that the data for each pixel is written band by band. For example, with the same three-band image, the data for bands one, two, and three is written for the first pixel in column one; the data for bands one, two, and three is written for the first pixel in column two; and so on. Band sequential format stores information for the image one band at a time. In other words, data for all pixels for band one is stored first, then data for all pixels for band two, and so on.
For further information see: http://www.esri.com/library/whitepapers/pdfs/eximgav.pdf^{Image Removed}

...

ASCII grids are stored in a format compatible with ESRI (and many other) software. The ASCII raster file format is a simple format that can be used to transfer raster data between various applications. The header data includes the following keywords and values:

1. 1. ncols - number of columns in the data set.
   2. nrows - number of rows in the data set.
   3. xllcenter or xllcorner - x-coordinate of the centre or lower-left corner of the lower-left cell.
   4. yllcenter or yllcorner - y-coordinate of the centre or lower-left corner of the lower-left cell.
   5. cellsize - cell size for the data set.
   6. nodata_value - value in the file assigned to cells whose value is unknown. This keyword and value is optional. The nodata_value defaults to -9999.

...

The EA Time Series Schema established in draft from for hydrometric data can be mapped to the time series XML schema established as a part of the published interface. This mapping is illustrated in the table below (* indicate mandatory fields).

...

Following the discussion between Delft Hydraulics and CEH & Wallingford Software on Friday 2 May 2003 in Wallingford on providing an enumeration of quality flags as a part of the SIS, a proposed quality flag enumeration is given in the table below. As described in the timeseries XML schemas provided we have made allowance for a single quality flag. This is contrary to the EA XML interchange formats which provide for three separate quality flags. A single flag seems more appropriate to the level of communication we are dealing with and should avoid ambiguity. The flags are single byte values and are incorporated in the XML Schema for time series. Quality flags are provided for each data point.
Quality flags are constructed on a philosophy of two qualifiers. The first described the origin of the data and the second the quality.
Possible origins of data are:

1. Original: This entails the data value is the original value. It has not been amended by NFFS
2. Completed: This entails the original value was missing and was replaced by a non-missing value.
3. Corrected: This entails the original value was replaced with another non-missing value.

...

1. Reliable: Data is reliable and valid
2. Doubtful: The validity of the data value is uncertain
3. Unreliable: The data value is unreliable and cannot be used.

...

...

• No difference is made between historic and forecast data. This is not considered a quality flag. The data model of NFFS is constructed such that this difference is inherent to the data type definition.
• External sources may either be an actual external source, a forecasting module or a transformation. The convention in NFFS the definition of data series parameter types identifies the data source.

...

1. to provide a description of the preferred approach in developing module adapters, and,
2. to illustrate the approach through an example taken from the Northeast region.

...

1. Export of data required by the module through the published interface from the NFFS database to the module native format. This data can cover dynamic time series data, parameter value sets, module states etc. A full description of the data types supported is given in SIS Module Adapter Specification, Version 2.0. To determine what data is exported to a given module, a configuration file (XML formatted) is passed to the General Adapter as an argument. This file is a part of the NFFS and is configured during set-up of a forecasting system. After the general adapter has run the input files required by the module are available in the published interface format. This can entail time series, module states etc. Figure 3 shows a schematic description of this first step. The module adapter must provide on completion an XML formatted diagnostic file giving the general adapter information on the status of the translation process. For the format of this file and associated enumeration see the published interface specification.

...

1. In the second step the module itself is run. This is achieved by the general adapter executing a call to the module executable, with the possible addition of command line arguments. This module executable must be able to run in batch mode, without any user interaction required. The module executable reads the required input data, parameters and states in its native format, performs the required calculation and establishes a set of output files and states, again in the native module format. This set of output files must include a file giving diagnostic information on the module run. This file can be in the native module format. Figure 4 gives a detailed schematic overview of concept of running a module executable within NFFS.

...

1. The third step comprises the import of module output data into NFFS. Again this is using the published interface format. The module adapter is called to transform native module output formats to the published interface format. Once completed the output data is retrieved through the published interface format for insertion into the NFFS central database. As with the module inputs an XML formatted configuration file is used to determine what data are imported.

...

...

...

The interface is indeed intended to be operated an on-line operational system. It's prime communication with external forecasting modules is through dynamic data such as time series, model states and diagnostics (see below). The published interface formats have been defined to cover a wider range of data types, but these need not be used in all cases - i.e. there is no requirement to allow communication of all data types. Indeed there is currently no module that caters for the full range of formats. Although the EA is to decide to what extent data should be passed, the practical approach followed with e.g. ISIS and the CEH modules is to pass only time series, module states and diagnostics information.
In exchanging data between a module and NFFS, a prioritisation of data types can be identified.

...

In its current form the General Adapter does not use the HarmonIT (OpenMI) module interchange facilities. The DELFT-FEWS application will in the near future be extended with an OpenMI compatible adapter, but this is as yet not considered an NFFS requirement.

...

This assumption is correct - see also G.1.1

...

The DELFT-FEWS system has been developed in JAVA and is platform independent. However, in the configuration of NFFS, all runs of forecasting modules are executed on dedicated servers. Presently, these are Windows systems.

...

There is no specific requirement for the module adapter. The only requirement is that it can communicate through the XML file formats. The use of standard methods for reading and writing XML files, based on the schemas provided is highly recommended. This not only reduces implementation efforts but also guarantees compatibility. The general adapter currently allows the external module to be run either as an executable, or as a Java method. A minor extension will allow running of DLL's. Batch files are not recommended as the General Adapters monitors return codes from the external module. These are not always correctly passed in Batch files, and as such ungraceful failure of a module is not easily identified.

...

The general adapter initiates the executable, Java method (or DLL) through the system/Java calls. Arguments may be passed when required, though these must be static. The general adapter may also be used to set environment variables for the module where required.
The module normally runs in a dedicated directory (tree). a working directory can be configured to act as the root of the module run.

...

This is correct. The most specific requirement is that the module must not have any manual interaction. This would preclude running the module in a distributed system.

...

This is a sole choice of the module adapter. There is a slight preference to ASCII formats as this eases debugging. However, if this has implications as to module performance (additional pre-processing requirements) then the binary format is advocated.

...

The general adapter has full state management facilities. This means the module is provided with an initial state for the start of run. The module should also provide a resulting state at the end of run, or at an intermediate time. When required this resulting state is administered and stored in the NFFS database for later use.
It is important to note that the actual module state file is handled blindly - ie it is not interpreted. NFFS takes the state and time-stamps it for storage in the database. NFFS will not change the content of the state, though renaming of files/directories on import/export is possible.
For cold start runs a default state is provided to the module.
Module results are generally passed back through the Published Interface XML format and stored in the database. Obviously only the required locations need be passed.

...

DELFT-FEWS provides a full set of generic tools for these derived data, including all of the above. Generally DELFT-FEWS is configured to pre-process all data and provide the module directly with required inputs. The module need not do any additional processing.

...

DELFT-FEWS provides an internal ARMA error modelling tool. This is module independent and is run in sequence to the module when required.

...

Sequences of modules and data handling functions are run in sequence by so-called workflows. This means that a workflow may be configured to first run a rainfall-runoff model (e.g. NAM, PDM) through the General adapter, then run an error correction routine to the output and subsequently run a hydrodynamic module (e.g. ISIS, MIKE11, SOBEK)

...

There is no difference to the module if it is on or off line. This layer is managed by DELFT-FEWS.

...

In its current form the diagnostics file to each external module run should be one file. The general adapter does, however, allow for executing a sequence of modules, with a diagnostics file for each.

...

The diagnostics passed back should be relatively limited - there is no strict requirement on the length of the message, but to the point messaging is advocated. The message should provide the level of knowledge to the operational forecaster to monitor the system. Different levels of messages should be used, with information useful in debugging being labelled appropriately.
For more detailed messaging - native module formats can be used, but this is only for specialist analysis. It should be noted that when a module fails, a zipped dump files is made of all module files and I/O. This is set aside for later, detailed, analysis.

...

It is good practice for a module to clean up redundant files after use. The general adapter does allow configuration of a purge activity either before the module run or after the module run (or both).

...

In the general adapter configuration (XML) the location of the PI-XML files to pass to the module is given, as well as the location where PI-XML files are expected. The location of diagnostics files and work directories can also be configured. The structure is dictated by the Published Interface. All files passed to the module will be in the same directory and all files passed back are expected in the same directory (with the exception of the diagnostics file).

...

To ensure simplicity, the length of the module run is dictated by the length of the time series passed. When a distinction is to be made between forecast and historic run, these will be passed as separate time series. The module adapter should identify run times from these.

...

Indeed. The module state returned will be administered by DELFT-FEWS. Depending on the status of the run it will be administered as a state to be used in ensuing forecasts, or disregarded.

...

Indeed. The General Adapter is configured to provide the module with the required data. Data may be provided at equidistant or non-equidistant intervals.

...

Yes. A state represents the state of a module at a single point in time.

...

The following quality checks can be made within DELFT-FEWS (when configured by the user)

• Detection of outliers (data flagged unreliable)
• Check of exceedance of hard limits (data flagged unreliable)
• Check of exceedance of soft limits (data flagged doubtful)
• Rate of change check. When a rate of change exceeds a pre-set value, data is flagged unreliable
• Same readings check: When data remains within a pre-set range for a configured period, data is flagged unreliable
• Temporary shift check. When a 'temporary shift' is detected, data is flagged unreliable

...

Several levels are possible.