In this section the interaction between the DELFT-FEWS and external modules is described. Additional details are given in Appendix F.

Principles

To allow running modules within the DELFT-FEWS through the general adapter, these modules must adhere to a number of conventions. The general adapter does provide a flexible interface, which entails that there is not a singular strict definition on how modules are run. However, the options provided do not allow complete freedom. This section describes in detail how a module is run and what conventions exist.

In all cases, the central philosophy of the general adapter is that it is to as large an extent as possible ignorant of module specific details. This intelligence is strictly separated from the DELFT-FEWS side of the system, as this guarantees an open system. Module specific intelligence required by the module for running this in real time is vested in the module adapter itself. Figure 1 gives a schematic view on the interaction between DELFT-FEWS (central database) on the one side and the Module on the other. Communication is established through the published interface. The general adapter (this module is a part of DELFT-FEWS) is configured to provide the required data (both dynamic and static) in the published interface format. A module adapter (supplied by the same supplier as the module itself) is used to translate the data from the published interface format to the native module format.
DELFT-FEWS

Figure 1 Schematic interaction between the General Adapter and the Module adapter through the published interface.

Module runtime environment

Most conventions for running modules from DELFT-FEWS through the general adapter concern the runtime environment.

Module input and output

In configuring the general adapter to provide data for use in a specific module, an agreed location (URI path) is given. All files are then written to this location for use by the module adapter. This location, as with all locations configured in the general adapter is a static location, i.e. once configured it does not change from run to run.
In a similar fashion, the module output is expected by the general adapter to be read from an agreed location (URI path). This need not be the same location as that used to make data available to the module.

Running modules from DELFT-FEWS

Running the executable of a third-party module can be defined to follow two principles.

• A single task may be defined, where the general adapter makes a call to a single executable. This is for example a call to the module adapter which then imports data to the module specific format, runs the actual module and subsequently exports data from the module specific format.
• Alternatively, the general adapter can be configured to run three separate tasks in series. The first task is a call to the module adapter to import data to the module native format, the second is a call to the module executable itself and the third and final task is to the module adapter to export results from the native module format to the published interface format.

Of these two options, use of the second option is advocated, as it is then only the general adapter that is required to maintain control over an executable (thread) run by a processor, while the module adapter is only required for reading and writing data. In the first option both the general adapter and the module adapter need to control an executable (thread).

For each task (executable) that is initiated from the general adapter, a working location (URI path) can be defined that is set as the work directory for that executable. This location is again static and does not vary per module run.

Tasks initiated by the general adapter may be given a command line argument. This argument is as with the locations static, and can therefore not contain any dynamic settings (e.g. start of run time etc.).

Running Java Classes from DELFT-FEWS

It is also possible to call external java classes (*.jar) from the general adapter. Within this option it is NOT necessary to have the java *.jar file stored in the Delft-FEWS \bin\ directory. In this way, the dependency is clear and should it be seen as separate from the 'core' code of Delft-FEWS. The <command> element is important here in which you specify the full name of the Java runnable class and the option to specify its location in the <binDir>. The use of global properties is allowed. An example can be found below and more information can also be found here.

...
  <executeActivities>
	<executeActivity>
		<command>
			<className>nl.wldelft.fews.ftpPull.FtpPull</className>
			<binDir>%REGION_HOME%/Modules/FtpPull</binDir>
		</command>
		<arguments>
			<argument>-configFile</argument>
			<argument>$FTPPULL_CONFIG_FILE$</argument>
			<argument>-root</argument>
			<argument>%ROOT_DIR%</argument>
		</arguments>
		<timeOut>900000</timeOut>
	</executeActivity>
</executeActivities>
... 

User interaction

Modules called from the DELFT-FEWS system should in principle be defined such that these require no user interaction through menus and screens. This principle holds for all modules that are run from the system as a requirement in making an automatic forecast. These modules are then typically run on forecasting shell servers, i.e. dedicated servers for module execution. An exception to this rule can be made for modules run on the operator client that require manual interaction. The interface to these is from the point of view of the general adapter exactly the same as any other module. However, the module itself may require user interaction to complete the task required. An example is the interactive correlation utility.

Diagnostics

Should the module be run in a special diagnostic mode, then this can be achieved where applicable though command line arguments.

Data exchange between General Adapter and Modules

The prime means of communication between external modules and DELFT-FEWS through the general adapter is in the form of data exchange. For running modules in a dynamic situation, this data comprises mainly exchange of dynamic data (see also next sections).

Module input data

Exchange of data is primarily through time series. These are formatted as XML files. The data passed to the modules is in principle already validated by DELFT-FEWS and if applicable gaps in data are filled using the interpolation module. The modules will not be required to perform any data validation or interpolation to fill missing values. Depending on their usage, data series can be passed with or without missing values replaced.
For data series used as boundary conditions to modules, these must typically be complete and missing values are not allowed. The interpolation module (part of DELFT-FEWS) will then be configured such that this is guaranteed.
Data series used for data assimilation, the raw data must be passed. This entails that missing values are not replaced through interpolation. These series will, however, be validated using DELFT-FEWS functionality to remove outliers.
Each data point in a data series is supplied with an additional flag. This flag can be used as a quality flag, indicating for example if a gauge reading is in a range that makes its accuracy questionable. This is referred to as a soft limit, as opposed to a hard limit. Data exceeding hard limits is in principle removed by the validation module.

Module results

Results from external modules may be complete series, but may also contain missing values. For equidistant data series (where a time interval is defined), a time value not included in the file is considered as missing. A missing value constant may also be explicitly defined.

Dealing with module states

Module states can be configured to be exchanged between the central database and the module. These module states are handled in proprietary module formats, and the DELFT-FEWS system will not be able to make changes in module states. This would require extensive intelligence on the part of the DELFT-FEWS system on the syntax and semantics of module states. States are administered by the DELFT-FEWS system passively, with a time stamp associated with each module state. This time stamp indicates the time for which that state is valid. On running a module, the state associated with the start time of that run is made available through the module adapter.

For each module a default state is also administered which can be used in for example cold start conditions. This default state does not have a fixed time stamp. Besides this defaults state, the system can also administer a number of scenario states in the same way, where these do not have a definite time stamp. For some modules, there is a requirement that when the state is used to initiate a run, then that state must have within it the exact time stamp for which it is valid. If this is the case then the module adapter is responsible for setting that time stamp.

On completion of or during a module run module states can be exchanged through the general adapter. Each state carries in the XML file definition the time stamp with which it is associated.

The DELFT-FEWS system itself does not allow for amending values in system states. If a value from an external source (e.g. SMD values, reservoirs levels) is required to be amendable in the module state, then this is passed to the module a (non-equidistant) time series. The module adapter is responsible for incorporating these values into the module state in a meaningful way.

Module diagnostics

Communication of module diagnostics is through the applicable published interface formats. The general adapter starts a task, and on completion reads the output of that task. There is no provision for intermediate communication.

Module run information

There are two approaches in the exchange of run-time information to the module. This includes for example when start date/time of the module run and the end date/time of the module run. The first approach is through the exchange of time series whereby the module adapter derives the run information from the time series identified as inputs. The run should then start at the first time step in that timeseries, which typically coincides with the date/time of the state, and the last time step in the timeseries.

In some cases the multiple time series may be used as input to the module, and if these are of differing lengths a driving timeseries is required to be defined in the adapter.

Another approach is to export additional information from FEWS. The general adapter can be configured to export a run information file, which includes the start and end date of the run, the forecast start time (time0), as well as an optional indication of the time until observed data is available (where this is different to the time0). The file also includes specification of all input and output files as defined, with the full path names, as well as a configurable amount of key value pairs (synonomous to command line arguments).