How to add a custom transformation to my FEWS system?

Adding a custom transformation involves several steps which will be explained in detail in the upcoming section:

  1. writing the code for the custom transformation.
  2. configuring the custom transformation,

How to write a custom transformation?

First step is to write the actual code for the transformation. Running and debugging the actual transformation can be done by writing unit tests.

Lets say we want to write a simple transformation that calculates the output value by using the following formula:

output = input1 x correctionFactor1 + input2 x correctionFactor2

The code needed for such a transformation would be quite basic and is given below:

package example;

import nl.wldelft.fews.openapi.transformationmodule.Calculation;
import nl.wldelft.fews.openapi.transformationmodule.Input;
import nl.wldelft.fews.openapi.transformationmodule.Output;
import nl.wldelft.util.timeseries.Variable;
import org.apache.log4j.Logger;

public class Example1 implements Calculation {
    private static final Logger log = Logger.getLogger(Example1.class.getName());

    @Input
    float option1;

    @Input
    float option2;

    @Input
    Variable input1 = null;

    @Input
    Variable input2 = null;

    @Output
    Variable output = null;

    @Override
    public void calculate() throws Exception {
        if (Float.isNaN(input1.value) || Float.isNaN(input2.value)) return;
        log.info("Input is " + input1.value + " and " + input2.value);
        output.value = (input1.value * option1 + input2.value * option2) / 2;
        log.info("Output is " + output.value);
    }
}


Now the code above will explained step by step to understand the details. First note that transformation implements the interface Calculation.

The interface Calculation has one method void calculate().

package nl.wldelft.fews.openapi.transformationmodule;

import nl.wldelft.util.Initializable;
import nl.wldelft.util.TimeZeroConsumer;

public interface Calculation extends Function {

    /**
     * This method is called by the framework to do the actual calculations.
     * The implementing class should get the input from its InputVariable
     * fields and put the calculation output in its OutputVariable fields.
     * The framework will initialize the InputVariable and OutputVariable
     * fields and set the input values in the InputVariables before calling
     * this method and get the output values out of the OutputVariables
     * afterwards.
     *
     * @throws Exception
     */
    void calculate() throws Exception;
}

The method calculate() will contain the actual code for the transformation. The FEWS Transformation framework will automaticly invoke this method.

Once the workflow containing the custom transformation is executed. The java code in the calculate method is self explaining will not be explained in more detail here.

Next step is to explain the correctionfactors and the input values used in the calculate-method.

They are both defined as fields in the java-class.

@Input
float option1;

@Input
float option2;

@Input
Variable input1 = null;

@Input
Variable input2 = null;

The fields option1 and option2 however dont have any values assigned! How can they have a value during runtime?
The answer is that the FEWS Transformation framework will inject the values at runtime in these variables. The actual values
are defined in the configuration of the transformation. In the upcoming section, the configuration will be explained in detail.

The input timeseries are defined as input1 and input2. They are marked as being input timeseries by the annotation @Input.
It is important to note that input timeseries can be defined as a:

  1. Variable (as is done in the example above)
  2. TimSeriesArray
  3. Variable[]
  4. TimeSeriesArray[]

The major difference between the use of the class Variable and TimeSeriesArray is that Variable is used when the output is only dependend on input values

at the same timestep. For example when the transformation is calculating the discharge from the stage by using a rating curve than the output is only dependend of

the input value at the same time step. However when a transformation is calculating the aggregation of the input time series over a certain period than the output

is dependend on other values at other time steps in the input time series. In this case one should use the TimeSeriesArray.

The same logic applies to the cases when a Variable[] or TimeSeriesArray[] is used. The difference between a Variable and a Variable[] is that when in the configuration several

timeseries are defined in the timeseriesset config a Variable[] is used. For example by configuring a locationSet in the timeSeriesSet. When the timeseriesset only defines a single

timeseries than a Variable is used.

In our example the output is calculated from the input values at the same time step. Therefore we define the input time series as Variable.

The output is marked by the annotation @Output

The output can be:

  1. Variable
  2. TimeSeriesArray

In our example the output (just as the input) is defined as a Variable.

@Output
Variable output = null;

Previously it was explained that the options are injected during runtime in the correction factor fields. Something similar is done for the input time series.

If the transformation, for example, is supposed to run for a period of a day and the input and output time series are both hourly time series than this

calculation must be executed 24 times. In the case that the output time series is defined as a Variable than the FEWS transformation framework will execute the

calculate() method in the transformation 24 times. One time for each time step in the output period.  After each execution the calculated output value, which should be

stored in the output Variable in the value field is read by the framework and stored in the database.

The values of the input time series are assigned to the input1 and input2 fields prior to invoking the calculate() method.

When the output is defined as TimeSeriesArray than the calculate() method will be only called once. Prior to executing the method the entire input time series is injected

in the input (Note than when the output is defined as a TimeSeriesArray, the input should also be defined as a TimeSeriesArray). The example code we are using, could for example also

be writting with the input and output defined as a TimeSeriesArray.

The example code of such a transformation is given below.

The example code is given below.

package example;

import nl.wldelft.fews.openapi.transformationmodule.Calculation;
import nl.wldelft.fews.openapi.transformationmodule.Input;
import nl.wldelft.fews.openapi.transformationmodule.Output;
import nl.wldelft.util.timeseries.TimeSeriesArray;
import org.apache.log4j.Logger;

public class Example2 implements Calculation {
    private static final Logger log = Logger.getLogger(Example1.class.getName());

    @Input
    private float option1;

    @Input
    private float option2;

    @Input
    TimeSeriesArray input1 = null;

    @Input
    TimeSeriesArray input2 = null;

    @Output
    TimeSeriesArray output = null;

    @Override
    public void calculate() throws Exception {
        for (int i = 0; i < output.size(); i++) {
            long time = output.getTime(i);

            int index1 = input1.indexOfTime(time);
            if (index1 == -1) continue;
            float value1 = input1.getValue(index1);

            int index2 = input2.indexOfTime(time);
            if (index2 == -1) continue;
            float value2 = input2.getValue(index2);

            if (Float.isNaN(value1) || Float.isNaN(value2)) return;
            log.info("Input is " + value1 + " and " + value2);
            float outputValue = (value1 * option1 + value2 * option2) / 2;
            log.info("Output is " + outputValue);
            output.setValue(i, outputValue);
        }
    }
}

The main difference between the first and the second example is that in the second example the code needs
to have a for loop to calculate the output value for each time step available in the output time series. In the first
example the FEWS transformation framework takes care of the loop.

How to configure a custom transformation?

The section above explained the coding of a custom transformation. In addition to the coding the custom transformation should be added

to a workflow so that it can be used in the FEWS system.

Below you can find an example of a custom fews transformation.

<?xml version="1.0" encoding="UTF-8"?>
<transformationModule xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/transformationModule.xsd" version="1.0">
        <variable>
		<variableId>input1</variableId>
	       <timeSeriesSet>
                <moduleInstanceId>POFlathead_MergeMAP_UpdateStates</moduleInstanceId>
                <valueType>scalar</valueType>
                <parameterId>MAP</parameterId>
                <locationId>WGCM8U</locationId>
                <timeSeriesType>simulated historical</timeSeriesType>
                <timeStep unit="hour" multiplier="6"/>
                <relativeViewPeriod unit="week" start="-520" end="0"/>
                <readWriteMode>add originals</readWriteMode>
       </timeSeriesSet>
       </variable>
	<variable>
		<variableId>input2</variableId>
		<timeSeriesSet>
        <moduleInstanceId>POFlathead_MergeMAP_UpdateStates</moduleInstanceId>
        <valueType>scalar</valueType>
        <parameterId>MAP</parameterId>
        <locationId>WGCM8L</locationId>
        <timeSeriesType>simulated historical</timeSeriesType>
        <timeStep unit="hour" multiplier="6"/>
           <relativeViewPeriod unit="week" start="-520" end="0"/>
        <readWriteMode>add originals</readWriteMode>
    </timeSeriesSet>
	</variable>
	<variable>
		<variableId>output</variableId>
		<timeSeriesSet>
			<moduleInstanceId>example1</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>MAP</parameterId>
			<locationId>WGCM8</locationId>
			<timeSeriesType>simulated forecasting</timeSeriesType>
			<timeStep unit="hour" multiplier="6"/>
			    <relativeViewPeriod unit="week" start="-520" end="0"/>
			<readWriteMode>add originals</readWriteMode>
		</timeSeriesSet>
	</variable>
	<transformation id="Example1">
<custom>
	<userDefined>
		<input>
			<fieldName>input1</fieldName>
			<inputVariable>
				<variableId>input1</variableId>
			</inputVariable>
		</input>
		<input>
			<fieldName>input2</fieldName>
			<inputVariable>
				<variableId>input2</variableId>
			</inputVariable>
		</input>
		<options>
			<float value="0.3" key="option1"></float>
			<float value="0.7" key="option2"></float>
		</options>
		<output>
			<fieldName>output</fieldName>
			<outputVariable>
				<variableId>output</variableId>
			</outputVariable>
		</output>
		<binDir>example</binDir>
		<className>example.Example1</className>
	</userDefined>
</custom>
	</transformation>
</transformationModule>

In this example we showed a custom transformation which uses the example function in the section above.
In the className the name of the implementing class is defined. The binDir section is used to define
the directory which contains the jar(s) with the implementing class and its helper classes.

The section options is used to define optional fields.

Optional fields can be of type:

  • String,
  • int,
  • float,
  • boolean.

The optional fields have a name. By using the name defined in the xml-config a field with the same name is looked up in the java-class.

Secondly a check is done to verify that the type defined int the xml-config is the same as the type defined in the java class. If the check

succeeds the value is injected into the field.

The output section defined the output time series. A output variable (identified by @Output) is searched for in the java class which has the same

name as the name identified in the tag fieldName. If such a variable is found it is linked to the time series configured in the xml file.

The input section defines the input time series. The same logic as is used for the output time series is used here for the output time series.

Using location attributes in a custom transformation

In order to access location attributes in a custom trasformation a couple of things need to be done.

  • The attibuteId's should be passed via the options as string key / value pairs
  • Have \@Input fields with the same name as the key from the string options
  • The custom transformation should implement the interface LocationAttributeValuesProviderConsumer
  • Store the LocationAttributeValuesProvider from the setLocationAttributeValuesProvider method as a field 
  • Get the attribute values from the LocationAttributeValuesProvider via the getXXXValue (single value) or getXXXValues (multivalued) methods

Example config:

Example config for passing attribute id's to custom transformation
<?xml version="1.0" encoding="UTF-8"?>
<transformationModule version="1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.wldelft.nl/fews" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/transformationModule.xsd">
	<variable>
		<variableId>input</variableId>
		<timeSeriesSet>
			<moduleInstanceId>UserDefinedFunctionTest</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>Q.m</parameterId>
			<locationSetId>locationAttributeTest</locationSetId>
			<timeSeriesType>external historical</timeSeriesType>
			<timeStep unit="day" multiplier="1"/>
			<relativeViewPeriod unit="day" start="0" end="1"/>
			<readWriteMode>editing visible to all future task runs</readWriteMode>
		</timeSeriesSet>
	</variable>
	<variable>
		<variableId>output</variableId>
		<timeSeriesSet>
			<moduleInstanceId>UserDefinedFunctionTest</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>Q.dis</parameterId>
			<locationSetId>locationAttributeTest</locationSetId>
			<timeSeriesType>external historical</timeSeriesType>
			<timeStep unit="day" multiplier="1"/>
			<relativeViewPeriod unit="day" start="0" end="1"/>
			<readWriteMode>editing visible to all future task runs</readWriteMode>
		</timeSeriesSet>
	</variable>
	<transformation id="userDefinedFunctionTestWithLocationAttributes">
		<custom>
			<userDefined>
				<input>
					<fieldName>input</fieldName>
					<inputVariable>
						<variableId>input</variableId>
					</inputVariable>
				</input>
				<options>
					<string key="stringAttributeID" value="stringAttributeKey"></string>
					<string key="booleanAttributeID" value="booleanAttributeKey"></string>
					<string key="dateTimeAttributeID" value="dateTimeAttributeKey"></string>
					<string key="doubleAttributeID" value="doubleAttributeKey"></string>
				</options>
				<output>
					<fieldName>output</fieldName>
					<outputVariable>
						<variableId>output</variableId>
					</outputVariable>
				</output>
				<className>nl.wldelft.fews.openapi.transformationmodule.CustomTestFunctionWithLocationAttributes</className>
			</userDefined>
		</custom>
	</transformation>
</transformationModule>

Example Java class:

Example Java class for custom transformation processing location attributes
package nl.wldelft.fews.openapi.transformationmodule;

import nl.wldelft.util.timeseries.Variable;

import java.util.Arrays;

public class CustomTestFunctionWithLocationAttributes implements Calculation, LocationAttributeValuesProviderConsumer {
    @Input
    Variable input = null;

    @Input
    String stringAttributeID = null;
    @Input
    String booleanAttributeID = null;
    @Input
    String dateTimeAttributeID = null;
    @Input
    String doubleAttributeID = null;

    @Output
    Variable output = null;

    private LocationAttributeValuesProvider locationAttributeValuesProvider = null;

    @Override
    public void calculate() throws Exception {
        String locationId = input.header.getLocationId();

        // Single value
        locationAttributeValuesProvider.getBooleanValue(locationId, booleanAttributeID);
        locationAttributeValuesProvider.getStringValue(locationId, stringAttributeID);
        locationAttributeValuesProvider.getNumericValue(locationId, doubleAttributeID);
        locationAttributeValuesProvider.getDateTimeValue(locationId, dateTimeAttributeID);

        // Multivalued
        locationAttributeValuesProvider.getBooleanValues(locationId, booleanAttributeID);
        locationAttributeValuesProvider.getStringValues(locationId, stringAttributeID);
        locationAttributeValuesProvider.getNumericValues(locationId, doubleAttributeID);
        locationAttributeValuesProvider.getDateTimeValues(locationId, dateTimeAttributeID);
    }

    @Override
    public void setLocationAttributeValuesProvider(LocationAttributeValuesProvider locationAttributeValuesProvider) {
        this.locationAttributeValuesProvider = locationAttributeValuesProvider;
    }
}

Specifying whether unreliable data should be used as input

Since 2023.02 custom transformations can implement the UseUnreliableInputValueFunction to determine whether or not unreliable data is allowed as input for the transformation.

When this interface is implemented the method useUnreliables() must be specified. A boolean must be returned which defines whether or not unreliable data should be included in the input.

Unfortunately it is not possible to use a configured option to determine if unreliables are allowed, because the usage of unreliables are handled before the configured options are processed. 

Example Java class for custom transformation processing location attributes
package nl.wldelft.fews.openapi.transformationmodule;

import nl.wldelft.util.timeseries.TimeSeriesArray;

public class CustomIncludeUnreliablesTestFunction implements Calculation, UseUnreliableInputValueFunction {

    @Input
    TimeSeriesArray inputSeries = null;
    @Output
    TimeSeriesArray outputSeries = null;

    @Override
    public boolean useUnreliables() {
        return true;
    }

    @Override
    public void calculate() throws Exception {
        for (int i = 0, size = inputSeries.size(); i < size; i++) {
            long time = inputSeries.getTime(i);
            outputSeries.putValue(time, inputSeries.getValue(i));
            int indexOfTime = outputSeries.indexOfTime(time);
            outputSeries.setFlag(indexOfTime, inputSeries.getFlag(i));
        }
    }
}
  • No labels

Disclaimer