You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 2 Current »

This page describes how to add PASTAS groundwater timeseries models to FEWS. PASTAS is a modelling framework developed by Artesia (contact: d.brakenhoff@artesia-water.nl) and based on Python.

How to derive/make PASTAS models is not covered on this page. Keep in mind that for 1 groundwater well location 1 PASTAS needs to be made because PASTAS is a timeseries model. Because PASTAS is a modelling framework this means a model is not set in stone and is highly flexible to what the modeller wants to achieve. Therefore it could be that when integrating it operationally in FEWS that you will encounter errors not previously encountered in deriving the model and using the model in model studies. You will have to make sure yourselves to apply correct error handling to these errors in the PASTAS Python model script.

Required external modules/binaries

  • Python 3.6 or higher. 
    • Use 'pip install pastas' to install the main package
    • Use 'pip install pastastore' to install the package that will manage the pastas project and timeseries
    • To import FEWS PI xml files you will have to install hydropandas using 'pip install hydropandas'
    • Optionally if you want to use PASTAS to validate groundwater data, use 'pip install traval'. The usage of traval is not covered on this page

Workflow

Below an example of a workflow to run PASTAS model(s).

<?xml version="1.0" encoding="UTF-8"?>
<workflow xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/workflow.xsd" version="1.1">
	<activity>
		<runIndependent>true</runIndependent>
		<moduleInstanceId>PASTAS_TRAVAL_validatie</moduleInstanceId>
		<moduleConfigFileName>PASTAS_prep</moduleConfigFileName>
	</activity>
		<activity>
		<runIndependent>true</runIndependent>
		<moduleInstanceId>PASTAS_TRAVAL_validatie</moduleInstanceId>
		<moduleConfigFileName>PASTAS_TRAVAL_validatie_Export_modellen</moduleConfigFileName>
	</activity>
	<activity>
		<runIndependent>true</runIndependent>
		<moduleInstanceId>PASTAS_TRAVAL_validatie</moduleInstanceId>
		<moduleConfigFileName>PASTAS_TRAVAL_validatie_Export_tijdreeksen</moduleConfigFileName>
		<loopLocationSetId>Grondwater_PASTAS</loopLocationSetId>
	</activity>
	<activity>
		<runIndependent>true</runIndependent>
		<moduleInstanceId>PASTAS_TRAVAL_validatie</moduleInstanceId>
		<moduleConfigFileName>PASTAS_TRAVAL_validatie_Run_validatie</moduleConfigFileName>
	</activity>
	<activity>
		<runIndependent>true</runIndependent>
		<moduleInstanceId>PASTAS_TRAVAL_validatie</moduleInstanceId>
		<moduleConfigFileName>PASTAS_TRAVAL_validatie_Import</moduleConfigFileName>
	</activity>
</workflow>

Prepare data for model(s)

PASTAS models typically are forced using precipitation and evaporation data and always start at some far point in the past (like since 1990) to get a correct model state for the current time period. In other cases, the model could be forced using groundwater levels from nearby groundwater wells or using river waterlevels. In this section we focus on the precipitation and evaporation.

Below a FEWS prep module using transformation to prepare the precipitation and evaporation data. In this case we use KNMI-IRIS data (because it is measured on many weather stations somewhat equally distributed in the Netherlands). Note: set startOverrulable and endOverrulable on 'true' to make sure when you specify a different model run period in FEWS that possibly missing data is added too. 

PASTAS_prep 
<?xml version="1.0" encoding="UTF-8"?>
<transformationModule version="1.0" xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/transformationModule.xsd">
	<!--VARIABELEN-->
	<!--Neerslag/verdamping-->
	<variable>
		<variableId>Neerslag_meting_KNMI_stations_valid</variableId>
		<timeSeriesSet>
			<moduleInstanceId>ImportWIWB</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>P.meting</parameterId>
			<qualifierId>gevalideerd</qualifierId>
			<locationSetId>KNMI-IRIS</locationSetId>
			<timeSeriesType>external historical</timeSeriesType>
			<timeStep unit="day" multiplier="1" timeZone="GMT-8"/>
			<relativeViewPeriod unit="day" start="-30" end="0" startOverrulable="true" endOverrulable="true"/>
			<readWriteMode>read only</readWriteMode>
		</timeSeriesSet>
	</variable>
	<variable>
		<variableId>Neerslag_meting_KNMI_stations_onvalid</variableId>
		<timeSeriesSet>
			<moduleInstanceId>ImportWIWB</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>P.meting</parameterId>
			<locationSetId>KNMI-IRIS</locationSetId>
			<timeSeriesType>external historical</timeSeriesType>
			<timeStep unit="day" multiplier="1" timeZone="GMT-8"/>
			<relativeViewPeriod unit="day" start="-30" end="0" startOverrulable="true" endOverrulable="true"/>
			<readWriteMode>read only</readWriteMode>
		</timeSeriesSet>
	</variable>
	<variable>
		<variableId>Neerslag_meting_KNMI_stations_temp</variableId>
		<timeSeriesSet>
			<moduleInstanceId>PASTAS_prep</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>P.meting</parameterId>
			<locationSetId>KNMI-IRIS</locationSetId>
			<timeSeriesType>temporary</timeSeriesType>
			<timeStep unit="day" multiplier="1" timeZone="GMT-8"/>
			<relativeViewPeriod unit="day" start="-30" end="0" startOverrulable="true" endOverrulable="true"/>
			<readWriteMode>add originals</readWriteMode>
		</timeSeriesSet>
	</variable>
	<variable>
		<variableId>Neerslag_meting_KNMI_stations</variableId>
		<timeSeriesSet>
			<moduleInstanceId>PASTAS_prep</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>P.meting</parameterId>
			<locationSetId>KNMI-IRIS</locationSetId>
			<timeSeriesType>external historical</timeSeriesType>
			<timeStep unit="day" multiplier="1" timeZone="GMT-8"/>
			<relativeViewPeriod unit="day" start="-30" end="0" startOverrulable="true" endOverrulable="true"/>
			<readWriteMode>add originals</readWriteMode>
		</timeSeriesSet>
	</variable>
	<variable>
		<variableId>Neerslag_grondwater_temp_temp</variableId>
		<timeSeriesSet>
			<moduleInstanceId>PASTAS_prep</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>P.meting</parameterId>
			<qualifierId>temp</qualifierId>
			<locationSetId>Grondwater_PASTAS</locationSetId>
			<timeSeriesType>temporary</timeSeriesType>
			<timeStep unit="day" multiplier="1" timeZone="GMT-8"/>
			<relativeViewPeriod unit="day" start="-30" end="0" startOverrulable="true" endOverrulable="true"/>
			<readWriteMode>add originals</readWriteMode>
		</timeSeriesSet>
	</variable>
	<variable>
		<variableId>Neerslag_grondwater_temp</variableId>
		<timeSeriesSet>
			<moduleInstanceId>PASTAS_prep</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>P.meting</parameterId>
			<locationSetId>Grondwater_PASTAS</locationSetId>
			<timeSeriesType>temporary</timeSeriesType>
			<timeStep unit="day" multiplier="1" timeZone="GMT-8"/>
			<relativeViewPeriod unit="day" start="-30" end="0" startOverrulable="true" endOverrulable="true"/>
			<readWriteMode>add originals</readWriteMode>
		</timeSeriesSet>
	</variable>
	<variable>
		<variableId>Neerslag_grondwater</variableId>
		<timeSeriesSet>
			<moduleInstanceId>PASTAS_prep</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>P.meting</parameterId>
			<locationSetId>Grondwater_PASTAS</locationSetId>
			<timeSeriesType>external historical</timeSeriesType>
			<timeStep unit="day" multiplier="1" timeZone="GMT-8"/>
			<relativeViewPeriod unit="day" start="-30" end="0" startOverrulable="true" endOverrulable="true"/>
			<readWriteMode>add originals</readWriteMode>
		</timeSeriesSet>
	</variable>
	<variable>
		<variableId>Verdamping_meting_KNMI_stations_valid</variableId>
		<timeSeriesSet>
			<moduleInstanceId>ImportWIWB</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>E.meting</parameterId>
			<locationSetId>KNMI-EVP</locationSetId>
			<timeSeriesType>external historical</timeSeriesType>
			<timeStep unit="day" timeZone="GMT+1"/>
			<relativeViewPeriod unit="day" start="-30" end="0" startOverrulable="true" endOverrulable="true"/>
			<readWriteMode>read only</readWriteMode>
		</timeSeriesSet>
	</variable>
	<variable>
		<variableId>Verdamping_meting_KNMI_stations_temp</variableId>
		<timeSeriesSet>
			<moduleInstanceId>PASTAS_prep</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>E.meting</parameterId>
			<locationSetId>KNMI-EVP</locationSetId>
			<timeSeriesType>temporary</timeSeriesType>
			<timeStep unit="day" timeZone="GMT+1"/>
			<relativeViewPeriod unit="day" start="-30" end="0" startOverrulable="true" endOverrulable="true"/>
			<readWriteMode>add originals</readWriteMode>
		</timeSeriesSet>
	</variable>
	<variable>
		<variableId>Verdamping_meting_KNMI_stations</variableId>
		<timeSeriesSet>
			<moduleInstanceId>PASTAS_prep</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>E.meting</parameterId>
			<locationSetId>KNMI-EVP</locationSetId>
			<timeSeriesType>external historical</timeSeriesType>
			<timeStep unit="day" timeZone="GMT+1"/>
			<relativeViewPeriod unit="day" start="-30" end="0" startOverrulable="true" endOverrulable="true"/>
			<readWriteMode>add originals</readWriteMode>
		</timeSeriesSet>
	</variable>
	<variable>
		<variableId>Verdamping_grondwater_temp_temp</variableId>
		<timeSeriesSet>
			<moduleInstanceId>PASTAS_prep</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>E.meting</parameterId>
			<qualifierId>temp</qualifierId>
			<locationSetId>Grondwater_PASTAS</locationSetId>
			<timeSeriesType>temporary</timeSeriesType>
			<timeStep unit="day" timeZone="GMT+1"/>
			<relativeViewPeriod unit="day" start="-30" end="0" startOverrulable="true" endOverrulable="true"/>
			<readWriteMode>add originals</readWriteMode>
		</timeSeriesSet>
	</variable>
	<variable>
		<variableId>Verdamping_grondwater_temp</variableId>
		<timeSeriesSet>
			<moduleInstanceId>PASTAS_prep</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>E.meting</parameterId>
			<locationSetId>Grondwater_PASTAS</locationSetId>
			<timeSeriesType>temporary</timeSeriesType>
			<timeStep unit="day" timeZone="GMT+1"/>
			<relativeViewPeriod unit="day" start="-30" end="0" startOverrulable="true" endOverrulable="true"/>
			<readWriteMode>add originals</readWriteMode>
		</timeSeriesSet>
	</variable>
	<variable>
		<variableId>Verdamping_grondwater</variableId>
		<timeSeriesSet>
			<moduleInstanceId>PASTAS_prep</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>E.meting</parameterId>
			<locationSetId>Grondwater_PASTAS</locationSetId>
			<timeSeriesType>external historical</timeSeriesType>
			<timeStep unit="day" timeZone="GMT+1"/>
			<relativeViewPeriod unit="day" start="-30" end="0" startOverrulable="true" endOverrulable="true"/>
			<readWriteMode>add originals</readWriteMode>
		</timeSeriesSet>
	</variable>
	<!--BEWERKINGEN-->
	<!--Neerslag/verdamping-->
	<transformation id="Merge valid, onvalid, meteo neerslag gemeten">
		<merge>
			<simple>
				<inputVariable>
					<variableId>Neerslag_meting_KNMI_stations_valid</variableId>
				</inputVariable>
				<inputVariable>
					<variableId>Neerslag_meting_KNMI_stations_onvalid</variableId>
				</inputVariable>
				<outputVariable>
					<variableId>Neerslag_meting_KNMI_stations_temp</variableId>
				</outputVariable>
			</simple>
		</merge>
	</transformation>
	<transformation id="Neerslag naar grondwaterputten (closestDistance)">
		<interpolationSpatial>
			<closestDistance>
				<inputVariable>
					<variableId>Neerslag_meting_KNMI_stations_temp</variableId>
				</inputVariable>
				<outputVariable>
					<variableId>Neerslag_grondwater_temp</variableId>
				</outputVariable>
			</closestDistance>
		</interpolationSpatial>
	</transformation>
	<transformation id="Neerslag naar grondwaterputten (IDW)">
		<interpolationSpatial>
			<inverseDistance>
				<inputVariable>
					<variableId>Neerslag_meting_KNMI_stations_temp</variableId>
				</inputVariable>
				<includeMissingValues>false</includeMissingValues>
				<searchRadius>1000000000000</searchRadius>
				<distanceGeoDatum>Rijks Driehoekstelsel</distanceGeoDatum>
				<inverseDistancePower>2</inverseDistancePower>
				<numberOfPoints>10</numberOfPoints>
				<outputVariable>
					<variableId>Neerslag_grondwater_temp_temp</variableId>
				</outputVariable>
			</inverseDistance>
		</interpolationSpatial>
	</transformation>
	<transformation id="Merge neerslag closestDistance en IDW">
		<merge>
			<simple>
				<inputVariable>
					<variableId>Neerslag_grondwater_temp</variableId>
				</inputVariable>
				<inputVariable>
					<variableId>Neerslag_grondwater_temp_temp</variableId>
				</inputVariable>
				<outputVariable>
					<variableId>Neerslag_grondwater_temp</variableId>
				</outputVariable>
			</simple>
		</merge>
	</transformation>
	<transformation id="Extrapoleer neerslag gemeten (default)">
		<interpolationSerial>
			<default>
				<inputVariable>
					<variableId>Neerslag_grondwater_temp</variableId>
				</inputVariable>
				<extrapolateDirection>both</extrapolateDirection>
				<outputVariable>
					<variableId>Neerslag_grondwater_temp</variableId>
				</outputVariable>
			</default>
		</interpolationSerial>
	</transformation>
	<transformation id="Interpoleer neerslag gemeten (default)">
		<interpolationSerial>
			<default>
				<inputVariable>
					<variableId>Neerslag_grondwater_temp</variableId>
				</inputVariable>
				<outputVariable>
					<variableId>Neerslag_grondwater</variableId>
				</outputVariable>
			</default>
		</interpolationSerial>
	</transformation>
	<transformation id="Verdamping naar grondwaterputten (closestDistance)">
		<interpolationSpatial>
			<closestDistance>
				<inputVariable>
					<variableId>Verdamping_meting_KNMI_stations_valid</variableId>
				</inputVariable>
				<outputVariable>
					<variableId>Verdamping_grondwater_temp</variableId>
				</outputVariable>
			</closestDistance>
		</interpolationSpatial>
	</transformation>
	<transformation id="Verdamping naar grondwaterputten (IDW)">
		<interpolationSpatial>
			<inverseDistance>
				<inputVariable>
					<variableId>Verdamping_meting_KNMI_stations_temp</variableId>
				</inputVariable>
				<includeMissingValues>false</includeMissingValues>
				<searchRadius>1000000000000</searchRadius>
				<distanceGeoDatum>Rijks Driehoekstelsel</distanceGeoDatum>
				<inverseDistancePower>2</inverseDistancePower>
				<numberOfPoints>10</numberOfPoints>
				<outputVariable>
					<variableId>Verdamping_grondwater_temp_temp</variableId>
				</outputVariable>
			</inverseDistance>
		</interpolationSpatial>
	</transformation>
	<transformation id="Merge verdamping closestDistance en IDW">
		<merge>
			<simple>
				<inputVariable>
					<variableId>Verdamping_grondwater_temp</variableId>
				</inputVariable>
				<inputVariable>
					<variableId>Verdamping_grondwater_temp_temp</variableId>
				</inputVariable>
				<outputVariable>
					<variableId>Verdamping_grondwater_temp</variableId>
				</outputVariable>
			</simple>
		</merge>
	</transformation>
	<transformation id="Extrapoleer verdamping gemeten (constant)">
		<interpolationSerial>
			<extrapolateConstant>
				<inputVariable>
					<variableId>Verdamping_grondwater_temp</variableId>
				</inputVariable>
				<extrapolateDirection>both</extrapolateDirection>
				<outputVariable>
					<variableId>Verdamping_grondwater_temp</variableId>
				</outputVariable>
			</extrapolateConstant>
		</interpolationSerial>
	</transformation>
	<transformation id="Interpoleer verdamping gemeten (linear)">
		<interpolationSerial>
			<linear>
				<inputVariable>
					<variableId>Verdamping_grondwater_temp</variableId>
				</inputVariable>
				<outputVariable>
					<variableId>Verdamping_grondwater</variableId>
				</outputVariable>
			</linear>
		</interpolationSerial>
	</transformation>
</transformationModule>

Run model(s)

Export data to model(s)

See an example below to export the timeseries data to the model(s) using loopLocationSetId / $LOOP_LOCATION_ID$. Notice that the precipitation and evaporation data is set with a RVP using <relativeViewPeriod unit="day" start="-15000" end="0" startOverrulable="false" endOverrulable="true"/> to force FEWS to always export the data from long in the past (since 1990). The element <omitEmptyFiles>true</omitEmptyFiles> is crucial if you expect to have no data on some groundwater well locations or if groundwater well locations are time-dependent in FEWS.

<?xml version="1.0" encoding="UTF-8"?>
<generalAdapterRun xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/generalAdapterRun.xsd">
	<general>
		<description>PASTAS TRAVAL validatie</description>
		<piVersion>1.23</piVersion>
		<rootDir>$PASTAS_TRAVAL_VALIDATION_MODULE_FOLDER$</rootDir>
		<workDir>%ROOT_DIR%</workDir>
		<exportDir>%ROOT_DIR%/from_fews</exportDir>
		<exportDataSetDir>%ROOT_DIR%</exportDataSetDir>
		<exportIdMap>IdExportPASTAS</exportIdMap>
		<importDir>%ROOT_DIR%</importDir>
		<importIdMap>IdImportPASTAS</importIdMap>
		<dumpFileDir>%REGION_HOME%/Dumpfiles</dumpFileDir>
		<dumpDir>%ROOT_DIR%</dumpDir>
		<diagnosticFile>%ROOT_DIR%/logs/diag.xml</diagnosticFile>
		<missVal>-9999.0</missVal>
		<timeZone>
			<timeZoneName>GMT+1</timeZoneName>
		</timeZone>
		<time0Format>yyyy-MM-dd</time0Format>
		<startDateTimeFormat>yyyy-MM-dd</startDateTimeFormat>
		<endDateTimeFormat>yyyy-MM-dd</endDateTimeFormat>
	</general>
	<activities>
		<exportActivities>
			<exportTimeSeriesActivity>
				<exportFile>Neerslag\$LOOP_LOCATION_ID$.xml</exportFile>
				<timeSeriesSets>
					<timeSeriesSet>
						<moduleInstanceId>PASTAS_prep</moduleInstanceId>
						<valueType>scalar</valueType>
						<parameterId>P.meting</parameterId>
						<locationId>$LOOP_LOCATION_ID$</locationId>
						<timeSeriesType>external historical</timeSeriesType>
						<timeStep unit="day" multiplier="1" timeZone="GMT-8"/>
						<relativeViewPeriod unit="day" start="-15000" end="0" startOverrulable="false" endOverrulable="true"/>
						<readWriteMode>read only</readWriteMode>
					</timeSeriesSet>
				</timeSeriesSets>
				<omitMissingValues>true</omitMissingValues>
				<omitEmptyTimeSeries>true</omitEmptyTimeSeries>
				<omitEmptyFiles>true</omitEmptyFiles>
			</exportTimeSeriesActivity>
			<exportTimeSeriesActivity>
				<exportFile>Verdamping\$LOOP_LOCATION_ID$.xml</exportFile>
				<timeSeriesSets>
					<timeSeriesSet>
						<moduleInstanceId>PASTAS_prep</moduleInstanceId>
						<valueType>scalar</valueType>
						<parameterId>E.meting</parameterId>
						<locationId>$LOOP_LOCATION_ID$</locationId>
						<timeSeriesType>external historical</timeSeriesType>
						<timeStep unit="day" timeZone="GMT+1"/>
						<relativeViewPeriod unit="day" start="-15000" end="0" startOverrulable="false" endOverrulable="true"/>
						<readWriteMode>read only</readWriteMode>
					</timeSeriesSet>
				</timeSeriesSets>
				<omitMissingValues>true</omitMissingValues>
				<omitEmptyTimeSeries>true</omitEmptyTimeSeries>
				<omitEmptyFiles>true</omitEmptyFiles>
			</exportTimeSeriesActivity>
			<exportTimeSeriesActivity>
				<exportFile>Grondwaterstanden\$LOOP_LOCATION_ID$.xml</exportFile>
				<timeSeriesSets>
					<timeSeriesSet>
						<moduleInstanceId>ImportKeller</moduleInstanceId>
						<valueType>scalar</valueType>
						<parameterId>Stijghoogte.PASTAS.ongevalideerd</parameterId>
						<locationId>$LOOP_LOCATION_ID$</locationId>
						<timeSeriesType>external historical</timeSeriesType>
						<timeStep unit="hour"/>
						<relativeViewPeriod unit="day" start="-2" end="0" startOverrulable="true" endOverrulable="true"/>
						<readWriteMode>read only</readWriteMode>
					</timeSeriesSet>
				</timeSeriesSets>
				<omitMissingValues>true</omitMissingValues>
				<omitEmptyTimeSeries>true</omitEmptyTimeSeries>
				<omitEmptyFiles>true</omitEmptyFiles>
			</exportTimeSeriesActivity>
			<exportTimeSeriesActivity>
				<exportFile>Referentiehoogte\$LOOP_LOCATION_ID$.xml</exportFile>
				<timeSeriesSets>
					<timeSeriesSet>
						<moduleInstanceId>Bewerk_Grondwater</moduleInstanceId>
						<valueType>scalar</valueType>
						<parameterId>Referentie.hoogte</parameterId>
						<locationId>$LOOP_LOCATION_ID$</locationId>
						<timeSeriesType>external historical</timeSeriesType>
						<timeStep unit="nonequidistant"/>
						<relativeViewPeriod unit="day" start="-2" end="0" startOverrulable="true" endOverrulable="true"/>
						<readWriteMode>read only</readWriteMode>
					</timeSeriesSet>
				</timeSeriesSets>
				<omitMissingValues>true</omitMissingValues>
				<omitEmptyTimeSeries>true</omitEmptyTimeSeries>
				<omitEmptyFiles>true</omitEmptyFiles>
			</exportTimeSeriesActivity>
			<exportTimeSeriesActivity>
				<exportFile>Inhangdiepte\$LOOP_LOCATION_ID$.xml</exportFile>
				<timeSeriesSets>
					<timeSeriesSet>
						<moduleInstanceId>Bewerk_Grondwater</moduleInstanceId>
						<valueType>scalar</valueType>
						<parameterId>Inhang.diepte</parameterId>
						<locationId>$LOOP_LOCATION_ID$</locationId>
						<timeSeriesType>external historical</timeSeriesType>
						<timeStep unit="nonequidistant"/>
						<relativeViewPeriod unit="day" start="-2" end="0" startOverrulable="true" endOverrulable="true"/>
						<readWriteMode>read only</readWriteMode>
					</timeSeriesSet>
				</timeSeriesSets>
				<omitMissingValues>true</omitMissingValues>
				<omitEmptyTimeSeries>true</omitEmptyTimeSeries>
				<omitEmptyFiles>true</omitEmptyFiles>
			</exportTimeSeriesActivity>
		</exportActivities>
	</activities>
</generalAdapterRun>

Export location attributes

You can use the exportLocationAttributesCsvActivity in the General Adapter to export location attributes in a csv file to the model(s) (particular useful when using PASTAS for validation).

<exportLocationAttributesCsvActivity>
	<exportFile>stijgbuis_parameters.csv</exportFile>
	<locationSetId>Grondwater_PASTAS</locationSetId>
	<locationIdColumn columnName="Location"/>
	<attributeColumn columnName="X" attributeId="X_number"/>
	<attributeColumn columnName="Y" attributeId="Y_number"/>
	<attributeColumn columnName="BOVENKANT_BUIS" attributeId="BOVENKANTBUIS"/>
	<attributeColumn columnName="MAAIVELD" attributeId="MAAIVELD"/>
	<attributeColumn columnName="ONDERKANT_FILTER" attributeId="HARD_MIN"/>
	<attributeColumn columnName="BOVENKANT_FILTER" attributeId="HARD_MAX"/>
	<attributeColumn columnName="HARD_MIN" attributeId="HARD_MIN"/>
	<attributeColumn columnName="HARD_MAX" attributeId="HARD_MAX"/>
</exportLocationAttributesCsvActivity>

Run model(s)

A possible General Adapter module configuration could look like this to run the model(s) with Python. See attached run_validation.bat

<?xml version="1.0" encoding="UTF-8"?>
<generalAdapterRun xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/generalAdapterRun.xsd">
	<general>
		<description>PASTAS TRAVAL validatie</description>
		<rootDir>$PASTAS_TRAVAL_VALIDATION_MODULE_FOLDER$</rootDir>
		<workDir>%ROOT_DIR%</workDir>
		<exportDir>%ROOT_DIR%</exportDir>
		<exportDataSetDir>%ROOT_DIR%</exportDataSetDir>
		<exportIdMap>IdExportPASTAS</exportIdMap>
		<importDir>%ROOT_DIR%</importDir>
		<importIdMap>IdImportPASTAS</importIdMap>
		<dumpFileDir>%REGION_HOME%/Dumpfiles</dumpFileDir>
		<dumpDir>%ROOT_DIR%</dumpDir>
		<diagnosticFile>%ROOT_DIR%/logs/diag.xml</diagnosticFile>
		<missVal>NaN</missVal>
		<timeZone>
			<timeZoneName>CET</timeZoneName>
		</timeZone>
		<time0Format>yyyy-MM-dd</time0Format>
		<startDateTimeFormat>yyyy-MM-dd</startDateTimeFormat>
		<endDateTimeFormat>yyyy-MM-dd</endDateTimeFormat>
	</general>
	<activities>
		<executeActivities>
			<executeActivity>
				<command>
					<executable>%ROOT_DIR%/bin/run_validation.bat</executable>
				</command>
				<arguments>
					<argument>$PYTHON_EXE$</argument>
				</arguments>
				<timeOut>12800000</timeOut>
			</executeActivity>
		</executeActivities>
		<shutDownActivities>
			<!--Purge activities to free up disk space because exported input PI xml files take up a lot of storage space-->
			<purgeActivity>
				<filter>%ROOT_DIR%/data/pastas_db/models/*.*</filter>
			</purgeActivity>
			<purgeActivity>
				<filter>%ROOT_DIR%/from_fews/*.*</filter>
			</purgeActivity>
			<purgeActivity>
				<filter>%ROOT_DIR%/from_fews/Grondwaterstanden/*.*</filter>
			</purgeActivity>
			<purgeActivity>
				<filter>%ROOT_DIR%/from_fews/Referentiehoogte/*.*</filter>
			</purgeActivity>
			<purgeActivity>
				<filter>%ROOT_DIR%/from_fews/Inhangdiepte/*.*</filter>
			</purgeActivity>
			<purgeActivity>
				<filter>%ROOT_DIR%/from_fews/Neerslag/*.*</filter>
			</purgeActivity>
			<purgeActivity>
				<filter>%ROOT_DIR%/from_fews/Verdamping/*.*</filter>
			</purgeActivity>
		</shutDownActivities>
	</activities>
</generalAdapterRun>

Important PASTAS Python script code blocks

To import FEWS PI xml data into the PASTAS model(s), use the following code block as inspiration. Notice that the data is imported using the 'hydropandas' package and added to a pastastore object.

Important: the hpd.ObsCollection.from_fews_xml() function is written by Artesia and is based on the pi xml functions written for RTC-Tools 2

import os
import hydropandas as hpd
import pastastore as pst


pstore2 = pst.PastaStore("online", pst.DictConnector("online"))


oc_rawdir = "Grondwaterstanden"
oc_raw = hpd.ObsCollection.from_fews_xml(
            os.path.join(settings["inputdir"], oc_rawdir),
            ObsClass=hpd.GroundwaterObs,
            name="head",
            to_mnap=False,
            remove_nan=False,
            verbose=False
        )


oc_raw.to_pastastore(pstore=pstore2, obs_column="value",
                     kind="oseries", verbose=False, overwrite=True)

Import model results

The PASTAS model results are typically written to csv files per groundwater well location. The location id is in the filename typically.

Below an example import module (which is used in a validation workflow hence the first import block).

<?xml version="1.0" encoding="UTF-8"?>
<timeSeriesImportRun xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/timeSeriesImportRun.xsd">
	<import>
		<general>
			<importType>generalCsv</importType>
			<folder>$PASTAS_TRAVAL_VALIDATION_MODULE_FOLDER$/to_fews</folder>
			<fileNamePatternFilter>*_val.csv</fileNamePatternFilter>
			<fileNameLocationIdPattern>(.*)_val.csv</fileNameLocationIdPattern>
			<deleteImportedFiles>false</deleteImportedFiles>
			<table>
				<dateTimeColumn name="datetime" pattern="yyyy-MM-dd HH:mm:ss"/>
				<valueColumn name="values" parameterId="Stijghoogte.PASTAS.gevalideerd"/>
				<flagColumn name="flag"/>
				<flagSourceColumn name="flagsource"/>
				<commentColumn name="comment"/>
			</table>
			<idMapId>IdPASTAS_validatie</idMapId>
			<flagConversionsId>ImportPASTAS_TRAVALFlagConversions</flagConversionsId>
			<missingValue>-9999.0</missingValue>
			<missingValue>NaN</missingValue>
			<importTimeZone>
				<timeZoneOffset>+01:00</timeZoneOffset>
			</importTimeZone>
			<disableDataFeedInfo/>
			<columnSeparator>,</columnSeparator>
			<decimalSeparator>.</decimalSeparator>
		</general>
		<timeSeriesSet>
			<moduleInstanceId>ImportKeller</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>Stijghoogte.PASTAS.gevalideerd</parameterId>
			<locationSetId>Grondwater_PASTAS</locationSetId>
			<timeSeriesType>external historical</timeSeriesType>
			<timeStep unit="hour"/>
			<readWriteMode>add originals</readWriteMode>
		</timeSeriesSet>
	</import>
	<import>
		<general>
			<importType>generalCsv</importType>
			<folder>$PASTAS_TRAVAL_VALIDATION_MODULE_FOLDER$/to_fews</folder>
			<fileNamePatternFilter>*_sim.csv</fileNamePatternFilter>
			<fileNameLocationIdPattern>(.*)_sim.csv</fileNameLocationIdPattern>
			<deleteImportedFiles>false</deleteImportedFiles>
			<table>
				<dateTimeColumn name="datetime" pattern="yyyy-MM-dd HH:mm:ss"/>
				<valueColumn name="sim_meteo" parameterId="Stijghoogte.PASTAS.gesimuleerd"/>
				<valueColumn name="sim_meteo_lower" parameterId="Stijghoogte.PASTAS.gesimuleerd.onder"/>
				<valueColumn name="sim_meteo_upper" parameterId="Stijghoogte.PASTAS.gesimuleerd.boven"/>
			</table>
			<idMapId>IdPASTAS_validatie</idMapId>
			<flagConversionsId>ImportPASTAS_TRAVALFlagConversions</flagConversionsId>
			<missingValue>-9999.0</missingValue>
			<missingValue>NaN</missingValue>
			<importTimeZone>
				<timeZoneOffset>+01:00</timeZoneOffset>
			</importTimeZone>
			<disableDataFeedInfo/>
			<columnSeparator>,</columnSeparator>
			<decimalSeparator>.</decimalSeparator>
		</general>
		<timeSeriesSet>
			<moduleInstanceId>ImportKeller</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>Stijghoogte.PASTAS.gesimuleerd</parameterId>
			<locationSetId>Grondwater_PASTAS</locationSetId>
			<timeSeriesType>external historical</timeSeriesType>
			<timeStep unit="hour"/>
			<readWriteMode>add originals</readWriteMode>
		</timeSeriesSet>
		<timeSeriesSet>
			<moduleInstanceId>ImportKeller</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>Stijghoogte.PASTAS.gesimuleerd.boven</parameterId>
			<locationSetId>Grondwater_PASTAS</locationSetId>
			<timeSeriesType>external historical</timeSeriesType>
			<timeStep unit="hour"/>
			<readWriteMode>add originals</readWriteMode>
		</timeSeriesSet>
		<timeSeriesSet>
			<moduleInstanceId>ImportKeller</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>Stijghoogte.PASTAS.gesimuleerd.onder</parameterId>
			<locationSetId>Grondwater_PASTAS</locationSetId>
			<timeSeriesType>external historical</timeSeriesType>
			<timeStep unit="hour"/>
			<readWriteMode>add originals</readWriteMode>
		</timeSeriesSet>
	</import>
</timeSeriesImportRun>


  • No labels

Disclaimer