It is possible to edit archived data in FEWS and subsequently upload it back to the archive. The editing can be done in the Time Series Dialog ( / ), just as you would normally do. Subsequently, the edited data can be uploaded to the Open Archive. This can be done manually or automatically, depending on the configuration and whether the user is using a Stand Alone or Operator client. The data edits will be uploaded in pi-xml format to the webservice of the Archive. The webservice will save the uploaded pi-xml files in the folder upload/edit. To process these files a special task in the archive export module is available.

For a user guide on how to upload the data, please look at: 21 Archive display#21Archivedisplay-Uploadtoarchive.

This section will discuss the configuration to allow this.

Table of contents

Prerequisites

We assume the following is available already, before you start to add the configuration to allow uploads of edited data to the Archive.

  • Delft-FEWS Open Archive installed
    • Server is running and accessible
    • Basic archive configuration is in place
    • Some data has been archived
  • Stand alone or Operator Client
  • Time series is editable (timeSeriesType = external historical, and readWriteMode = add originals)
  • Time series is available in a filter
  • User has the right permissions to edit the data
  • Locations defined in a CSV/Shape file in MapLayerFiles (so not in locations.xml)

Config files requiring updates

Once the prerequisites are met, the following files require an update, for each an example will be provided at the bottom of the page.

Note that names could be different based on your configuration.

  • DisplayConfigFiles/
    • ArchiveModuleDisplay.xml
  • MapLayerFiles/
    • locations.csv
    • moduleInstanceDescriptors.csv
  • RegionConfigFiles/
    • WorkflowDescriptors.xml
  • SystemConfigFiles/
    • Archives.xml

The following files should be added:

  • ModuleConfigFiles/
    • MergeEdits.xml
  • WorkflowFiles/
    • Merge_Edits.xml

Archives.xml

<?xml version="1.0" encoding="UTF-8"?>
<archives xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews https://fewsdocs.deltares.nl/schemas/version1.0/archives.xsd">
	<enabled>$ArchiveEnabled$</enabled>
	<catalogueUrl>$ArchiveUrl$/deltares-archive-server/catalogue</catalogueUrl>
	<fileServerUrl>$ArchiveUrl$/thredds/fileServer/data</fileServerUrl>
	<elasticSearchUrl>$ArchiveUrl$/deltares-archive-server/elastic</elasticSearchUrl>
	<areaLocationAttributeFunction>@ArchiveArea@</areaLocationAttributeFunction>
	<uploadEditedDataToOpenArchive>true</uploadEditedDataToOpenArchive>
</archives>

In the Archives.xml the following configuration needs to be added:

	<areaLocationAttributeFunction>@ArchiveArea@</areaLocationAttributeFunction>

Optionally the UploadEditedData element can be added to automatically update the data when editing in an OC.

	<uploadEditedDataToOpenArchive>true</uploadEditedDataToOpenArchive>

Location attributes

Note that the attribute should be available for the location for which you want to edit the data, as well as properly configured in the locations.csv and locationsSets.xml

	<locationSet id="DummyLocations">
		<csvFile>
			<file>DummyLocations.csv</file>
			<geoDatum>WGS 1984</geoDatum>
			<id>%FewsId%</id>
			<parentLocationId>%ParentId%</parentLocationId>
			<x>%X%</x>
			<y>%Y%</y>
			<attribute id="ArchiveArea">
				<text>%ArchiveArea%</text>
			</attribute>
		</csvFile>
	</locationSet>

ArchiveModuleDisplay.xml

<?xml version="1.0" encoding="UTF-8"?>
<archiveModuleDisplay xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews https://fewsdocs.deltares.nl/schemas/version1.0/archiveModuleDisplay.xsd">
	<workFolder>$ArchiveDownloadFolder$/workdir</workFolder>
	<downloadFolders>
		<downloadFolderObserved>$ArchiveDownloadFolder$/observed</downloadFolderObserved>
		<downloadFolderSimulated>$ArchiveDownloadFolder$/simulated</downloadFolderSimulated>
		<downloadFolderSimulatedHistorical>$ArchiveDownloadFolder$/simulatedhistorical</downloadFolderSimulatedHistorical>
		<downloadFolderExternalForecasts>$ArchiveDownloadFolder$/externalforecasts</downloadFolderExternalForecasts>
		<downloadFolderRatingCurves>$ArchiveDownloadFolder$/ratingcurves</downloadFolderRatingCurves>
		<downloadFolderHistoricalEvents>$ArchiveDownloadFolder$/historicalevents</downloadFolderHistoricalEvents>
		<downloadFolderForecasterNotes>$ArchiveDownloadFolder$/messages</downloadFolderForecasterNotes>
		<downloadFolderConfiguration>$ArchiveDownloadFolder$/configuration</downloadFolderConfiguration>
		<downloadFolderEventAttachments>$ArchiveDownloadFolder$/attachments</downloadFolderEventAttachments>
	</downloadFolders>
	<archiveImportWorkflowId>Archive_Import_Data</archiveImportWorkflowId>
	<hideArchiveUploadTab>false</hideArchiveUploadTab>
	<archiveUploadIdMapId>IdMapToArchive</archiveUploadIdMapId>
</archiveModuleDisplay>

In the ArchiveModuleDisplay the following elements should be added:

 	<hideArchiveUploadTab>false</hideArchiveUploadTab>
	<archiveUploadIdMapId>IdMapToArchive</archiveUploadIdMapId>

WorkflowFiles

A configuration example for the WorkflowFile is shown below:

<?xml version="1.0" encoding="UTF-8"?>
<workflow xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews https://fewsdocs.deltares.nl/schemas/version1.0/workflow.xsd" version="1.1">
	<activity>
		<runIndependent>true</runIndependent>
		<moduleInstanceId>MergeEdits</moduleInstanceId>
	</activity>
</workflow>

Don't forget to list the workflow in the WorkflowDescriptors:

<workflowDescriptor id="Merge_Edits" name="Merge uploaded edits of archived data" visible="true"/>

The workflow should be scheduled on regular basis so that edited data can be processed. The uploaded pi-xml files are processed in 'First in First out' order. Note that the current procedure of merging data edits into the archive is based on reading the time series of a netcd-files into a FEWS workflow and replacing the edited values and exporting a new netcdf-file to the archive. The downside of this procedure is that custom attributes which added in the original netcdf file will not be exported. This is an enhancement which is scheduled for a later version.

ModuleConfigFiles

An configuration example for the ModuleConfigFile to merge the edited data can be found below:

<?xml version="1.0" encoding="UTF-8"?>
<exportArchiveModule xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews https://fewsdocs.deltares.nl/schemas/version1.0//exportArchiveModule.xsd">
	<mergeEditedObservedData>
		<archiveFolder>$ArchiveFolder$</archiveFolder>
	</mergeEditedObservedData>
</exportArchiveModule>

The archiveFolder should contain the location of the archiveFolder at the server which runs the FSS, we strongly recommend using global.properties to set the archiveFolder. 

Don't forget to list the moduleInstance in the ModuleInstanceDescriptors (we recommend using a CSV for this).

  • No labels

Disclaimer