Contents
Overview
Imports time series data in a tabular ASCII format from the Australian Bureau of Meteorology (HCS). The files consist of a set of header lines and then lines with a fixed number of fields. The fields are separated by a comma and the order is fixed. Multiple locations and parameters can be put in a single file.
Functionality and limitations
- The import can read scalar data in any timestep from the HCS files.
- The header information in the HCS file is mostly ignored but the timezone information (line 7) is used.
- The unit is set during import and can be used to convert data on import using the Unit Conversion functionality.
- Comments from the HCS file are also imported.
- The HCS data quality flags are set on importing (a flagconversion has to be set up to actually use them, see below)
Configuring the Import
The reader is named HCS which should be configured in the general section of the import. An example configuration is shown below:
<?xml version="1.0" encoding="UTF-8"?> <!-- edited with XMLSpy v2007 sp2 (http://www.altova.com) by WL | Delft Hydraulics (WL | Delft Hydraulics) --> <timeSeriesImportRun xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/timeSeriesImportRun.xsd"> <import> <general> <importType>HCS</importType> <folder>$IMPORT_FOLDER$/hcs</folder> <backupFolder>$BACKUP_FOLDER$/hcs</backupFolder> <idMapId>IdImportHcs</idMapId> <flagConversionsId>HcsFlagConversions</flagConversionsId> <importTimeZone> <timeZoneOffset>+10:00</timeZoneOffset> </importTimeZone> </general> <timeSeriesSet> <moduleInstanceId>ImportHcs</moduleInstanceId> <valueType>scalar</valueType> <parameterId>P.obs</parameterId> <locationSetId>Gauges_P.obs</locationSetId> <timeSeriesType>external historical</timeSeriesType> <timeStep unit="hour"/> <readWriteMode>add originals</readWriteMode> </timeSeriesSet> <timeSeriesSet> <moduleInstanceId>ImportHcs</moduleInstanceId> <valueType>scalar</valueType> <parameterId>H.obs</parameterId> <locationSetId>Gauges_H.obs</locationSetId> <timeSeriesType>external historical</timeSeriesType> <timeStep unit="nonequidistant"/> <readWriteMode>add originals</readWriteMode> </timeSeriesSet> </import> </timeSeriesImportRun>
An example IdMapping file (that maps the location and parameter Ids) is shown below. since 2024.02 FEWS release the ID mapping can also use an externalqualifier, this is the SensorDataType column of hte HCS file.
<?xml version="1.0" encoding="UTF-8"?> <idMap version="1.1" xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/idMap.xsd"> <parameter internal="H.obs" external="WL"/> <location internal="44198-01-01" external="44198-01-01"/> </idMap>
Flag Conversion
Flag conversions can be used to convert data quality flag from external sources to Delft-FEWS flags. The BOM HCS format defines the following quality flags:
Quality Code | Description | Example |
|---|---|---|
1 | As observed. | |
2 | As observed with normalised time window. | e.g 9AM or rounded to nearest hour. |
3 | Derived from other observation/s. | |
4 | Interpolated from other observation event/s. | |
5 | As observed and validated | (Quality Controlled). |
6 | Void (Bad) observation. | |
7 | Observation where canister reset or calibration change has occurred. |
To use the flags in Delft-FEWS a flagconversion file has to be set up. A working example is attached to this page. The table below summarizes the translation used:
HCS code | HCS Description | Delft-FEWS code | Delft-FEWS Name | Delft-FEWS Description |
|---|---|---|---|---|
1 | As observed | 0 | ORIGINAL_RELIABLE | Observed value retrieved from external data source. Value is valid, marked as original reliable as validation is yet to be done |
2 | As observed with normalised time window | 0 | ORIGINAL_RELIABLE | Observed value retrieved from external data source. Value is valid, marked as original reliable as validation is yet to be done |
3 | Derived from other observations | 0 | CORRECTED_RELIABLE | The original value was removed and corrected. Correction may be through by interpolation or manual editing |
4 | Interpolated from other observation events | 0 | CORRECTED_RELIABLE | The original value was removed and corrected. Correction may be through by interpolation or manual editing |
5 | As observed and validated (quality controlled) | 0 | CORRECTED_RELIABLE | The original value was removed and corrected. Correction may be through by interpolation or manual editing |
6 | Void (bad) observation | 6 | Missing_Unreliable | Observed value retrieved from external data source. Value is invalid due to validation limits set. Value is removed |
7 | Observation where canistor reset or calibration change has occured | 0 | ORIGINAL_DOUBTFUL | Observed value retrieved from external data source. Value is valid, but marked as suspect due to soft validation limits being exceeded |
Unit conversions
When importing the unit from the HCS file the column Unit and the column SensorTypeParam1 are concatenated. (e.g. mm LHG, meter AHD)
These can be used in the Unitconversion to convert the data on import.
The file format
The file format is described in the BOM document "External Agency Hydrological data Transfer - Client Requirements, Version 2.20".
An example of a file is shown below:
# HEADER: Agency Id: BoM # HEADER: File Generation Date: 2008-08-01T14:30:45z # HEADER: File Format: BOM-HCS # HEADER: File Format Version: 2.0 # HEADER: Generated by (system): TimeStudio # HEADER: Number of Records: 11 # HEADER: Local ObsTime Offset: 0 # HEADER: Data Fields: IndexNo, SensorType, SensorDataType, SiteIdType, SiteId, ObservationTimestamp, RealValue, Unit, SensorTypeParam1, SensorTypeParam2, Quality, Comment 1,"WL",1,"SLSR","44198-01-01","2008-02-01T01:07:06z",1.150000,"metres","LGH",,1,"" 2,"WL",1,"SLSR","44198-01-01","2008-02-01T01:08:06z",1.200000,"metres","LGH",,1,"" 3,"WL",1,"SLSR","44198-01-01","2008-02-01T01:43:06z",1.150000,"metres","LGH",,1,"" 4,"WL",1,"SLSR","44198-01-01","2008-02-01T01:46:06z",1.200000,"metres","LGH",,1,"" 5,"WL",1,"SLSR","44198-01-01","2008-02-01T02:04:06z",1.150000,"metres","LGH",,1,"" 6,"WL",1,"SLSR","44198-01-01","2008-02-01T02:11:06z",1.200000,"metres","LGH",,1,"" 7,"WL",1,"SLSR","44198-01-01","2008-02-01T02:16:06z",1.150000,"metres","LGH",,1,"" 8,"WL",1,"SLSR","44198-01-01","2008-02-01T02:22:06z",1.200000,"metres","LGH",,1,"" 9,"WL",1,"SLSR","44198-01-01","2008-02-01T02:23:06z",1.150000,"metres","LGH",,1,"" 10,"WL",1,"SLSR","44198-01-01","2008-02-01T02:25:06z",1.200000,"metres","LGH",,1,"" 11,"WL",1,"SLSR","44198-01-01","2008-02-01T02:28:06z",1.150000,"metres","LGH",,1,""
Performance
On a 2.7Ghz Dual core laptop the import is able to import 188Mb of hcs files (1899984 lines) in 24 seconds including basic validation of the data.
