Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

No Format
<serverUrl>http://nomads.ncep.noaa.gov:9090/dods/gfs/gfs%RELATIVE_TIME_IN_SECONDS(yyyyMMdd, -18000 )%/gfs_%RELATIVE_TIME_IN_SECONDS(HH,-18000)%z</serverUrl>
ftpPassiveMode

When using "ftp://" import the option of passive mode is supported.  This is an optinal field, default value is false, and can be configured in the general section.

Code Block
languagexml
linenumberstrue
<general>
		<importType>importType</importType>
		<folder>ftp://import/external</folder>
		<ftpPassiveMode>true</ftpPassiveMode>		
</general>
failedFolder

Folder to move badly formatted files to. This may be a UNC path (ie located on the network).

importTriggeringFile

Use this option to configure a path to one or more files that trigger the import.  The import will only start when all  triggering files are available. The triggering files will be deleted at the end of the import run.

Some examples:

<importTriggeringFile>p:\myProject\trigger</importTriggeringFile>

or

<importTriggeringFile>$REGION_HOME$/import/external/trigger1</importTriggeringFile>
<importTriggeringFile>$REGION_HOME$/import/external/trigger2</importTriggeringFile>
fileNameDateTimeFilter

One can filter the reading of files and subdirectories by explicity defining which files and (sub)folders are to be read. This can be done using date-strings in the names to be able to read data from a folder setup like "data\2022\01\10\*.*" which in this example lists the year, month and day. This is very useful to read data from large datasets where only the most recent data is to be read. Note that zip files are always internally handled as if it were a folder. The relativeViewperiod defines in which folders FEWS will actively and only look for available data files.
Note that the subFolderLevel of 0 can be used for defining a file filter. The counting of real folders starts from 1 onwards.

The below example reads only the latest files from a server where many files are stored. In this example these are zipfiles provided at a 6-hour timestep. 

Code Block
languagexml
linenumberstrue
<fileNameDateTimeFilter subFolderLevel="1">
	<timeStep unit="hour" multiplier="6"/>
	<dateTimePattern>'hr44_'yyyyMMddHH'.zip'</dateTimePattern>
	<preFixLength>5</preFixLength>
	<postFixLength>4</postFixLength>
</fileNameDateTimeFilter>
<relativeViewPeriod unit="hour" start="-5" end="0"/>


Another example is where data is to be read from a folder structure like "data\<year>\<month>\<day>\*.*"
In this example FEWS will read the data for the given relativeViewPeriod of the last 10 days only.

Code Block
languagexml
linenumberstrue
<folder>data</folder>
<fileNameDateTimeFilter subFolderLevel="1">
	<timeStep unit="year" multiplier="1"/>
	<dateTimePattern>yyyy</dateTimePattern>
	<preFixLength>0</preFixLength>
	<postFixLength>0</postFixLength>
</fileNameDateTimeFilter>
<fileNameDateTimeFilter subFolderLevel="2">
	<timeStep unit="month" multiplier="1"/>
	<dateTimePattern>MM</dateTimePattern>
	<preFixLength>0</preFixLength>
	<postFixLength>0</postFixLength>
</fileNameDateTimeFilter>
<fileNameDateTimeFilter subFolderLevel="3">
	<timeStep unit="day" multiplier="1"/>
	<dateTimePattern>dd</dateTimePattern>
	<preFixLength>0</preFixLength>
	<postFixLength>0</postFixLength>
</fileNameDateTimeFilter>
<relativeViewPeriod unit="day" start="-10" end="0"/>
ftpPassiveMode

When using "ftp://" import the option of passive mode is supported.  This is an optinal field, default value is false, and can be configured in the general section.

Code Block
languagexml
linenumberstrue
<general>
		<importType>importType</importType>
		<folder>ftp://import/external</folder>
		<ftpPassiveMode>true</ftpPassiveMode>		
</general>
failedFolder

Folder to move badly formatted files to. This may be a UNC path (ie located on the network).

importTriggeringFile

Use this option to configure a path to one or more files that trigger the import.  The import will only start when all  triggering files are available. The triggering files will be deleted at the end of the import run.

Some examples:

<importTriggeringFile>p:\myProject\trigger</importTriggeringFile>

or

<importTriggeringFile>$REGION_HOME$/import/external/trigger1</importTriggeringFile>
<importTriggeringFile>$REGION_HOME$/import/external/trigger2</importTriggeringFile>

Please note that the  triggering files cannot be located on the FTP.

groupImportPattern

Since 2024.01 - Option to specify groups of files that need to be imported together. This can be useful for forecasts spread over several files which should be imported as one forecast i.e. a group of files, in particular if each file is added to the import folder at a different time.

The filenames must contain the forecast date and time. The pattern to define a group is specified in fileNameDateTimePattern, with the expected number of files for one forecast in numberOfFiles. The files will be imported only when all expected files are in the import folder. A time span also has to be specified in waitingTime to define when to consider the forecast as failed and move it to the failed folder. If more than the waiting time has passed between the forecast time and T0, the forecast is considered failed and the files are moved to the failed folder.

This option can be used in combination with the importTriggeringFile option.

For example, for the following configuration:

Code Block
languagexml
<groupImportPattern>
	<numberOfFiles>6</numberOfFiles>
    <fileNameDateTimePattern>'timeseries_'MMddHHmm'_?'</fileNameDateTimePattern>
    <waitingTime unit="day" multiplier="1"/>
</groupImportPattern>

with the import folder containing the files: 

  • timeseries_01011200_1
  • timeseries_01011200_2
  • timeseries_01011200_3
  • timeseries_01011200_4
  • timeseries_01011200_5
  • timeseries_01011200_6
  • timeseries_01021200_1

The first six files will be imported because six files are matching the pattern "timeseries_01011200_?". But the last file will not be imported yet because it is the only file matching the pattern "timeseries_01021200_?" Please note that the  triggering files cannot be located on the FTP .

fileNamePatternFilter

Option to filter files that need to be imported based on their filename. This can be useful if there are files in the import folder that should not be imported:

...

eg: <fileNamePatternFilter>%TIME_ZERO(yyyy)%??????.nc</fileNamePatternFilter>???.nc</fileNamePatternFilter>

Note that the filter does not apply to any zip file directly, but only to the files that are within the zip file.

fileNameObservationDateTimePattern

...

Currently supported only by import from OpenDAP and from local folders.
fileNameForecastCreationDateTimePattern

...

User name, required when importing from protected database connections or protected servers. The username/password is not used for accessing ftp folders. There the credentials have to be defined in the ftp address like ftp://<user>:<password>@ftp.addres.com

password, encryptedPassword

Password, required when importing from protected database connections or protected servers. Please use the Hex value for special characters (e.g. a @ must be specified as %40)the Hex value for special characters (e.g. a @ must be specified as %40)
It is strongly recommend to using encryptedPassword, although the encryption within FEWS is relatively limited. Encrypting your password can be done by the "F12 →  S - clipboard → encrypt password" debug menu option.

relativeViewPeriod

The relative period for which data should be imported. This period is relative to the time 0 of the run. When the start and end time are overrulable the user can specify the download length with the cold state time and forecast length in the manual forecast dialog. It is also possible to import data for an absolute period of time using the startDateTime and endDateTime elements.

...

ID of the IdMap used to convert external parameterId's and locationId's to internal parameter and location Id's. Each of the formats specified will have a unique method of identifying the id in the external format. See section on configuration for Mapping Id's units and flags.method of identifying the id in the external format. See section on configuration for Mapping Id's units and flags.

moduleInstanceAware

Since 2023.01. If value is set to true, data will only be imported if the module instance id specified in the config file matches the module instance id of the downloaded data.

useStandardName (since stable build 2012.02)

...

For some data formats an external unit is not defined in the file to be imported. This elements The element <externUnit> allows the unit to be specified explicitly. This unit , which is then used in possible unit conversions.to find the corresponding unit conversions as configured in a UnitConversionsFiles (see 02 Unit Conversions).

Code Block
languagexml
<externUnit parameterId="P.nwp.fcst" unit="mm" cumulativeSum="false"/>

Attributes of <externUnit>:Attributes;

  • parameterId: Id of the parameter for which a unit is specified. This is the internal parameter Id.
  • unit: specification of unit. This unit must be available in the UnitConversions specified in the unitConversionsId element.
  • cumulativeSum:  if this option is set to "true", then it is possible to receive the parameters as cumulative sums. Cumulative sums are partial sums of a given sequence of numbers. For example, if the sequence is:  {a, b, c, d, ...}, then the cumulative sums are: a, a+b, a+b+c, a+b+c+d, .... . The import module will then calculate the values for the individual time steps {a, b, c, d, ...} by subtracting the cumulative sum of the previous cumulative sum value.
  • cumulativeMean: similar to cumulativeSum, if cumulativeMean is set to "true", then it is possible to receive the parameters as cumulative means. Cumulative means are partial means of a given sequence of numbers. Therefore, for a given sequence of numbers s{x1,x2,...,xn,xn+1} cumulative means are calculated as follows: CM (xn+1) = [xn+1 + n * CM (xn)] ⁄ n+1. The import module will then calculate the values for the individual time steps {a, b, c, d, ...} by multiplying the cumulative mean with the amount of time steps already processed and then subtracting of the previous value that has been also multiplied with the amount of timestep processed minus 1.

...