Module Datasets
ModuleDataSets are zip files holding custom content (ie files, direcory structures etc). Fews exports and unzips this file to a destination folder as instructed via the autoExportModuleDataSet configuration in the clientConfig.xml file or via the exportDataSetActivity of the General Adapter. Fews conducts a checksum to determine if an already exported moduleDataSet (as registered in the exportedModuleDataSetFiles.txt in REGION_HOME) corresponds to the dataset to be exported. In the General Adapter this check is conducted on request (see updateExportDataSetdirOnlyOnChange).
Within the General Adapter, the zip file to be exported can be identified via a registered moduleInstanceId, an (unregistered) moduleDataSetName or an moduleDataSetNameMap. In all three situations, the name of the zip-file should correspond to the moduleInstanceId or moduleDataSetName.
Within a client-server system, the Config Manager (or the config uploading process of the AdminInterface API) will place the zip file in a record in the ModuleDataSetFiles database table. Note that a zip file should never be larger than 512Mb as the Postgres backup process needs to be able to hold two versions of the record in memory, and its memory is max.1Gb.
Many Fews-configurations are managed in version control systems like svn or GIT. These systems are efficient in managing ascii based files, but less efficient in managing binary files (like zip files). Over the years, it has been noted that many updates of moduleDataSet zip file are conducted as a single ascii file had to be updated. Therefore, Delft-FEWS received a new feature since 2023.02: it now can cope with .zip folders as part of the Config/ModuleDataSetFiles configuration. When uploading the configuration to the central database, both the ConfigManager as well as the AdminInterface API will recognize this .zip folder, pack it to a zip-file and insert this into the database.
Please be aware that the file/foldername of a zip-file/zip-folder should not have spaces and the resulting zip is still not allowed to be larger than 512 Mb.
Note: for Linux, executable files that are part of a module dataset may require extra attention as these need to be marked as executable but the zip file format does not facilitate this. When exporting module datasets FEWS will attempt to mark files without an extension as well as files with the extensions .exe, .sh, .pl, .hydra and .ix as executable. There is no fixed list of executable file extensions for Linux so module datasets containing executable files using different extensions may require extra attention, for example an extra shell script that marks relevant files as executable (using chmod) before running the module.
