Include Page | ||||
---|---|---|---|---|
|
Contents
Table of Contents |
---|
Introduction
The XBeach Skillbed is an automated testing environment for the XBeach model. The skillbed monitors the quality of the model developments. In order to do so, a collection of model setups is collected in the Subversion repository of Deltares together with the latest model source code, possibly arranged in different branches. When the skillbed is run, both the source code revisions as well as the model setups are downloaded from the repository, binaries are built and tests are run.
Tests can be run in different configurations, with different parameter files and different versions of the XBeach executable. After all tests are finished, a collection of Matlab scripts performs the post-processing of the model results. The use of Matlab scripts makes management of the post-processing easy accessible for everyone. The scripts generate figures, tables and statistical performance scores. The result of the post-processing is then captured in several PDF reports. The reports are automatically e-mailed to the developers of XBeach.
The XBeach Skillbed runs on a daily basis and contains over 50 model setups and 4 different executables. The skillbed is written in Python and can be run over different computers using XML-RPC. The skillbed is therfore splitted into a server and client part, which are both automatically updated from the subversion repository as well.
Installation
Requirements
The XBeach Skillbed requires the following software to be installed:
Info | ||
---|---|---|
| ||
Install Python(x,y) to install a full set of python tools including easy_install. Easy_install can be used to install the Mako templates. Do not use Python 3.x or up! |
The XBeach Skillbed requires the following environment variables to be set:
These environment variables can also be set through the skillbed.inst file in the tools directory.
Download
The skillbed can be installed by running the following python script: https://repos.deltares.nl/repos/openearthtools/trunk/python/applications/skillbed/tools/python/install.py. The script downloads the skillbed from the OpenEarthTools Subversion repository.
Run
The server can be started by running the tools/python/skillbed/server.py script.
The client can be started by running the tools/python/skillbed/client.py script.
The tools/skillbed.cfg file on the client machine should be adapted to hold the hostnames and ports of the servers to be used. Adding -h to the script call will display help information.
Overview
Adding a test
- Create a checkout of the skillbed repository: https://repos.deltares.nl/repos/XBeach/skillbed/
- Create a test directory with the name of the test in the input directory
- Create a working model in the test directory
It is possible to group related tests in a single directory. Simply use subdirectories for this purpose. In de test configuration file, the subdirectories should be mentioned (see further down).
- Create a Matlab script that analyses the model output in the test directory
The Matlab script takes two arguments: info and dirs. Both arguments are structs that have the following contents:
Code Block |
---|
info = struct( ...
'revision', 1234, ... // revision number of current binary
'binary', 'trunk', ... // name of current binary
'type', 'custom', // mode of current run (custom or default)
'test', 'CarrierGreenspan', // name of current test
'run', 'default' ... // name of current run
);
dirs = struct( ...
'output', 'C:\SKILLBED\analysis\trunk\custom\CarrierGreenspan\default', ... // Path where output should be stored
'data', '', ... // Path where static data can be found
'network', 'P:\STORAGE\DATA\CarrierGreenspan' ... |
Wiki Markup |
{include:ContentHeader} {tip}Within the Deltares network, a public testbed client is available through the following URL: http://d00753:5000/interface/index{tip} h1. Contents {toc} h1. Introduction The XBeach Testbed is an automated testing environment for the [XBeach model|XBEACH:Home]. The testbed monitors the quality of the model developments. In order to do so, a collection of model setups is collected in the [Subversion repository|https://repos.deltares.nl/repos/XBeach/testbed2011/] of Deltares together with the latest model source code, possibly arranged in different branches. When the testbed is run, both the source code revisions as well as the model setups are downloaded from the repository, binaries are built and tests are run. Tests can be run in different configurations, with different paramerter files and different versions of the XBeach executable. After all tests are finished, a collection of Matlab scripts performs the post-processing of the model results. The use of Matlab scripts makes management of the post-processing easy accessible for everyone. The scripts generate figures, tables and statistical performance scores. The result of the post-processing is then captured in several PDF reports. The reports are automatically e-mailed to the developers of XBeach. The XBeach Testbed runs on a daily basis and contains over 50 model setups and 4 different executables. The testbed is written in Python and can be run over different computers using XML-RPC. The testbed is therfore splitted into a server and client part, which are both automatically updated from the subversion repository as well. h1. Installation h2. Requirements The XBeach Testbed requires the following software to be installed: {table-plus:width=800|enhableHighlighting=true} || Name || URL || Remarks || | Python 2.6 | [http://www.python.org/] | | | PySVN 1.7 | [http://pysvn.tigris.org/] | | | Pylons 1.0 | [http://pylonshq.com/] | Client only | | Mako 0.4.0 | [http://www.makotemplates.org/] | Client only | | Miktex 2.8 | [http://miktex.org/] | Windows client only | | pdflatex | | Linux client only | | Matlab | [http://www.mathworks.nl/products/matlab/] | Client only | | MPICH2 | [http://www.mcs.anl.gov/research/projects/mpich2/] | Server only | | Microsoft Visual Studio 2008 | [http://www.microsoft.com/visualstudio/en-us/products/professional/default.mspx] | Windows server only | | Intel Fortran Compiler | [http://software.intel.com/en-us/intel-compilers/] | Windows server only | | GNU Fortran Compiler | [http://gcc.gnu.org/fortran/] | Linux server only | {table-plus} The XBeach Testbed requires the following environment variables to be set: {table-plus:width=800|enhableHighlighting=true} || Name || Value || Remarks || | PYTHON_PATH | Path to python executable | | | PDFLATEX_PATH | Path to pdflatex executable | Client only | | MATLAB_PATH | Path to Matlab executable | Client only | | MPIEXEC_PATH | Path to mpiexec executable | Server only | | VS90COMNTOOLS | Microsoft Visual Studio 2008 installation directory | Windows server only | | IFORT_COMPILER11 | Intel Fortran Compiler installation directory | Windows server only | | INTEL_LICENSE_FILE | Path to Intel Fortran Compiler license file | Windows server only | | GFORTRAN_PATH | Path to gfortran executable | Linux server only | {table-plus} h2. Download The XBeach Testbed can be installed by running the following python script: [https://repos.deltares.nl/repos/XBeach/testbed2011/tools/python/install.py]. The script downloads the testbed from the subversion repository. h2. Run The server can be started by running the tools/python/testbed/server.py script. The client can be started by running the tools/python/testbed/client.py script. The tools/testbed.cfg file on the client machine should be adapted to hold the hostnames and ports of the servers to be used. Adding -h to the script call will display help information. A public client is available via the webpage [http://d00753:5000/interface/index]. Keep in mind that this client is also a server, preforming too many requests will slow down it's performance. Consider installing a dedicated client by that time. h1. Overview !testbed.png|width=800! h1. Adding a test * Create a checkout of the testbed repository: [https://repos.deltares.nl/repos/XBeach/testbed2011/] * Create a test directory with the name of the test in the _input_ directory * Create a working model in the test directory It is possible to group related tests in a single directory. Simply use subdirectories for this purpose. In de test configuration file, the subdirectories should be mentioned (see further down). * Create a Matlab script that analyses the model output in the test directory The Matlab script takes two arguments: _info_ and _dirs_. Both arguments are structs that have the following contents: {code} info = struct( ... 'revision', 1234, ... // Network storage that can be used ); |
- Create a configuration file called .config in the test directory
The configuration file has the following or similar contents:
Code Block |
---|
[general] // revision number of current binary 'binary', 'trunk', ... // nameSECTION: ofGeneral current binaryinformation enable 'type', 'custom', = 1 // modeEnable test ofin currentskillbed runs run (custom or default) 'test', = 'CarrierGreenspan'default, stationary // nameDefinition of currenttest testruns responsible 'run', = Bas 'default' ...Hoonhout <bas.hoonhout@deltares.nl> // Name and e-mail of maintainer [run_default] // name of current run ); dirs = struct( ... 'output', 'C:\TESTBED\analysis\trunk\custom\CarrierGreenspan\default', ... // Path where output should be stored 'data', '', ... // SECTION: Configuration of test run "default" enable = 1 // Path where static data can be found 'network', 'P:\STORAGE\DATA\CarrierGreenspan' ... // Enable run in skillbed params // Network storage that can be used ); {code} * Create a configuration file called _.config_ in the test directory The configuration file has the following or similar contents: {code} [general] = params_original.txt // Parameter file to use in run binaries = all // SECTION: General information enable = 1 // Binaries that should use this run types = all // Enable test in testbed runs = default, stationary // Modes in which this run should be used (custom/default/all) platforms = all // Definition of test runs responsible = Bas Hoonhout <bas.hoonhout@deltares.nl> // Name and e-mail of maintainer [run_default] // Platforms on which this run should be used (win32/unix/all) nodes = 1 // SECTION: Configuration of test run "default" enable = 1 // Number of nodes that should be used during this run (> 1 for MPI) analysis = analyze_this // Enable run in testbed params = params_original.txt // Matlab function to be used for the postprocessing of the results [run_stationary] // Parameter file to use in run binaries = all // SECTION: Configuration of test run "stationary", see above enable = 1 params = params_stationary.txt binaries // Binaries= that should use this runtrunk, speedup types = allcustom platforms = win32 nodes = 1 analysis = analyze_this [categories] // Modes in which this run should be used (custom/default/all) platforms = all // SECTION: Category definitions used for classification of test configuration = 1D // Platforms on which this run should be used (win32/unix/all) nodes = 1 // Grid configuration (1D/2D) waves = stationary // Number of nodes that should be used during this run (>// 1Wave for MPIconfiguration (stationary/non-stationary) analysiswaterlevels = constant = analyze_this // MatlabWater functionlevel to be used for the postprocessing of the results [run_stationary]configuration (constant/varying) morphology = no // Morphology configuration // SECTION: Configuration of test run "stationary", see above enable(yes/no) fractions = 1 = 1 params = params_stationary.txt binaries = trunk, speedup types = custom platforms = win32 nodes // Number of =sediment 1fractions analysis hardlayers = analyze_this [categories]no // SECTION:Hard Categorylayer definitions used for classification of test configuration configuration (yes/no) groundwaterflow = 1Dno // GridGround water flow configuration (1Dyes/2D) waves = stationary // Wave configuration (stationary/non-stationary) waterlevels = constant // Water level configuration (constant/varying) morphology = no // Morphology configuration (yes/no) fractions = 1 // Number of sediment fractions hardlayers = no // Hard layer configuration (yes/no) groundwaterflow = no // Ground water flow configuration (yes/no) {code} If you use subdirectories to group related tests, the configuration file(s) can be stored in the root directory or in any subdirectory. The location of the configuration file determines where the test definition end and the run definition starts. In case the configuration file is located in one or more subdirectories, the names of the subdirectories are concatenated to the test name, separated by an underscore (\_). Any other subdirectories should be mentioned in the run definition of the configuration file itself. In the path of directories to a configuration file, no other configuration files may be found, otherwise this configuration file is used instead. A simple script to generate a default configuration file can be found in the root of the _input_ directory. It searches for subdirectories to be interpreted as runs or otherwise generates a single run for the root directory. Please at least alter responsibilities and test characteristics manually. * Create a TEX file in the _latex/___tests_ directory with a descriptive text for the added test and references to the postprocessing output Images are referenced to as shown below. The first argument is the path containing the test and run name of the output. The second argument is the file name to be referenced. The third argument is an optional caption of the image. {code} ${tools.image('CarrierGreenspan/default/', 'fig1.eps', 'Model output')} {code} Tables are referenced in a similar manner: {code} ${tools.table('Assateague_Island/profA/', 'conditions.tex', 'Hydrodynamic boundary conditions XBeach simulations')} {code} * Create one or more references to the test description TEX file in one or more documents References to descriptive texts are made as follows: {code} <%include file="carriergreenspan.tex"/> {code} * Commit the additions to the Subversion repository h3. Troubleshoot Sometimes the resulting report is not what you expected. The following information might be relevant to get the report just as you want. * References turn into question marks (??) The reference is not valid. Please make sure you use the full reference identifier: <test name>\_<run name>\_<filename> _without_ extension for generated files. For static files that are not generated by the analysis scripts, you simply use the filename _without_ extension. * My formulas are not aligned as they should Often formulas are aligned using table-like alignment commands. These end in line endings (\ \). This line ending is also the command for Python to concatenate lines to a single line. In order to use the former functionality only, simply add a space after the line ending command. h2. Coding guidelines The [Matlab XBeach Toolbox|Toolbox] is available within the testbed environment. The following commands from this toolbox are preferred to use, including the names of the variables. Please see the corresponding documentation for the exact syntax: * Read model dimensions {code}no) |
If you use subdirectories to group related tests, the configuration file(s) can be stored in the root directory or in any subdirectory. The location of the configuration file determines where the test definition end and the run definition starts. In case the configuration file is located in one or more subdirectories, the names of the subdirectories are concatenated to the test name, separated by an underscore (_). Any other subdirectories should be mentioned in the run definition of the configuration file itself. In the path of directories to a configuration file, no other configuration files may be found, otherwise this configuration file is used instead.
A simple script to generate a default configuration file can be found in the root of the input directory. It searches for subdirectories to be interpreted as runs or otherwise generates a single run for the root directory. Please at least alter responsibilities and test characteristics manually.
- Create a TEX file in the latex/_tests directory with a descriptive text for the added test and references to the postprocessing output
Images are referenced to as shown below. The first argument is the path containing the test and run name of the output. The second argument is the file name to be referenced. The third argument is an optional caption of the image.
Code Block |
---|
${tools.image('CarrierGreenspan/default/', 'fig1.eps', 'Model output')}
|
Tables are referenced in a similar manner:
Code Block |
---|
${tools.table('Assateague_Island/profA/', 'conditions.tex', 'Hydrodynamic boundary conditions XBeach simulations')}
|
- Create one or more references to the test description TEX file in one or more documents
References to descriptive texts are made as follows:
Code Block |
---|
<%include file="carriergreenspan.tex"/>
|
- Commit the additions to the Subversion repository
Troubleshoot
Sometimes the resulting report is not what you expected. The following information might be relevant to get the report just as you want.
- References turn into question marks (??)
The reference is not valid. Please make sure you use the full reference identifier: <test name><run name><filename> without extension for generated files. For static files that are not generated by the analysis scripts, you simply use the filename without extension.
- My formulas are not aligned as they should
Often formulas are aligned using table-like alignment commands. These end in line endings (two back-slashes). This line ending is also the command for Python to concatenate lines to a single line. In order to use the former functionality only, simply add a space after the line ending command.
- I have to wait a day before I see the results of the changes I made in the report
Using the skillbed client, you can test the report generation. Since no real data is available, the test report will be crowded with giraffes. Nevertheless, it enables you to check whether you texts and formulas come out well and your references are properly configured. From the tools/python/skillbed directory, you need to run the following command to test the generation of the trunk_custom report:
Code Block |
---|
>> python client.py --test-report trunk_custom --skip-clean
|
You will find the result in the run directory.
Coding guidelines
The Matlab XBeach Toolbox is available within the skillbed environment. The following commands from this toolbox are preferred to use, including the names of the variables. Please see the corresponding documentation for the exact syntax:
- Read model dimensions
Code Block |
---|
d = xb_read_dims;{code}
* Read model |
- Read model input
Code Block |
---|
input
{code}xbi = xb_read_input; |
- Read model output
Code Block |
---|
{code} * Read model output {code} xbo = xb_read_output(pwd, 'vars', { ... }, ...); xbo = xb_peel(xb_read_output(pwd, 'vars', { ... }, ...)); {code} * Generate grid information |
- Generate grid information
Code Block |
---|
{code}s = xb_stagger(d.x,d.y);{code} * Miscellaneous functions {code} |
- Miscellaneous functions
Code Block |
---|
path = fullfile( ... ); k = disper(w,h); S = xb_spectrum(zs); [r2 sci relbias bss] = xb_skill( ... ); |
- Create figures and tables
Code Block |
---|
{code} * Create figures and tables {code} xb_plot_profiles(xbo, ...); xb_plot_profiles(xbo, 'measured', [x y], 'durosta', [x y], ...); xb_plot_wavetrans(xbo, ...); xb_plot_wavetrans(xbo, 'Hrms_lf', zs, 'Hrms_hf', H); xb_write_skilltable(xbo, [x y]); {code} |