Running regression tests using rt.sh

Shell script-based Regression Test: rt.sh

$ git clone https://github.com/NOAA-EMC/DATM-MOM6-CICE5 DATM-MOM6-CICE5
$ cd DATM-MOM6-CICE5
$ git submodule update --init --recursive
$ cd tests
$ vi rt.sh (The default account to use is nems. Change is needed on line 196 if you cannot use nems account, e.g., ACCNR=marine-cpu
$ rt.sh -f -k (This command will compile and run all 4 test cases and keep the run directory after the run is done.)

detect_machine.sh: detect and assign machine name, set account (nems is default)
compile.sh: build a model using GNUmakefile (make app=*)
Run regression test
- run_test.sh: sets environment variables, run directory, etc., and calls rt_datm.sh
- rt_datm.sh: prepares a canned case in the run directory, and calls rt_utils functions
- rt_utils.sh: contains utility functions, e.g.,
  - submit_and_wait
  - check_results
  - rocoto_create_compile_task, rocoto_create_run_task, rocoto_run, rocoto_kill
  - ecflow_create_compile_task, ecflow_create_run_task, ecflow_run, ecflow_kill

rt.conf: specify compile and run cases corresponding to ../compsets/all.input
COMPILE: specify <appBuilder-name> located in ../
RUN: specify <test-name> located in tests/
Each row is processed sequentially
Workflow managers: RUN depends on preceding COMPILE. Currently, only one COMPILE at a time (cf. ufs-weather-model)

Two levels to set simulation parameters
- default_vars.sh sets default values, similar to datm_defaults in ../compsets/mom6cice5datm.input
- <test-name> overrides default values, adds test-specific parameters, e.g.,
  - IATM=1760, JATM=880, FHMAX=48, WLCLK=45
Set environment variables that are passed onto various template files in ../parm/
- input.mom6.nml.IN
- nems.configure.*.IN
- model_configure.IN
Specify configuration templates to use, e.g.,
- NEMS_CONFIGURE=”nems.configure.medcold_atm_ocn_ice.IN”
- NEMS_CONFIGURE=”nems.configure.med_atm_ocn_ice.IN”

Set up input data, grid data, etc. by copying files from baseline directory to run directory
Baseline directory contains
- Sub-directories for input data (e.g., CICE_IC, MOM6_IC, DATM, MEDIATOR_CFSR)
- Sub-directories for previous run results (e.g., RT-Baselines_2d_CFSR)
Make sure directories and files exist in RTPWD

Baseline directory (RTPWD)
- Hera: /scratch1/NCEPDEV/nems/emc.nemspara/RT/DATM-MOM6-CICE5/develop-YYYYMMDD
Run directory root (RUNDIR_ROOT)
- RUNDIR_ROOT=/scratch1/NCEPDEV/stmp2/${USER}/S2S_RT/rt_$$
- RUNDIR=${RUNDIR_ROOT}/${TEST_NAME}
New baseline directory (NEW_BASELINE)
- Hera: /scratch1/NCEPDEV/stmp4/${USER}/S2S_RT/REGRESSION_TEST_INTEL

Triggered by COMPILE row in rt.conf with specified <appBuilder-name>
As in NEMSCompsetRun, build is done using GNUmakefile in ../NEMS/
compile.sh is a simple wrapper around GNUmakefile to interface with rt.sh
- $ ./compile.sh coupled_DATM_MOM6_CICE
- make app=coupled_DATM_MOM6_CICE build
If you prefer to build exe file separately (i.e., w/o rt.sh), place a copy in DATM-MOM6-CICE5/tests
- $ cd DATM-MOM6-CICE5/tests
- $ cp ../NEMS/exe/NEMS.x datm_mom6_cice_0.exe
- $ cp ../NEMS/src/conf/modules.nems modules.datm_mom6_cice_0
If you want to reuse your exe, keep a copy with a different name

If you make code changes that are not expected to change simulation results, you can run full regression tests afterward to demonstrate your changes do not break anything
Currently, there are 4 standard regression tests on Hera
In DATM-MOM6-CICE5/tests/ directory, use any one of the following:
- $ ./rt.sh -f >output 2>&1 &
- $ ./rt.sh -f -e (use ecFlow)
- $ ./rt.sh -fr (use Rocoto)
- $ ./rt.sh -fek (use ecFlow, keep run directory for post-run diagnosis)

Create a file, say my_test.conf, with a single COMPILE and a single RUN
- $ cp rt.conf my_test.conf
- $ vi my_test.conf
- $ ./rt.sh -l my_test.conf
Or make a copy of original rt.conf file
- $ cp rt.conf rt.conf.orig
- $ vi rt.conf
- $ ./rt.sh -f

Remove COMPILE row in rt.conf
$ cp ../NEMS/exe/NEMS.x datm_mom6_cice_0.exe
$ cp ../NEMS/src/conf/modules.nems modules.datm_mom6_cice_0
- This module file needs to be identical to the one you used for build
$ ./rt.sh -f
This approach does not work with workflow managers because RUN depends on COMPILE

Summary files
- Hera: RegressionTests_hera.intel.log, Compile_hera.intel.log
./rt.sh >output 2>&1 &: output of rt.sh
Log files in log_hera.intel/
- compile_*.log: output of compile.sh and GNUmakefile
- run_*.log: output of run_test.sh
Run directory RUNDIR_ROOT/
- .log: output of rt_datm.sh. If rocoto used, also contains err & out from sbatch job
- subdir: contains all files necessary for simulation, e.g., sbatch job_card
- run_test*.env: exported environment variables, e.g., MACHINE_ID, RTPWD, PATHRT, ACCNR, QUEUE.
- QUEUE is set to batch in rt.sh