Test suites are carried out every weekend automatically on a subset of machines. The goal is to run comprehensive test suites on multiple machines with multiple compilers and to compare to results from the prior week. The test suites leverage several features associated with the icepack.setup and cice.setup scripts. These are documented in the user guide. In particular, the weekly script tends to look something like

set testdir = "/glade/work/tcraig/icepack_testing"
set basedir = "/glade/derecho/scratch/tcraig/ICEPACK_BASELINES_MAIN"
set date0 = `date -u "+%y%m%d"`
set date = ${date0}a

mkdir -p ${testdir}
mkdir -p ${basedir}

cd ${testdir}
git clone https://github.com/cice-consortium/ICEPACK icepack.main.${date} --recursive
cd icepack.main.${date}

./icepack.setup --suite base_suite,io_suite,travis_suite,quick_suite -m derecho -e intel,cray,gnu,nvhpc,intelclassic,inteloneapi --testid T${date} --bgen default --bcmp default --bdir ${basedir} --report

In general, the latest code from github main is cloned and a test suite is run on multiple compilers both saving results for the next week and comparing results from the prior week. "testdir" is the location of the testing sandbox. "basedir" is the location of the results to support regression testing. The variable "date" defines both the test sandbox name and the testid. Use of "--bgen default" and "--bcmp default" allows the regression testing to automatically save the current results as well as compare to the last set of test results. Setting a unique "basedir" just for automated testing is important to segregate those results from other development testing that may occur on the same machine. "--report" automatically reports results to the wiki page, https://github.com/CICE-Consortium/Test-Results/wiki. Results from the main branch are always reported on the test results wiki under "Main by hash" and "Main by machine".

The full test suite is not expected to pass all tests on all machines all the time. The coverage is comprehensive enough that things like 128 bit MPI and hdf5 IO are not always working properly on all machines, so some tests fail. There can also be cases where on any given weekend, a test will run into a hardware issue or will time out because it hangs or runs slowly. As a result, the assessment of weekly test results is a process that involves judgement, especially when a non-bit-for-bit change is introduced that might affect only a subset of test cases.

Two strategies have been used to submit the automated test suites. The first leverages cron where a cron job is setup to run the test suite every weekend at a certain time. However, cron is not always available to users. The second strategy involves submitting a run to the queue but delaying the run time until the next weekend. Use of delayed job submission is done on Derecho, and the full test script looks like

#!/bin/csh 
#PBS -N cice_test
#PBS -l job_priority=regular
#PBS -A P93300065
#PBS -q main
#PBS -l select=1:ncpus=1:mpiprocs=1:ompthreads=1
#PBS -l walltime=12:00:00
#PBS -j oe
#PBS -M [email protected]
#PBS -m be

set scrdir = "~"
set testdir = "/glade/work/tcraig/cice_testing"
set basedir = "/glade/derecho/scratch/tcraig/CICE_BASELINES_MASTER"
set date0 = `date -u "+%y%m%d"`
set date = ${date0}a

set nextsub = `date --date="next saturday" +%y%m%d0300`
cd $scrdir
echo "nextsub = $nextsub"
qsub -a $nextsub cice_test.csh

mkdir -p ${testdir}
mkdir -p ${basedir}

cd ${testdir}
git clone https://github.com/cice-consortium/CICE cice.master.${date} --recursive
cd cice.master.${date}

./cice.setup --suite first_suite,prod_suite,base_suite,travis_suite,decomp_suite,reprosum_suite,io_suite,omp_suite,gridsys_suite,perf_suite,quick_suite,unittest_suite -m derecho -e intel,cray,gnu,nvhpc,intelclassic,inteloneapi --testid T${date} --bgen default --bcmp default --bdir ${basedir} --report

Some sample test scripts can be found on Derecho at

/glade/u/home/tcraig/icepack_test.csh
/glade/u/home/tcraig/icepack_test_coverage.csh  
/glade/u/home/tcraig/cice_coverage.csh
/glade/u/home/tcraig/cice_test_coverage.csh