Preparing and running CROCO simulations on Irene-Rome (TGCC)
The official documentation on downloading and installing the CROCO code can be found on the CROCO website
If you are new to TGCC, see this page for information on how to get an account.
Some parts of the preprocessing are complicated to execute on Irene-Rome due to access restrictions, and they are therefore run on the CLIMERI server spirit.
1. Module and Python environments
To access to all the project directories of your group, you need to do this (replace gen6035 by your project ID):
export PROJECT='gen6035' ## adapt project
module switch dfldatadir/$PROJECT
TMP="${PROJECT^^}_CCCSCRATCHDIR"; export SCRATCHDIR=${!TMP}
TMP="${PROJECT^^}_CCCWORKDIR"; export WORKDIR=${!TMP}
TMP="${PROJECT^^}_CCCSTOREDIR"; export STOREDIR=${!TMP}
TMP="${PROJECT^^}_ALL_CCCHOME"; export SHAREDHOME=${!TMP}
TMP="${PROJECT^^}_ALL_CCCSCRATCHDIR"; export SHAREDSCRATCHDIR=${!TMP}
TMP="${PROJECT^^}_ALL_CCCWORKDIR"; export SHAREDWORKDIR=${!TMP}
TMP="${PROJECT^^}_ALL_CCCSTOREDIR"; export SHAREDSTOREDIR=${!TMP}
To install a good python environment on TGCC, see this page, then define the alias to activate it:
alias py='source /ccc/workflash/cont003/gen6035/jourdain/py37/bin/activate'
All the commands above can typically be included in ~/.bashrc.
2. Get the sources
If not already done, create the a models repository on Irene-rome:
mkdir -pv ${WORKDIR}/models
Then, on spirit:
git clone --branch v2.1.0 https://gitlab.inria.fr/croco-ocean/croco.git croco-v2.1.0
git clone --branch v2.1.0 https://gitlab.inria.fr/croco-ocean/croco_tools.git croco_tools-v2.1.0
git clone https://gitlab.inria.fr/croco-ocean/croco_pytools.git croco_pytools-v1.0.3
# change paths and project name in the following lines:
rsync -av --chmod=Dg+s --chown=:gen6035 croco-v2.1.0 jourdain@irene-amd-fr.ccc.cea.fr:/ccc/work/cont003/gen6035/jourdain/models/.
rsync -av --chmod=Dg+s --chown=:gen6035 croco_tools-v2.1.0 jourdain@irene-amd-fr.ccc.cea.fr:/ccc/work/cont003/gen6035/jourdain/models/.
rsync -av --chmod=Dg+s --chown=:gen6035 croco_pytools-v1.0.3 jourdain@irene-amd-fr.ccc.cea.fr:/ccc/work/cont003/gen6035/jourdain/models/.
3. Prepare the running directory for a new configuration
Define the name of your new configuration:
export CONFIG='Saigon_02'
mkdir -pv ${SCRATCHDIR}/run_croco # where the simulations will be run
cd ${WORKDIR}/models/croco-v2.1.0
vi create_config.bash # change to MY_CONFIG_NAME=Run_${CONFIG}
./create_config.bash
Then, if this is the first time you create a configuration on Irene-rome, download Run_croco_save.tar on your local computer and copy it to Irene-rome in the WORKDIR. Then:
cd $WORKDIR
ls -al Run_croco_save.tar
tar xvf Run_croco_save.tar
4. Preprocessing: preparing the grid and forcing data
On spirit, get the datasets required for the preprocessing:
cd /scratchu/njourdain
wget https://data-croco.ifremer.fr/DATASETS/DATASETS_CROCOTOOLS.tar.gz
tar xzf DATASETS_CROCOTOOLS.tar.gz
rsync -av --chmod=Dg+s --chown=:gen6035 DATASETS_CROCOTOOLS jourdain@irene-amd-fr.ccc.cea.fr:/ccc/work/cont003/gen6035/jourdain/.
# new topography to be downloaded on https://cloud.univ-grenoble-alpes.fr/s/aTgTskfCaFd3nHf
rsync -av --chmod=Dg+s --chown=:gen6035 Topo_Mekong_Saigon_merged_3sec_2025_v0.nc jourdain@irene-amd-fr.ccc.cea.fr:/ccc/work/cont003/gen6035/jourdain/DATASETS_CROCOTOOLS/Topo/.
# new tide data to be downloaded on https://cloud.univ-grenoble-alpes.fr/s/PBaq8xjTCAGXtFp
rsync -av --chmod=Dg+s --chown=:gen6035 TPXO9v5_Mekong.nc jourdain@irene-amd-fr.ccc.cea.fr:/ccc/work/cont003/gen6035/jourdain/DATASETS_CROCOTOOLS/TPXO9/.
Then, on Irene-rome:
cd $SCRATCHDIR/run_croco/Run_${CONFIG}
cp -p $WORKDIR/Run_croco_save/crocotools_param.m
vi crocotools_param.m
4a- Create the grid(s)
To create the grid of the parent domain:
module load matlab
matlab -nodesktop
>> start
>> make_grid
% Do you want to link the GSHHS data coastlines (+ borders and rivers) ? y
% Do you want to use interactive grid maker ? y
# Updated values for Saigon_LR (2km resolution):
# xsize = 320
# ysize = 344
# Rotation = 20
# Longitu = 106.75
# Latitude = 10.2
% Do you want to use editmask ? y
# turn the part of the Gulf of Thailand into land
>> exit
ls -al CROCO_FILES/croco_grid.nc
To create the grid of the child domain(s), check this page, then:
module load matlab
matlab -nodesktop
>> start
>> nestgui
4b- Create the atmospheric forcing
To extract the surface conditions from ERA5, start doing this on spirit :
cd /scratchu/njourdain/croco_tools-v2.1.0/Aforc_ERA5
vi era5_crocotools_param.py # adapt everything (for direct ERA5 download on some pre-defined lon-lat domain which can be larger than the grid)
submit_python.sh ERA5_request.py 24 32
cd ../DATA
ls -al ERA5_native_Vietnam # or other name
rsync -av --chmod=Dg+s --chown=:gen6035 ERA5_native_Vietnam jourdain@irene-amd-fr.ccc.cea.fr:/ccc/work/cont003/gen6035/jourdain/DATASETS_CROCOTOOLS/.
Then, on Irene-rome:
cd ${SCRATCHDIR}/run_croco/Run_${CONFIG}
vi crocotools_param.m # adapt My_ERA5_dir
cd ${WORKDIR}/models/croco_tools-v2.1.0/Aforc_ERA5
vi era5_crocotools_param.py #adapt everything (put month_start=1 and month_end=12)
py
python ERA5_convert.py
ls ${SCRATCHDIR}/run_croco/Run_${CONFIG}/DATA/ERA5_Saigon_LR/
4c- Create the ocean lateral boundary conditions
To extract the lateral boundary consitions from GLORYS12, start doing this on spirit :
cd /scratchu/njourdain/croco_tools-v2.1.0/Oforc_OGCM
Then, at first use of this script, use the following command lines to put the right header in the existing bash script (still on spirit):
cat << EOF > tmp1.tmp
#SBATCH --ntasks=1
#SBATCH --mem=16000
#SBATCH --threads-per-core=1
#SBATCH -J glorys
#SBATCH -e glorys.e%j
#SBATCH -o glorys.o%j
#SBATCH --time=47:59:00
EOF
awk 'NR==2{while(getline line < "tmp1.tmp"){print line}}1' download_glorys_data.sh > tmp2.tmp
mv tmp2.tmp download_glorys_data.sh
rm -f tmp1.tmp
Then, still on spirit, adjust dates, area, frequency and run the script. Note that you need to create a login to access these data:
vi download_glorys_data.sh # edit dates, area, frequency, ...
sbatch ./download_glorys_data.sh
rsync -av --chmod=Dg+s --chown=:gen6035 raw_mercator_Y*M*.nc XXXXXXX
Then, back to Irene-rome, create this script to format the GLORYS outputs:
cd ${WORKDIR}/models/croco_tools-v2.1.0/Oforc_OGCM
cat << EOF > convert_raw2crocotools.m
clear all;
close all;
crocotools_param;
GLORYS_dir = ['/ccc/work/cont003/gen6035/jourdain/DATASETS_CROCOTOOLS/GLORYS12_Vietnam/']
vars = {'zos' ...
'uo' ...
'vo' ...
'thetao' ...
'so'};
disp(['Making output data directory ',OGCM_dir])
eval(['!mkdir ',OGCM_dir, ' 2> /dev/null'])
%-----
for Y=Ymin:Ymax
if Y==Ymin
mo_min=Mmin;
else
mo_min=1;
end
if Y==Ymax
mo_max=Mmax;
else
mo_max=12;
end
for M=mo_min:mo_max
thedatemonth=['Y',num2str(Y),'M',num2str(M)];
time1=datenum(Y,M,01);
time2=datenum(Y,M+1,01) - 1;
time=cat(2,time1,time2);
raw_mercator_name=[GLORYS_dir,'raw_',OGCM_prefix,thedatemonth,'.nc'];
disp(['Processing ',raw_mercator_name])
if mercator_type==1
write_mercator(OGCM_dir,OGCM_prefix,raw_mercator_name, ...
mercator_type,vars,time,thedatemonth,Yorig); % write data
end
end
end
EOF
cd ${SCRATCHDIR}/run_croco/Run_${CONFIG}
vi crocotools_param.m # level=0 ; OGCM = 'mercator';
# itolap_a = 2; % before
# itolap_p = 2; % after
# Also adapt YMIN, YMAX, MMIN, MMAX...
vi ${WORKDIR}/models/croco_tools-v2.1.0/Oforc_OGCM/make_OGCM_mercator.m
module load matlab
matlab -nodesktop
>> start
>> convert_raw2crocotools
>> make_OGCM_mercator
# check mercator files in CROCO format (generated by convert_raw2crocotools.m):
ls -al ${SCRATCHDIR}/run_croco/Run_${CONFIG}/DATA/mercator_Saigon_LR/.
# check initial file(s) (generated by make_OGCM_mercator.m):
ls -al ${SCRATCHDIR}/run_croco/Run_${CONFIG}/CROCO_FILES/croco_ini_mercator_*.nc
# check bdy files (generated by make_OGCM_mercator.m):
ls -al ${SCRATCHDIR}/run_croco/Run_${CONFIG}/CROCO_FILES/croco_bry_mercator_*.nc
If you use passive tracers, you need to manually add variable “tpasxx” in the initial state and boundary condition, with xx in 01, 02, 03, etc. This can be done like this:
cd ${SCRATCHDIR}/run_croco/Run_${CONFIG}/CROCO_FILES
module load nco
# Initial state :
for file in croco_ini_mercator_*
do
newfile=`echo $file |sed -e "s/mercator/mercator_tpas/g"`
oldfile=`echo $file |sed -e "s/mercator/mercator_old/g"`
ncap2 -s 'tpas01=salt*0.0 ; tpas02=salt*0.0 ; tpas03=salt*0.0 ; tpas04=salt*0.0 ; tpas05=salt*0.0 ; tpas06=salt*0.0 ; tpas07=salt*0.0' $file $newfile
for ipas in $(seq 1 7)
do
ncatted -a long_name,tpas0${ipas},m,c,"tpas0${ipas} passive tracer" $newfile
ncatted -a units,tpas0${ipas},m,c,'no unit' $newfile
done
mv $file $oldfile
ln -s -v $newfile $file
done
4d- Create the tides lateral boundary conditions
cd ${SCRATCHDIR}/run_croco/Run_${CONFIG}
vi crocotools_param.m # level = 0 ; check that tidename and salname point to the right dataset
vi ${WORKDIR}/models/croco_tools-v2.1.0/Tides/make_tides_interannual.m
module load matlab
matlab -nodesktop
>> start
>> make_tides_interannual
# check that files have been generated:
ls -al CROCO_FILES/croco_frc_*.nc
4e- Create boundary conditions for passive tracers (optional)
TBC
5. Compile and run the regional model
5a- Compilation
The compilation has to be redone for any change in the grid size or model options.
cd ${SCRATCHDIR}/run_croco/Run_${CONFIG}
cp -p $WORKDIR/Run_croco_save/jobcomp . # adapted for Irene-rome
cp -p $WORKDIR/Run_croco_save/myenv_mypath.sh . # adapted for Irene-rome
cp -rp $WORKDIR/Run_croco_save/MY_SRC .
vi MY_SRC/cppdefs.h # Change configuration name # define SAIGON_LR
# under "if defined REGIONAL" :
# ----- define whether an AGRIF nest is used -----
# undef AGRIF
# undef AGRIF_2WAY
# ----- define whether tides and OBCs are used -----
# define TIDES
# define OBC_EAST
# define OBC_WEST
# undef OBC_NORTH
# define OBC_SOUTH
# ----- passive tracers -----
# define PASSIVE_TRACER (in the REGIONAL part)
# ----- define point river sources as mass fluxes -----
# undef PSOURCE
# define PSOURCE_MASS (check double S: MASS !!)
vi MY_SRC/param.h # Change LLm0 and MMm0. They correspond to the number of interior points
# (domain without boundaries). It means xi_rho -2 and eta_rho - 2, in e.g. :
# # elif defined SAIGON_LR
# parameter (LLm0=172, MMm0=185, N=10)
# Change the number of MPI domains by specifying NP_XI and NP_ETA
# (the product should be the number of PROCs required in the bash job).
# If you use passive tracers, you may need to change ntrc_pas (for the case where KH_INST is not defined)
Then, compile the code:
./jobcomp # should end with "CROCO is OK"
ls -al croco
5b- Running CROCO
At first use of run_croco_inter.bash, use the following command lines to put the right header in the existing bash script:
cp -p run_croco_inter.bash run_croco_inter.bash_save
cat << EOF > tmp1.tmp
#MSUB -r run_croco
#MSUB -o run_croco.o%j
#MSUB -e run_croco.e%j
#MSUB -n 8
#MSUB -x
#MSUB -T 86000
#MSUB -A gen6035
#MSUB -q rome
#MSUB -m store,work,scratch
EOF
awk 'NR==2{while(getline line < "tmp1.tmp"){print line}}1' run_croco_inter.bash > tmp2.tmp
mv tmp2.tmp run_croco_inter.bash
rm -f tmp1.tmp
Then, adapt this script further:
vi run_croco_inter.bash # RUNCMD="ccc_mprun ./"
# NBPROCS=54 # = NP_XI * NP_ETA in param.h
# BULK_FILES=1
# FORCING_FILES=1
# BOUNDARY_FILES=1
# ATMOS_BULK=ERA5
# ATMOS_FRC=TIDETPXO
# OGCM=mercator
# DT=120
#
# Set the output frequency and averaged period (in nb of days): ND_AVG = 1 (for 1-day outputs).
# If you want sub-daily outputs, e.g. 20 outputs per day, replace ND_AVG with ND_AVG_H (in hours), then
# replace NUMAVG=$((ND_AVG * 86400 / DT )) with NUMAVG=$((ND_AVG_H * 3600 / DT ))
# (check that you have a finite number of outputs per hour)
Then adapt this script:
vi croco_inter.in # Put the number of runoff points in the "psource:" section
# with Nsrc the number of source points and
# one line per source point with grid location, discharge (m3/s), temperature (degC) and salinity (psu)
# Isrc and Jsrc are interior indices (in other words, python indexing convention on croco_grid.nc)
# Dsrc not used if PSOURCE_MASS rather than PSOURCE is defined.
# If PASSIVE_TRACER is defined, you need to put Lsrc to T for each tracer and
# add the tracer concentrations at the end of the line in addition to T,S, e.g. for 4 passive tracers :
# psource: Nsrc Isrc Jsrc Dsrc Qbar [m3/s] Lsrc Tsrc
# 2
# 23 602 0 <Q_Chau_Doc> 6*T 30. 0.1 100.0 0.0 0.0 0.0
# 75 641 0 <Q_Tan_Chau> 6*T 30. 0.1 0.0 100.0 0.0 0.0
#
# Adapt dates on 2nd last line (nb of time step per day in ERA5, last year, last month)
# and change last line to something like:
# /ccc/scratch/cont003/gen6035/jourdain/run_croco/Run_Saigon_02/DATA/ERA5_Saigon_LR/
Then launch the simulation:
source myenv_mypath.sh
ccc_msub ./run_croco_inter.bash
ccc_mpp -u `whoami`
Once completed, you can look at the outptuts:
ls -al croco_Y*M*.out # CROCO's standard output
ncdump -h SCRATCH/croco_avg.nc # time-averaged (e.g. 3-day) gridded outputs
ncdump -h SCRATCH/croco_his.nc # instantaneous gridded output (after first time step?)
5b- Running CROCO with an AGRIF nest
Generate ocean initial state and tidal forcing for the child grid:
cd ${SCRATCHDIR}/run_croco/Run_${CONFIG}
vi crocotools_param.m # level = 1
# choose date of initial state
module load matlab
matlab -nodesktop
>> start
>> make_OGCM_mercator
>> make_tides_interannual
ls -al CROCO_FILES/croco_ini_mercator_Y*.nc.1
Adjust grid and recompile:
ncdump -v grd_pos CROCO_FILES/croco_grd.nc.1
vi AGRIF_FixedGrids.in # adjust if inconsistent with above
vi MY_SRC/cppdefs.h # define AGRIF
# define AGRIF_2WAY
./jobcomp
NB: There may be a problem when using AGRIF and PASSIVE TRACER as it forces a 5th order scheme for tracer advection (to ensure positivity of the scheme), as described here. You basically need to replace ‘Npts’ with ‘MAXNPTS’ in the definition of size_X, size_E and size_Z for the case with AGRIF defined in MessPass2D.F and MessPass3D.F.
Then, same as above, but: put NLEVEL = 2 in run_croco_inter.bash and edit vi croco_inter.in.1