Table of content
A short video explaining what the “Common Workflow Language” is in 64 seconds
As part of this Work Package we investigate the use of workflows to improve reproducibility and automation of a sequence of tasks required for running an Earth System Model on various computing platforms, perform data processing, create graphs, etc. As an example a simple workflow was developped on Galaxy Climate with FATES, the Functionally Assembled Terrestrial Ecosystem Simulator which is part of the Community Terrestrial Systems Model (CTSM). The tasks involve:
Each task in such workflows rely on containers, and work in this area has also progressed with Docker container images developed (both with the homemade GCC toolchain and using “standard” packages from conda-forge/bioconda) and successfully tested with CESM/NorESM on Virtual Machines from the Norwegian Research and Education Cloud (NREC), on the Google Cloud Platform (GCP) and on Personal Computers (all multiple cores single node applications). This kind of container is particularly suited for training and model development purposes for which using a High Performance Computer (HPC) would be a waste of resources. Singularity is now available on HPCs from Sigma2 (Saga, Fram and Betzy), and our next challenge will be to develop portable containers to carry out tests with multiple nodes and assess the performance of containerized applications compared to “bare metal” (ran directly on the host).
NordicESMhub/eosc-nordic-jupyterhub: v0.8.2 JupyterHub conda environments for the NIRD toolkit
NordicESMhub/container-noresm: Version 2.0.0 NorESM container for running on HPC with singularity
NordicESMhub/noresm-containers-timings: Jupyter notebook for plotting timings from NorESM/CESM simulations
Our first report on the identified bottlenecks for an efficient usage of Nordic ESMs on EuroHPC is now available online.