Direkt zum Inhalt | Direkt zur Navigation

Personal tools

You are here: Home » Computing » M4HR » Metrics


Contributing to Balaji et al. [1], the IS-ENES2 WP JRA1 participated to the definition of the following set of metrics for assessing the sustained computational performance of Earth System Models.

The metrics are independent of hardware platform, programming and parallelisation paradigm, and provide a good basis for comparison across different ESMs. The collection of the metrics does not require any specialized measurement tools or hardware counters. Measurements can be done at two potential optima: one is to optimize time to solution by applying the maximum resource possible - speed (S) or capability mode ; or alternately, pick a spot lower down the scaling curve for the maximum aggregate simulated years for an ensemble of model runs within a given allocation - throughput (T) or capacity mode, which minimizes CHSY.

  • Resolution is measured as the number of grid points (or more generally, spatial degrees of freedom) NX x NY x NZ per component, denoted by Gc (where the subscript denotes a model component with an independent discretisation, typically atmosphere and ocean).
  • Complexity: Different measures were discussed before compromising on the number and dimension of variables in the ESM's restart files. The assumption is that the restart files represent the internal state of the ESM, thus allowing an estimate of the complexity to be deduced from the size dimension of the internal state space. The main advantage of this measure is that it is easy to obtain from any ESM.
  • Simulated years per day (SYPD): The number of years that can be simulated by the ESM in a given configuration on a given computing platform during a 24-hour period, assuming dedicated computing resources. Practically, this number is often deduced from shorter test runs.
  • Actual simulated years per day (ASYPD): The number of years that can be simulated by the ESM in a given configuration on a given platform in a multi-user environment (i.e. not assuming dedicated resources). This metric is usually measured using a long simulation with restarts, thus including queueing time between chunks, and workflow cost.
  • Core hours per simulated year (CHPSY): This metric measures the actual computational cost of the ESM simulation. It is usually determined by the product of the model run time and the number of cores used.
  • JPSY is the energy cost of a simulation, measured in Joules per simulated year. Given the energy E in Joules consumed over a budgeting interval T (generally 1 month or 1 year, in units of hours), and the aggregate compute hours A on a system (total cores x T) over the same interval T, we can measure the cost associated with 1 year of a simulation as follows: JPSY = CHSY x E / A
  • Memory bloat: This metric indicates the ratio of actual to ideal memory consumption of the ESM. The ideal consumption memory is deduced from complexity, as being the total memory needed to fit restart file variables. The actual memory is the only figure that requires a generic measurement tool, usually provided with the scheduler.
  • Coupler cost: Ratio of the time spent waiting in the coupler to the overall run time. This needs either a thorough performance analysis (tracing/profiling) or support in the coupler software.
  • Data output cost: Extra time that an ESM needs to write the model output to the file system. This is measured as the ratio of the run time for a standard run (including standard model output) to the run time for a run with model output switched off. For models using asynchronous I/O such as XIOS, a separate bank of PEs is allotted for I/O. In this case, it may be possible to measure it by simply looking at the allocation fraction of the I/O server, without needing a second “no I/O” run.
  • Data intensity: Amount of data that is read or written by an ESM in a given time during a typical run. For global climate models, it is mostly the written data that contributes to the data transfer, which is why the I/O speed metric may be limited to the output data.
  • Parallelisation: The number of computational units (cores or nodes as applicable) that is used for a certain ESM run. This number can be specified separately for the components of a coupled model and complemented by information about the parallelisation paradigm.
  • Platform: We propose two additional descriptors: chip name (e.g., Knights Landing) and machine name (e.g., titan). These should allow one to find links to configuration-specific information about the platform.

A systematic campaign is organised to collect the basic metric set in this paper routinely for CMIP6 before considering its growth and evolution. This will be done using currently planned systems of model documentation such as ES-DOC (http://es-doc.org) [2]. This comparative study of computational performance across models and machines, a CPMIP, will be an invaluable resource to the climate modeling community

[1] Balaji, V., Maisonnave, E., Zadeh, N., Lawrence, B. N., Biercamp, J., Fladrich, U., Aloisio, G., Benson, R., Caubel, A., Durachta, J., Foujols, M.-A., Lister, G., Mocavero, S., Underwood, S., and Wright, G., 2017: CPMIP: Measurements of Real Computational Performance of Earth System Models in CMIP6 , Geosci. Model Dev., 46, 19-34, doi:10.5194/gmd-10-19-2017

[2] Lawrence, B. N., Balaji, V., Bentley, P., Callaghan, S., DeLuca, C.,Denvil, S., Devine, G., Elkington, M., Ford, R. W., Guilyardi, E., Lautenschlager, M., Morgan, M., Moine, M.-P., Murphy, S.,Pascoe, C., Ramthun, H., Slavin, P., Steenman-Clark, L., Toussaint, F., Treshansky, A., and Valcke, S., 2012: Describing Earth system simulations with the Metafor CIM, Geosci. Model Dev., 5, 1493–1500, doi:10.5194/gmd-5-1493-2012