Add JOSS paper #1703

jfrost-mo · 2025-09-17T16:07:25Z

Still a draft, but please do comment!

Contribution checklist

Aim to have all relevant checks ticked off before merging. See the developer's guide for more detail.

Documentation has been updated to reflect change.
New code has tests, and affected old tests have been updated.
All tests and CI checks pass.
Ensured the pull request title is descriptive.
Conda lock files have been updated if dependencies have changed.
Attributed any Generative AI, such as GitHub Copilot, used in this PR.
Marked the PR as ready to review.

github-actions · 2025-09-17T16:09:26Z

It is still very much WIP.

This might need to be replaced with Earth Science New Zealand.

jfrost-mo · 2025-09-22T14:50:36Z

Current version: paper.pdf

daflack

Overall looking good. I've made several comments for what are generally minor wording tweaks just to improve accessibility, but see what you think.

docs/papers/paper.md

daflack · 2025-09-23T09:36:14Z

docs/papers/paper.md

+
+<!-- A summary describing the high-level functionality and purpose of the software for a diverse, non-specialist audience. -->
+
+The _Convective- [and turbulence-] Scale Evaluation Toolkit_ (**CSET**) is a community-driven open source library, command line tool, and workflow designed to support the evaluation of weather and climate models at convective and turbulent scales.


Is it better to put convective and turbulent scales as something more accessible like: km and 100 m scales or kilometre and hectometric scales?

That sounds good, especially as we already mention the convective and turbulence scales in CSET's name. Hectometric is probably too jargony, so maybe

support the evaluation of weather and climate models at kilometre and hundred metre scales.

daflack · 2025-09-23T09:37:14Z

docs/papers/paper.md

+<!-- A summary describing the high-level functionality and purpose of the software for a diverse, non-specialist audience. -->
+
+The _Convective- [and turbulence-] Scale Evaluation Toolkit_ (**CSET**) is a community-driven open source library, command line tool, and workflow designed to support the evaluation of weather and climate models at convective and turbulent scales.
+Developed by the Met Office in collaboration with the [Momentum® Partnership][momentum_partnership] and broader research community, CSET provides a reproducible, modular, and extensible framework for model diagnostics and verification.


Not clear what "broader research community" means here - are we talking industry, academic partners, something else?

Essentially anyone who is not in the Momentum Partnership. Currently this is just academics.

docs/papers/paper.md

daflack · 2025-09-23T10:34:19Z

docs/papers/paper.md

+Evaluation is essential for the model development process in atmospheric sciences.
+Typically, an evaluation includes both context and justification to demonstrate the benefit of model changes against other models or previous model versions.
+The verification provides the context or baseline for understanding the model’s performance through comparison against observation.
+The evaluation then demonstrates the benefit through comparison against theoretical expectations or previous or different version of the model and other models for similar application areas using diagnostics derived from model output to explain the context.


Suggested change

The evaluation then demonstrates the benefit through comparison against theoretical expectations or previous or different version of the model and other models for similar application areas using diagnostics derived from model output to explain the context.

The evaluation then demonstrates the benefit through comparison against theoretical expectations or known characteristics of previous or different versions of the same model, and (or) other models for similar applications using diagnostics derived from model output to explain the context.

I'm getting a bit lost in this sentence. I've altered it a little, but see what you think, I'm still not fully happy with what I have suggested. So, I think the main message is that there needs to be some further iteration on this sentence.

I'll have a crack at it.

daflack · 2025-09-23T10:43:18Z

docs/papers/paper.md

+The recipes and operators within CSET are well-documented, tested, and peer reviewed, increasing discoverability and giving confidence to users.
+The documentation covers information on the applicability and interpretation of diagnostics, ensuring they are appropriately used.
+
+CSET has been built with portability in mind.


Should not be starting a sentence with an acronym.

daflack · 2025-09-23T10:43:35Z

docs/papers/paper.md

+CSET has been built with portability in mind.
+It can run on a range of platforms, from laptops to supercomputers, and can be easily installed from conda-forge.
+It is built on a modern software stack that is underpinned by Cylc (a workflow engine for complex computational tasks) [@cylc8], Python 3, and Iris (a Python library for meteorological data analysis) [@scitools_iris].
+CSET is open source under the Apache-2.0 licence, and actively developed on GitHub, with extensive automatic unit and integration testing.


Should not be starting a sentence with an acronym.

daflack · 2025-09-23T10:43:57Z

docs/papers/paper.md

+
+# Conclusion
+
+CSET shows the benefits of open source evaluation software.


Should not be starting a sentence with an acronym.

daflack · 2025-09-23T10:45:40Z

docs/papers/paper.md

+
+<!-- Acknowledgement of any financial support. -->
+
+We acknowledge contributions and support from the Met Office and Momentum® Partnership for this project.


You should also be acknowledging funding e..g WCSSP projects that are contributing to people's time/have contributed towards peoples time and naming who is being supported by them.

Do we have a good list? Off the top of my head:

WCSSP South East Asia

WCSSP South Afric

NGMS, ML-INT (Not sure as these are internal Met Office programs.)

daflack · 2025-09-23T10:46:40Z

docs/papers/paper.md

+
+# Contribution to the field
+
+CSET addresses the need for an evaluation system that supports consistent and comparable evaluation.


Please don't start a sentence with an acronym.

Co-authored-by: David Flack <[email protected]>

jfrost-mo

Thank you for the comments.

I'll go ahead and adjust some of the wording as you suggest, and I'll also have a go at deacronyming the sentence starts.

jfrost-mo · 2025-09-23T14:24:29Z

docs/papers/paper.md

+The justification side of CSET consists of a range of diagnostics derived from model output.
+These derived diagnostics include process-based diagnostics for specific atmospheric phenomena and impact-based diagnostics that can be used to understand how model changes will affect customers.
+
+Compared to alternative open source evaluation tools, such as ESMValTool [@esmvaltool], CSET is focused on weather-relevant time scales and evaluating models towards a goal of operational usage.


Nothing JOSS specific.

I chose that reference as ESMValTool has a CITATION.cff file in their repository, which is used to indicate the preferred citation.

There are plenty of other good ESMValTool papers¹ though, so I'm not really sure which one to pick.

Footnotes

Such as https://gmd.copernicus.org/articles/18/4009/2025/ and https://gmd.copernicus.org/articles/18/1169/2025/ ↩

jfrost-mo · 2025-09-24T08:57:30Z

docs/papers/cset_ui.png

Any suggestions for nicer images for CSET?

jfrost-mo · 2025-09-24T08:57:53Z

docs/papers/paper.md

+title: "CSET: Toolkit for evaluation of weather and climate models"
+date: 17 September 2025
+bibliography: paper.bib
+tags:


Any additional keywords to suggest?

jfrost-mo · 2025-09-24T09:04:08Z

docs/papers/paper.md

+  - Weather
+  - Climate
+  - Atmospheric Science
+authors:


To start with I've ordered the authors by amount of code/commits, with a cut off for those that have changed $\lt 10$ lines of code (which is the same as those who have changed $\lt 100$ lines).

I'm not very experienced with this aspect of publication, so would welcome any advice.

Additionally, we need to get everyone listed here to sign off on the paper, and make sure we have their preferred name and affiliation.

jfrost-mo · 2025-09-24T09:21:57Z

docs/papers/paper.md

+
+<!-- A summary describing the high-level functionality and purpose of the software for a diverse, non-specialist audience. -->
+
+The _Convective- [and turbulence-] Scale Evaluation Toolkit_ (**CSET**) is a community-driven open source library, command line tool, and workflow designed to support the evaluation of weather and climate models at convective and turbulent scales.


That sounds good, especially as we already mention the convective and turbulence scales in CSET's name. Hectometric is probably too jargony, so maybe

support the evaluation of weather and climate models at kilometre and hundred metre scales.

jfrost-mo · 2025-09-24T09:23:47Z

docs/papers/paper.md

+<!-- A summary describing the high-level functionality and purpose of the software for a diverse, non-specialist audience. -->
+
+The _Convective- [and turbulence-] Scale Evaluation Toolkit_ (**CSET**) is a community-driven open source library, command line tool, and workflow designed to support the evaluation of weather and climate models at convective and turbulent scales.
+Developed by the Met Office in collaboration with the [Momentum® Partnership][momentum_partnership] and broader research community, CSET provides a reproducible, modular, and extensible framework for model diagnostics and verification.


Essentially anyone who is not in the Momentum Partnership. Currently this is just academics.

jfrost-mo · 2025-09-24T09:25:39Z

docs/papers/paper.md

+It gives users easy access to a wide selection of peer-reviewed diagnostics, including spatial plots, time series, vertical profiles, probability density functions, and aggregated analysis over multiple model simulations, replacing bespoke evaluation scripts.
+To cater for the full evaluation process, CSET provides a range of verification diagnostics to compare against observations and derived diagnostics based on model output, allowing for both physical process-based and impact-based understanding.
+
+<!-- TODO: Find a better image. -->


Definitely. I wanted an image of the output, and mainly put it in here to figure out the layout, but this was just a random screenshot I had to hand. I would welcome suggestions for what would make an interesting screenshot.

jfrost-mo · 2025-09-24T09:28:25Z

docs/papers/paper.md

+
+<!-- Acknowledgement of any financial support. -->
+
+We acknowledge contributions and support from the Met Office and Momentum® Partnership for this project.


Do we have a good list? Off the top of my head:

WCSSP South East Asia

WCSSP South Afric

NGMS, ML-INT (Not sure as these are internal Met Office programs.)

jfrost-mo · 2025-09-24T09:31:01Z

docs/papers/paper.md

+
+# Design
+
+CSET is built using operators, recipes and a workflow:


Is there a way to start the sentence without the acronym?

If you feel there is no way to start it without the acronym it should be spelt out in full rather than using the acronym (even if the acronym has already been defined).

jfrost-mo · 2025-09-24T09:33:35Z

docs/papers/paper.md

+Evaluation is essential for the model development process in atmospheric sciences.
+Typically, an evaluation includes both context and justification to demonstrate the benefit of model changes against other models or previous model versions.
+The verification provides the context or baseline for understanding the model’s performance through comparison against observation.
+The evaluation then demonstrates the benefit through comparison against theoretical expectations or previous or different version of the model and other models for similar application areas using diagnostics derived from model output to explain the context.


I'll have a crack at it.

jfrost-mo · 2025-09-24T10:55:20Z

docs/papers/paper.md

+  - name: John M Edwards
+    orcid:
+    affiliation: 1
+affiliations:


Should have "MetOffice@Reading" for Reading folk.

jfrost-mo self-assigned this Sep 17, 2025

jfrost-mo added documentation Improvements or additions to documentation full_review Requires a technical, scientific, and portability review labels Sep 17, 2025

jfrost-mo force-pushed the JOSS_paper branch 2 times, most recently from 36a59f6 to 050ab4f Compare September 22, 2025 12:33

jfrost-mo added 5 commits September 22, 2025 15:08

Initial version of JOSS paper

d178bef

It is still very much WIP.

Continue working on JOSS paper

b75b962

Add image of UI and tidy comments

02623a2

Fix and format YAML frontmatter

1b75bb3

Correct NIWA information

c474419

This might need to be replaced with Earth Science New Zealand.

jfrost-mo force-pushed the JOSS_paper branch from 9956711 to c474419 Compare September 22, 2025 14:08

jfrost-mo added 4 commits September 22, 2025 15:35

Add date for iris citation

bed630d

Adjust headings to match JOSS expection

df53cda

Add sentence on ESMValTool

dbbd8b1

Wording tweak

a25f503

jfrost-mo changed the title ~~Initial version of JOSS paper~~ Add JOSS paper Sep 22, 2025

daflack reviewed Sep 23, 2025

View reviewed changes

jfrost-mo and others added 4 commits September 24, 2025 09:56

Use David's preferred authorship information

fb6de0e

Co-authored-by: David Flack <[email protected]>

Verify multiple observations

aa01369

Co-authored-by: David Flack <[email protected]>

Clarify workflow variables

a3b7434

Co-authored-by: David Flack <[email protected]>

Build -> built

c3d599e

Co-authored-by: David Flack <[email protected]>

jfrost-mo commented Sep 24, 2025

View reviewed changes


		<!-- A summary describing the high-level functionality and purpose of the software for a diverse, non-specialist audience. -->

		The _Convective- [and turbulence-] Scale Evaluation Toolkit_ (CSET) is a community-driven open source library, command line tool, and workflow designed to support the evaluation of weather and climate models at convective and turbulent scales.

	The evaluation then demonstrates the benefit through comparison against theoretical expectations or previous or different version of the model and other models for similar application areas using diagnostics derived from model output to explain the context.
	The evaluation then demonstrates the benefit through comparison against theoretical expectations or known characteristics of previous or different versions of the same model, and (or) other models for similar applications using diagnostics derived from model output to explain the context.


		# Conclusion

		CSET shows the benefits of open source evaluation software.


		<!-- Acknowledgement of any financial support. -->

		We acknowledge contributions and support from the Met Office and Momentum® Partnership for this project.


		# Contribution to the field

		CSET addresses the need for an evaluation system that supports consistent and comparable evaluation.


		# Design

		CSET is built using operators, recipes and a workflow:

Add JOSS paper #1703

Are you sure you want to change the base?

Add JOSS paper #1703

Uh oh!

Conversation

jfrost-mo commented Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Contribution checklist

Uh oh!

github-actions bot commented Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jfrost-mo commented Sep 22, 2025

Uh oh!

daflack left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jfrost-mo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Footnotes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jfrost-mo commented Sep 17, 2025 •

edited

Loading

github-actions bot commented Sep 17, 2025 •

edited

Loading