CCSP docs and example schemas #747

braingram · 2025-10-21T16:22:17Z

This includes:

docs updates describing CCSP schemas. Further documentation will be added as interaction with the PITs proceeds and more is learned about their data products and needs. MAST will link to these new rad docs in a document they are preparing to provide to the PITs.
add a latest/CCSP directory which will house all CCSP/PIT schemas. These will be organized by team name (CCSP/SN for the SN team, etc). Tests were updated to check this new structure.
2 component schemas are added ccsp_minimal and ccsp_custom_product that will be referenced in CCSP schemas. The docs describes when each is supposed to be used and tests were added to check for the structure mentioned in the docs.
2 example CCSP example schemas are added example_custom_product and example_derived_mosaic. We may want to delete these prior to the next release if we have real PIT products (or even if we don't). They are useful for testing and illustration purposes but those benefits weight against the need to maintain functionally useless schemas.

Link to built docs: https://rad--747.org.readthedocs.build/en/747/

Tasks

Update or add relevant rad tests.
Update relevant docstrings and / or docs/ page.
Does this PR change any schema files?
- Schema changes were discussed at RAD Review Board meeting.
Does this PR change any API used downstream? (If not, label with no-changelog-entry-needed.)
- Write news fragment(s) in changes/: echo "changed something" > changes/<PR#>.<changetype>.rst (see below for change types).
- Start a romancal regression test (https://github.com/spacetelescope/RegressionTests/actions/workflows/romancal.yml) with this branch installed ("git+https://github.com/<fork>/rad@<branch>").
- Update relevant roman_datamodels utilities and tests.

News fragment change types:

changes/<PR#>.feature.rst: new feature
changes/<PR#>.bugfix.rst: fixes an issue
changes/<PR#>.doc.rst: documentation change
changes/<PR#>.removal.rst: deprecation or removal of public API
changes/<PR#>.misc.rst: infrastructure or miscellaneous change

Co-authored-by: Brett Graham <[email protected]>

We will instead enforce many requirements upon reference in the science product schemas

Based on MAST feedback we want to allow the posibility for a schema that doesn't reference a SOC schema to reference ccsp_minimal instead. The unit test that checks this was updated to allow for this case.

adrianlucy

This looks excellent! I just have a bunch of minor proposed changes. (For proposed changes to example_custom_product and example_derived_mosaic, I can adjust the docs later in accordance with any changes that get approved.)

docs/ccsp.rst

latest/CCSP/ccsp_custom_product.yaml

docs/ccsp.rst

latest/CCSP/EXAMPLE/example_derived_mosaic.yaml

Co-authored-by: Adrian Lucy <[email protected]>

for more information, see https://pre-commit.ci

docs/ccsp.rst

Co-authored-by: Adrian Lucy <[email protected]>

docs/ccsp.rst

Co-authored-by: Adrian Lucy <[email protected]>

PaulHuwe · 2025-10-25T21:12:42Z

latest/CCSP/ccsp_custom_product.yaml

+              center of the spatial image, or the target coordinates
+              of a spectrum or light curve. May be null for large
+              survey images, simulated data, etc.
+            type: number


Description says it can be null, so does that need to be allowed in the type?

PaulHuwe · 2025-10-25T21:12:50Z

latest/CCSP/ccsp_custom_product.yaml

+              center of the spatial image, or the target coordinates
+              of a spectrum or light curve. May be null for large
+              survey images, simulated data, etc.
+            type: number


Description says it can be null, so does that need to be allowed in the type?

PaulHuwe · 2025-10-25T21:17:12Z

latest/CCSP/ccsp_custom_product.yaml

+            description: |
+              Waveband of this data product, used to enable archival
+              searches. Select from: RADIO, MILLIMETER, INFRARED,
+              OPTICAL, UV, EUV, XRAY, GAMMARAY.


Should these be an enum for the strings below?

PaulHuwe · 2025-10-25T21:51:11Z

src/rad/resources/manifests/datamodels-1.5.0.yaml

@@ -1 +0,0 @@
-../../../../latest/manifests/datamodels.yaml


Shouldn't we be preserving datamodels-1.5.0.yaml‎? We have with previous versions.

braingram and others added 30 commits October 21, 2025 11:44

add HLIS PIT shear image schema

bec605f

add ccsp_mast_metadata

d5871d5

fix tests

bb37239

add HLIS PIT shear image schema

3ecbf21

Add pit_basic and pit_common schemas

5f150a4

Change floats to numbers

c98bff0

Remove ccsp_mast_metadata.yaml

55b02bb

Change schema titles, add descriptions

57ffc6b

Add endmatter lines

364c3b5

Clean up line wraps

2a5f7cb

cleaning up schema organization

97341e4

fix case

0da0444

fix style

58d068a

fix shear_image meta ref to pit_common

5456b81

Change title of pit_basic

8159aef

Co-authored-by: Brett Graham <[email protected]>

Edit source_observations_id title and description

6b69384

Co-authored-by: Brett Graham <[email protected]>

Move source_observations_id to basic

3bc67b2

Typos

742ffd9

Delete mjd times

90b0425

Add array type to instrument keys

80d1186

Remove spatial_dimensions

4f13e5a

Add nesting, drop unnecessary requirements and null patterns

2e54925

We will instead enforce many requirements upon reference in the science product schemas

Renaming to ccsp + rearrangement of ccsp_minimal under a ccsp key

acb78dc

Rename files to ccs_custom_product and ccsp_minimal

24d4c2c

Add link to UAT concept selector

66865b8

Rename PIT directory to CCSP

a59625e

Fix sym links

14828ed

Change ra/dec key descriptions

fe1af1a

Revise shear_image to accomodate changes in referenced schemas

3623fa6

In shear_image, move required list on ref into allOf

9b81ef7

braingram added 5 commits October 22, 2025 12:22

align examples and docs

1b68759

prefix schemas

fc4615b

update latest path test for tagged schemas

47778c7

add test that ccsp models ref expected schemas

2c3040e

update test to check for soc schema refs

b9b8291

braingram changed the title ~~WIP CCSP docs and schemas~~ CCSP docs and example schemas Oct 22, 2025

braingram mentioned this pull request Oct 23, 2025

Add example models required for ccsp docs spacetelescope/roman_datamodels#597

Open

5 tasks

update ccsp_minimal check test

19a6899

Based on MAST feedback we want to allow the posibility for a schema that doesn't reference a SOC schema to reference ccsp_minimal instead. The unit test that checks this was updated to allow for this case.

adrianlucy reviewed Oct 24, 2025

View reviewed changes

latest/CCSP/EXAMPLE/example_derived_mosaic.yaml Show resolved Hide resolved

braingram and others added 9 commits October 24, 2025 13:55

Update docs/ccsp.rst

f5b1636

Co-authored-by: Adrian Lucy <[email protected]>

Update docs/ccsp.rst

63ed3cd

Co-authored-by: Adrian Lucy <[email protected]>

Update docs/ccsp.rst

3a3a52e

Co-authored-by: Adrian Lucy <[email protected]>

Update docs/ccsp.rst

a849e53

Co-authored-by: Adrian Lucy <[email protected]>

Update docs/ccsp.rst

b53a7de

Co-authored-by: Adrian Lucy <[email protected]>

Update docs/ccsp.rst

cbf80d5

Co-authored-by: Adrian Lucy <[email protected]>

Update docs/ccsp.rst

1ea8b09

Co-authored-by: Adrian Lucy <[email protected]>

Apply suggestions from code review

95a61c2

Co-authored-by: Adrian Lucy <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

515c0a3

for more information, see https://pre-commit.ci

adrianlucy reviewed Oct 24, 2025

View reviewed changes

docs/ccsp.rst Outdated Show resolved Hide resolved

braingram and others added 3 commits October 24, 2025 15:05

Update docs/ccsp.rst

091354c

Co-authored-by: Adrian Lucy <[email protected]>

update docs to align with schema changes

68a2341

update docs to align with schema changes

b7473f1

adrianlucy reviewed Oct 24, 2025

View reviewed changes

docs/ccsp.rst Outdated Show resolved Hide resolved

Update docs/ccsp.rst

67eae74

Co-authored-by: Adrian Lucy <[email protected]>

braingram marked this pull request as ready for review October 24, 2025 19:41

braingram requested review from a team and WilliamJamieson as code owners October 24, 2025 19:41

braingram requested a review from PaulHuwe October 24, 2025 21:01

PaulHuwe requested changes Oct 25, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

CCSP docs and example schemas #747

CCSP docs and example schemas #747

Uh oh!

braingram commented Oct 21, 2025 •

edited

Loading

Uh oh!

adrianlucy left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

PaulHuwe Oct 25, 2025

Uh oh!

PaulHuwe Oct 25, 2025

Uh oh!

PaulHuwe Oct 25, 2025

Uh oh!

PaulHuwe Oct 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		@@ -1 +0,0 @@
		../../../../latest/manifests/datamodels.yaml No newline at end of file

CCSP docs and example schemas #747

Are you sure you want to change the base?

CCSP docs and example schemas #747

Uh oh!

Conversation

braingram commented Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tasks

Uh oh!

adrianlucy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

PaulHuwe Oct 25, 2025

Choose a reason for hiding this comment

Uh oh!

PaulHuwe Oct 25, 2025

Choose a reason for hiding this comment

Uh oh!

PaulHuwe Oct 25, 2025

Choose a reason for hiding this comment

Uh oh!

PaulHuwe Oct 25, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

braingram commented Oct 21, 2025 •

edited

Loading