example dataset for inference

[einzigsue]

Is it the following dataset defined in src/cbottle/config/environment.py supposed to be accessible by non-developers?

s3://asubramaniam-testing/ICON_v6_dataset.zarr

The inference with the coarse generator python scripts/inference_coarse.py cBottle-3d.zip inference_output --sample.min_samples 1 returns to the following error message.

INFO:root:opening s3://asubramaniam-testing/ICON_v6_dataset.zarr/
INFO:aiobotocore.credentials:Found credentials in environment variables.
[rank0]: Traceback (most recent call last):
[rank0]:   File "/g/data/z00/yxs900/HEALPix/repo/cBottle/scripts/inference_coarse.py", line 336, in <module>
[rank0]:     main()
[rank0]:   File "/g/data/z00/yxs900/HEALPix/repo/cBottle/scripts/inference_coarse.py", line 312, in main
[rank0]:     dataset = dataset_3d.get_dataset(
[rank0]:               ^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/conda/envs/cbottle/lib/python3.12/site-packages/cbottle/datasets/dataset_3d.py", line 729, in get_dataset
[rank0]:     ds = dataset_func(
[rank0]:          ^^^^^^^^^^^^^
[rank0]:   File "/opt/conda/envs/cbottle/lib/python3.12/site-packages/cbottle/datasets/dataset_3d.py", line 388, in _get_dataset_icon
[rank0]:     ZarrLoader(
[rank0]:   File "/opt/conda/envs/cbottle/lib/python3.12/site-packages/cbottle/datasets/zarr_loader.py", line 78, in __init__
[rank0]:     self.group = sync(
[rank0]:                  ^^^^^
[rank0]:   File "/opt/conda/envs/cbottle/lib/python3.12/site-packages/zarr/core/sync.py", line 163, in sync
[rank0]:     raise return_result
[rank0]:   File "/opt/conda/envs/cbottle/lib/python3.12/site-packages/zarr/core/sync.py", line 119, in _runner
[rank0]:     return await coro
[rank0]:            ^^^^^^^^^^
[rank0]:   File "/opt/conda/envs/cbottle/lib/python3.12/site-packages/zarr/api/asynchronous.py", line 839, in open_group
[rank0]:     raise FileNotFoundError(f"Unable to find group: {store_path}")
[rank0]: FileNotFoundError: Unable to find group: <FsspecStore(S3FileSystem, asubramaniam-testing/ICON_v6_dataset.zarr)>

[HP-eter]

I could also use an inference example for ERA5 data. I convert ERA5 single-level fields (downloaded via the CDS API) from a regular lat-lon grid to HEALPix (level 6, nside = 64) with earth2grid.
The regridded maps look correct in double-pixel visualisations I downloaded the following variables : u10, v10, t2m, msl, sst, tp, ssrd, tsr, ttr, tciw, tclw, siconc

I then rename those variables using the following mapping

{"u10":"uas","v10":"vas","t2m":"tas","msl":"pres_msl","sst":"sst", "tciw":"clivi","tclw":"cllvi","siconc":"sic","tp":"pr", "ssrd":"rsds","tsr":"rsut","ttr":"rlut"}

I then append a crs metadata with the required fields, but when I run them through the cBottle_SR model the resulting images are noisy (see attached examples).

[belkhir-nacim]

Hello guys,
First thank you for your amazing work. The is truly inspirational.
After hours of trial, can you provide us some data sample for running the code. I see in the code that there data download however the data is stored in a restricted NVIDIA S3...
Would it be possible to share some data sample, or give more details/scripts to transform open data to evaluate your code.

Cheers,

[nbren12]

Apologies. We have not released the input data for the ICON inference mode yet.

But the input data for ERA5 inference is available and should be automatically downloaded if you run with --dataset amip. Please see follow this tutorial: https://nvlabs.github.io/cBottle/amip_inference/.

Note to self: change the default to --dataset amip.

[nbren12]

@HP-eter I'm impressed you got this far.

As a debugging step, I suggest starting with end2end synthesis. You can generate a coarse ERA5 netCDF file by following this tutorial: https://nvlabs.github.io/cBottle/amip_inference/. Then, you can compare this with your data from CDS and try to find some difference. Perhaps the data are stored in different units?

[loliverhennigh]

Hey @HP-eter, This is fantastic work! We are actually working to get super resolution model in E2Studio now (as well as the other models). This is fairly high priority so hopefully we will have have a PR in the next week or two. Ill keep updating the progress here.

[HP-eter]

We solved the issue with @belkhir-nacim, we will be pushing a pull request soon !