Trial VirtualiZarr

[mattjbr123]

Trial VirtualiZarr for this workflow - avoids the need for beam but might lose the nice plug-and-play recipe creation of pangeo-forge-recipes.

[mattjbr123]

The background and preamble:

Pangeo-Forge-Recipes (PFR), or at least PFR as I know it, looks likely to be deprecated. Given the problems I have run into when using anything other than the Beam DirectRunner in order to scale up the workflow, it might be time to consider an alternative.

@TomNicholas @alxmrs have advised that VirtualiZarr with Cubed/Dask is a good alternative, and where the pangeo community's efforts are focused now, and more likely to have long-term support. The long-term support is important for FDRI, the parent project of this repo, and is a major advantage over PFR. The other major advantage being no more Beam wrangling...

But, we would lose the nice pipeline-style plug-and-play drag-and-drop build-your-own-recipe idea that PFR brings, which makes it more work to build something on top of this to make it easy for scientists to convert their data to ARCO (which is another important aspect of FDRI: no point building it if no one uses it). More work, but still possible. And a working scaled conversion tool is better than one that doesn't work at scale...

VirtualiZarr also does not allow for rechunking and conversion of chunk-sizes (as all it does is create references to the existing chunks in the form of a zarr dataset). Given how badly chunked most NetCDF datasets usually are, this could result in poorer performance than would be obtained with the original PFR rechunk-to-zarr approach I've gone with here. The desire to rechunk to better chunk sizes comes from this analysis, and my original assumption was that the key performance conclusions hold even when you're using Kerchunk/VirtualiZarr to access the data. But this might not be true, chunking is complicated.

All that said, there's not much to lose and lots to gain from just giving VirtualiZarr a spin on our trial dataset here, and see what happens...

The technical details are:
~15 years of hourly data on a 1km grid over the UK in monthly NetCDF files.
Dataset dimensions: ~131400 x 1000 x 1000 (time x Y x X)
File dimensions: ~720 x 1000 x 1000 - time dimension size variable as no. of hours in a month varies
No. of files: 180
Original chunk sizes: ~68 x 114 x 64 (also varies a little by file, helpfully)

What I have been doing is rechunking these to ~8760 x 100 x 100 (yearly, 100km) into zarr.

[TomNicholas]

(also varies a little by file, helpfully)

This is likely a blocker to using VirtualiZarr until ZEP003 (variable-length chunks) is accepted and implemented in Zarr. This limitation would have applied to Kerchunk too.

[mattjbr123]

Ok, thanks Tom, I'll keep this issue open but move it to the backlog as a "wait and see"