Work in progress to enable stores that reference bytes created outside Zarr (via virtualizarr)

• Manifest storage transformer zarr-developers/zarr-specs#287
• Zarr as a "universal reader" for netCDF etc., via new CF decoding codecs zarr-developers/zarr-specs#303

Notes

1. python3 virtualizarr_gen.py
2. Apply corrections for non-conformant zarr.json metadata
3. cargo test chunk_manifest

• Chunk manifests with URIs would make this much more complicated
  • URI parsing and store registration
  • Store caching
  • Auth?
  • ...
• Should absolute paths / relative paths outside of the store be permitted?

TODO

virtualizarr

manifest.json

• virtualizarr should write paths relative to the hierarchy root. Currently they are relative to the working directory.
  • Or is there a more general way of handling this (via metadata changes)? The store must be opened at the same location as the CWD when virtualize.to_zarr was called. Can't just open a store pointing directly at the hierarchy and then open an array at the root store prefix...
• Absolute paths Paths as URIs zarr-developers/VirtualiZarr#243

zarr.json

• Add variable length chunks support

  Cannot concatenate arrays with inconsistent chunk shapes: (...) vs (...) .Requires ZEP003 (Variable-length Chunks)

  • zarrs already supports variable length chunks
• chunk key encoding should match the path encoding ("0.0" etc).
  • the chunk-manifest-json storage transformer should not need to be aware of the chunk key encoding

    "chunk_key_encoding": {
        "configuration": {
            "separator": "."
        },
        "name": "v2"
    },

• Defaults to "codecs": null, without netCDF compression. Change to Conformant ZarrV3 codecs and fill values zarr-developers/VirtualiZarr#193

    "codecs": [
        {
            "name": "bytes",
            "configuration": {
                "endian": "little"
            }
        }
    ],

• Defaults to "fill_value": null. Must be set to a compatible value. Conformant ZarrV3 codecs and fill values zarr-developers/VirtualiZarr#193

zarrs

• Test on real/multifile netCDF data
• Support NetCDF/HDF5 compression
  • ZLIB codec
  • SZIP codec