All Images

Data Formats, Metadata, and Vocabulary

Figure 1

Common file formats in environmental data, including NetCDF, HDF5, Zarr, GRIB, BUFR, GeoTIFF, GeoJSON, Shapefile, Parquet/GeoParquet, and CSV.

Figure 2

Word cloud of common metadata elements, including title, abstract, keywords, contact, spatial extent, temporal extent, units, standard_name, cell_methods, coordinates, bounds, grid_mapping, quality flags, provenance.
Source

Challenges of N-Dimensional Data

Figure 1

Schematic showing a 3D field (latitude, longitude, height) evolving over time, with multiple ensemble members.
Source

Figure 2

Chart showing the growth of climate data volumes over time, with a steep increase in the last decades.
The volume of worldwide climate data is expanding rapidly 1

Cloud-Native Formats

Figure 1

Difference between traditional files and cloud-native objects, with metadata and chunks.
Difference between traditional files and cloud-native objects

Figure 2

Cloud-Optimized Geospatial Formats.
Source

Zarr Data Model and Chunked Storage

Figure 1

Zarr organisation, showing groups and arrays with chunked storage.
Source

Figure 2

Zarr V2 metadata structure, showing .zgroup, .zarray, and .zattrs files.
Source

Figure 3

Zarr V3 metadata structure, showing zarr.json files
Source

Figure 4

Effect of chunking on data access.
How chunking affects data access.

Python for Zarr

Figure 1

Xarray logo.
Xarray

Figure 2

Zarr dataset opened in Xarray.
Zarr in Xarray

Choosing Chunks at Scale

Figure 1

Diagram showing how different chunking strategies affect performance for different workloads.
Source

Figure 2

A sharded Zarr showing how chunk data is grouped into shard files.
A sharded Zarr showing how chunk data is grouped into shard files. Source: https://element84.com/software-engineering/is-zarr-the-new-cog/

Parallel Processing with Zarr

Figure 1

Diagram showing parallel vs serial processing.
Source

Figure 2

Dask setup showing a local cluster with one worker and four threads.
Dask Setup

Figure 3

Dask dashboard showing task progress and worker status.
Dask dashboard graph view

Figure 4

Dask dashboard showing task progress and worker status.
Dask dashboard task view

Figure 5

Dask task graph showing the dependencies of tasks for performing calculations in a variable.
Dask task graph

Reading Real-World Zarr Datasets in Python

Figure 1

Temperature at 2 meters above the surface from ECMWF AIFS SINGLE dataset.

Figure 2

Sofar Spotter drifters deployed by Brazilian Navy and INPE, in partnership with Sofar Ocean
Sofar Spotter drifters deployed by Brazilian Navy and INPE, in partnership with Sofar Ocean

Figure 3

Array of spotter buoys
Array of spotter buoys

Figure 4

incomplete Array representation
incomplete Array representation

Figure 5

Ragged array structure
Ragged array structure

Figure 6

Trajectory of SPOT-0164
Trajectory of SPOT-0164

Object Storage and Cloud Data Organization

Figure 1

File vs Object Storage.
Source

Figure 2

Cloud Object Storage Architecture.
Cloud Object Storage Architecture

Figure 3

Traditional workflows vs New workflows.
Traditional workflows vs New workflows

Converting Traditional Formats to Zarr

Figure 1

NetCDF needs to be converted to Zarr for cloud-native workflows.

Figure 2

Pipeline diagram from NetCDF ingest to chunking, Zarr writing, validation, and cloud publication.
NetCDF-to-Zarr conversion should include chunk design, validation, and publishing checkpoints.

Case Studies

Versioning Data with Icechunk

Figure 1

Zarr without Icechunk.
Source

Figure 2

Happy, sad and mix sharks!

Figure 3

Git Workflow, which is similar to Icechunk Workflow.
Source

Virtual Zarr with Virtualizarr

Figure 1

An illustration NetCDF files trying to looking like a Zarr Store.
Source

Organizing Cloud Zarr Data with STAC

Figure 1

STAC organisation diagram showing Catalog, Collection, Item, and Asset relationships.
Source

Visualizing Multiscale Zarr and GeoZarr

Figure 1

Diagram showing a multiscale pyramid with multiple levels of downsampled data, each level with its own group in the Zarr hierarchy.
Source

Figure 2

Diagram showing the workflow for visualising a multiscale Zarr dataset in the browser using zarr-cesium.
Source

Figure 3

Pipeline diagram showing the conversion of a NetCDF dataset to a multiscale Zarr pyramid using Topozarr.

Architecture and Best Practices