Key Points

Data Formats, Metadata, and Vocabulary

  • “Oceanography and meteorology rely on self‑describing array formats (NetCDF, HDF5, Zarr) and WMO exchange formats (GRIB, BUFR) for gridded and observational data.”
  • “Metadata is ‘data about data’ and lives both inside files (e.g. CF attributes) and in separate discovery records based on standards such as ISO 19115, INSPIRE and MEDIN.”
  • “Controlled vocabularies, delivered through services like the NERC Vocabulary Server, provide standardised terms and identifiers that make metadata consistent and machine‑actionable.”
  • “CF metadata conventions define how to describe Earth science variables, grids and coordinates in self‑describing formats and underpin many ocean and climate data tools.”
  • “Discovery profiles such as the MEDIN Discovery Metadata Standard align national practice with international standards (ISO 19115, INSPIRE), supporting data discovery across portals and organisations.”
  • “STAC catalogs (Catalogs, Collections, Items) provide a modern, JSON‑based way to organise and expose geospatial assets, including NetCDF and Zarr stores, in cloud‑native workflows.”
  • “Cloud‑native formats like Zarr become interoperable when combined with established metadata and vocabulary standards (CF, Unidata CDM, ISO profiles, NVS) and can be complemented by STAC for discovery.”

Challenges of N-Dimensional Data

  • “NetCDF provides a self‑describing, array‑oriented data model for n‑dimensional scientific datasets and is widely used in meteorology and oceanography.”
  • “GRIB is a compact, message‑based WMO format designed for operational transmission of gridded meteorological fields, using tables and codes to represent metadata.”
  • “Data volumes have grown dramatically with higher resolution, more frequent output, longer archives, and ensembles over the last three decades.”
  • “In many current workflows, large NetCDF and GRIB files are still downloaded in full to local or shared servers, even when only subsets are needed.”
  • “Server‑side subsetting (e.g. OPeNDAP) and centralised archives can reduce duplication, but long‑term storage on traditional file systems becomes costly at scale compared to object storage.”
  • “Understanding how existing formats organise n‑dimensional data, and where they strain under growth in size and shared use, prepares us to learn about chunking and cloud‑native solutions in later lessons.”

Cloud-Native Formats

  • “Cloud-native formats are designed for efficient access over object storage and web protocols, not just for local filesystems.”
  • “A cloud-native layout usually combines lightweight metadata with addressable chunks so that tools can read only the data they need.”
  • “Simply storing NetCDF or HDF5 files in the cloud does not automatically make them cloud-native.”
  • “In Earth system science, common approaches include Zarr, cloud-optimized NetCDF/HDF5 layouts, Kerchunk, VirtualiZarr, GeoParquet, and FlatGeobuf.”
  • “Cloud-native approaches can reduce data movement, improve sharing, and support scalable analysis of large environmental datasets.”

Zarr Data Model and Chunked Storage

  • “Zarr organises data into groups and arrays stored as addressable chunks, which supports efficient partial reads and writes.”
  • “Metadata in Zarr describes array structure, chunking, and attributes, and can be extended through shared conventions for geoscience workflows.”
  • “Chunked storage is central to Zarr performance because it lets tools read only the pieces needed for a given query or computation.”
  • “Inspecting a Zarr store with Python and xarray helps distinguish cheap metadata exploration from actual data loading.”
  • “Understanding data model, metadata, and chunk layout is the foundation for later lessons on rechunking, parallel processing, and cloud-native analysis.”

Python for Zarr

  • “The zarr Python package provides low-level access to Zarr stores, including groups, arrays, and attributes.”
  • “xarray is the main high-level tool for working with environmental Zarr datasets as labelled N-dimensional data.”
  • “Opening Zarr with xarray (xr.open_zarr) gives you variables, dimensions, and coordinates in a familiar Dataset structure.”
  • “Data values are loaded only when they are selected or used in a computation.”
  • “Basic xarray operations such as selection, reduction, and plotting work the same on Zarr data as on NetCDF data.”

Choosing Chunks at Scale

  • “Chunks are N-dimensional blocks that control how data is stored and accessed in Zarr.”
  • “Chunk shapes should be chosen based on dominant workloads (time series, spatial averages, ensembles) and practical constraints like chunk size in bytes.”
  • “Rechunking can reorganise a dataset to better match performance needs, at the cost of an initial rewrite step.”
  • “Tools like zarr and xarray make it possible to inspect chunk layouts, design new schemes, and save rechunked Zarr stores for analysis at scale.”
  • “Sharding is a technique that groups multiple chunks into larger storage objects, reducing overhead while keeping the benefits of chunked access.”

Parallel Processing with Zarr

  • “Dask is the main parallel tool used here.”
  • “Zarr and Dask work well together because Zarr stores data in independent chunks.”
  • “Xarray can open Zarr data lazily and hand chunked work to Dask.”
  • “Other Python parallelism tools exist, but Dask is the most natural fit for chunked environmental data.”

Reading Real-World Zarr Datasets in Python

  • “Many open Zarr datasets are available for oceanography, climate, and meteorology, including ERA5 ARCO, Sofar Spotter drifters, ECMWF IFS ensemble forecasts, CMIP6, and marine products.”
  • “Python tools like xarray, zarr, and fsspec make it straightforward to open and explore Zarr datasets hosted on cloud object storage.”
  • “Real-world Zarr datasets illustrate regular grids, ragged arrays, ensemble dimensions, and more, providing rich practice for accessing, analysing, and understanding chunked data.”
  • “Working hands-on with open Zarr datasets helps learners build intuition about data structures, performance, and best practices for cloud-native scientific workflows.”

Object Storage and Cloud Data Organization

  • “Object storage stores data as objects with keys and metadata in buckets, accessed via HTTP/S3-style APIs rather than local filesystems.”
  • “Cloud object stores (S3, GCS, Azure Blob, S3-compatible services) offer durable, scalable, and secure storage well suited to large scientific datasets.”
  • “Parallel and concurrent access are natural in object storage, making it a good fit for chunked formats and distributed processing frameworks.”
  • “Self-hosted object storage solutions like MinIO provide S3-compatible APIs and can be deployed with Docker on your own servers.”
  • “Thoughtful bucket and key organisation is essential for efficient data discovery and workflow design in the cloud.”

Converting Traditional Formats to Zarr

  • “Converting NetCDF to Zarr enables cloud-native, chunked, and parallel-friendly access to large scientific datasets.”
  • “Effective conversion requires understanding input data, choosing chunk sizes based on workloads, and using xarray’s to_zarr with appropriate encoding and compression.”
  • “Dask can parallelise the conversion process, making it feasible to handle large collections of NetCDF files.”
  • “Uploading Zarr stores to object storage allows distributed teams and tools to access the same datasets efficiently.”
  • “Checking basic statistics (e.g. mean values) in NetCDF and Zarr versions helps verify that conversions preserve scientific content.”

Case Studies

  • “Real-world case studies show that cloud-native architectures differ by mission, user needs, scale, and operational constraints.”
  • “Successful teams make explicit trade-offs among performance, cost, governance, reproducibility, and maintainability.”
  • “Common building blocks include Zarr (physical or virtual), STAC for discovery, and versioning approaches for controlled updates.”
  • “Architectural decisions should be guided by concrete workloads and user access patterns, not only by tool popularity.”
  • “Cross-team sharing of implementation lessons helps avoid repeated mistakes and speeds up adoption of robust workflows.”

Versioning Data with Icechunk

  • “Plain Zarr lacks built-in transactions, history, and recovery.”
  • “Icechunk adds repositories, transactions, snapshots, branches, and tags.”
  • “Transactions make multi-variable updates atomic.”
  • “Branches and tags make reproducible analysis possible.”
  • “Version history makes it much easier to recover from mistakes.”

Virtual Zarr with Virtualizarr

  • “Virtualizarr creates virtual Zarr datasets without copying the data.”
  • “It is useful when full conversion is impractical or unnecessary.”
  • “You can create a virtual dataset directly from NetCDF files already on the server.”
  • “xarray can open the virtual dataset and analyse it like a familiar dataset.”
  • “Virtualisation is a useful bridge between NetCDF archives and Zarr-style workflows.”

Organizing Cloud Zarr Data with STAC

  • “STAC standardises how we describe and organise geospatial assets, including Zarr data cubes, making them easier to discover and integrate into cloud-native workflows.”
  • “Catalogs, Collections, and Items form a three-layer hierarchy: Catalog as entry point, Collection as grouped datasets, Item as individual spatiotemporal assets with linked data files.”
  • “PySTAC provides a Python API to create and manage STAC Catalogs, Collections, and Items, and to save them as static JSON catalogs or feed them into STAC APIs.”
  • “Zarr stores in object storage can be referenced as STAC Item assets, connecting cloud-native array data to the broader STAC ecosystem.”
  • “STAC metadata can be stored either as static JSON (simple publishing) or in databases like pgSTAC/PgPystac (scalable, queryable STAC APIs), and tools like STAC Browser can visualise both.”

Visualizing Multiscale Zarr and GeoZarr

  • “Chunking and multiscale pyramids are central to interactive visualisation of geospatial Zarr datasets, allowing efficient tile-based access at different zoom levels.”
  • “GeoZarr defines modular conventions (proj, spatial, multiscales) for encoding geospatial metadata and multiscale layouts on top of Zarr.”
  • “Topozarr can be used to build multiscale Zarr stores from existing datasets, preparing them for efficient visualisation.”
  • “Browser clients such as OpenLayers (GeoZarr source), zarrita, and zarr-cesium can render Zarr data directly from object storage using WebGL and GPU acceleration, without a dedicated tiling server.”
  • “By combining GeoZarr conventions, multiscale Zarr, and simple HTML/JS clients, you can build fully cloud-native visualisation workflows for your geospatial datasets.”

Architecture and Best Practices

  • “Cloud-native workflows are strongest when formats, metadata, storage, and operations are designed together as one architecture.”
  • “There is no single best stack: choices between physical Zarr, virtual Zarr, STAC, Icechunk, and GeoZarr depend on workload, governance, and resources.”
  • “Practical trade-offs among simplicity, performance, cost, and maintainability should be documented and revisited over time.”
  • “Chunking, metadata quality, and discoverability are foundational decisions that strongly affect usability and long-term sustainability.”
  • “A short, explicit best-practice checklist helps teams apply course concepts consistently in real projects.”