The HDF Group is happy to announce the release of HSDS 1.0.0, h5pyd 1.0.0, and hdf5-json 2.0.0.
The three projects now share a single implementation of the HDF5 object model instead of maintaining three copies of it. Additionally, this release brings major architectural improvements, enhanced APIs, streamlined package dependencies, and a more robust foundation for future development.
HSDS enables users to access, manage, and analyze HDF5 data through a REST-based service designed for cloud and object-storage environments. With version 1.0.0, the HSDS platform takes an important step forward in maintainability, interoperability, and operational flexibility.
One Shared Foundation
One of the most significant changes in HSDS 1.0.0 is the integration of the hdf5-json package as a shared foundation across the HSDS ecosystem.
Historically, HSDS and h5pyd maintained their own separate implementations of HDF5 utility functions (object identifiers, datatypes, filters, links, etc.). With this release, that duplicated functionality has been consolidated into the standalone hdf5-json library.
HSDS and related projects such as h5pyd can now share a consistent, non-duplicated implementation, while simplifying long-term maintenance.
This architectural change also establishes hdf5-json as a reusable component for applications that need structured JSON representations of HDF5 metadata and content.
Creating Objects in Fewer Round Trips
With older versions of HSDS, building a domain (its equivalent to an HDF5 file) required one request per object, more requests to attach attributes, and more requests again to write initial values. Typical object hierarchies could take hundreds of round trips.
HSDS 1.0.0 now accepts a list of objects in a single POST groups or datasets request, and each item can carry with it a client-chosen identifier, its own creation properties, and, for small datasets, initial values. These items can even carry client-specified timestamps to allow for migration of data while preserving original creation times.
Behind this new API, the server creates the objects concurrently using multiple workers to prevent slowdown from serialization.
Breaking Changes
HSDS 1.0.0 has raised its minimum Python version to 3.11. Both HSDS and h5pyd now require hdf5-json 2.0.0 or newer, and all require NumPy 2.0 or newer. These updates enable the project to leverage modern language and library features while simplifying future maintenance.
For project-specific upgrade details, see the individual release notes.
Additional Improvements
- Support for region references
- Improved API documentation through OpenAPI specifications
- POSIX I/O delay simulation for cloud-storage testing scenarios
- Numerous bug fixes, typo corrections, and code cleanup activities
Looking Ahead
HSDS 1.0.0 represents a stronger architectural foundation for future releases by aligning HSDS, h5pyd, and hdf5-json around a common set of shared capabilities and release practices.
The HDF Group would like to thank everyone who contributed code, reviews, testing, documentation, and feedback throughout the development process. The collaborative effort behind this release has helped strengthen the entire cloud-native HDF5 ecosystem.
We look forward to continuing to expand HSDS capabilities and supporting the growing community of users building scalable scientific and engineering applications in cloud environments.
Learn More
- HSDS
- h5pyd
- h5json
- Explore HSDS documentation
- HDF Forum