HDF5 beyond the file: findings from our safety, security, and privacy audit

Excerpt: A new ecosystem-level audit identifies recurring failure classes, clarifies where authority and responsibility reside, and proposes a phased mitigation roadmap for HDF5 and its wider software and data ecosystem. HDF5 is foundational infrastructure for science and engineering. It stores experimental observations, simulation results, images, models, sensor data, and long-lived scientific records. That reach makes […]

HDF5 was the mechanism, not the vulnerability

In our July 31 post, “When an HDF5 file points outside itself,” we described the use of HDF5 external raw storage in the Hugging Face incident. Some later accounts have labeled this as an “HDF5 vulnerability.” A recent Black Hat presentation, around 27:13, refers to it as “a vulnerability in HDF5 dataset parsing” and “the

Storing bfloat16 and FP8 tensors in HDF5 without a conversion tax

by Scot Breitenfeld, Director of Engineering, The HDF Group Training and deploying large models requires transferring significant volumes of low-precision tensors, including weights, gradients, activations, and KV caches. bfloat16 maintains float32’s exponent range, ensuring numerical stability with reduced storage requirements. FP8 E4M3 and E5M2 are standard for H100-class hardware, while FP6 and FP4 enable the

When an HDF5 file points outside itself

by Gerd Heber, Executive Director, The HDF Group When something goes wrong, someone is going to tell the story. You’ll be better off if it’s you. Otherwise, you create an opportunity for rumors, hearsay, and false information to spread. (Jason Fried & David Heinemeier Hansson, Rework) Reports on the July 2026 OpenAI agent intrusion have

Release of HDF5 2.2.0 and Two August Events (Newsletter #210)

HDF5 2.2.0 Now Available The HDF Group is pleased to announce the release of HDF5 2.2.0, delivering improved cloud performance, stronger security capabilities, enhanced build and deployment workflows, and numerous reliability and usability improvements across the HDF5 ecosystem. This release focuses on making HDF5 easier to deploy, more secure in production environments, and more efficient

Standardizing Tabular Data: Introducing the HDF5 Extension Proposal HEP001

In this episode of “Call the Doctor,” Aleksandar Jelenak explores the HDF5 Extension Proposal HEP001. This proposal aims to modernize how we handle tabular data in HDF5 by introducing a columnar storage specification, moving beyond legacy row-based approaches to enable better interoperability, performance, and metadata flexibility. Relevant Links HDF Alliance GitHub Repository HEP001 Proposal Document

String-Based Filter Configuration with TOML – Scot Breitenfeld – Call the Doctor (5/12/26)

In this session of “Call the Doctor,” The HDF Group’s Scot Breitenfeld discusses a major modernization effort for HDF5: moving from opaque integer-based filter parameters to human-readable TOML strings. This update, targeted for the HDF5 2.2 release, simplifies the developer experience for plugin authors and users of high-level languages like Python, Java, and Fortran. Configuring

Solving S3 Pricing Traps and h5diff Testing with Aleksandar Jelenak

In this “Call the Doctor” session, Aleksandar Jelenak (Senior Informatics Architect) and a user engage in a live troubleshooting discussion. They dive into new h5diff features for cleaner regression testing and share critical benchmarks on why standard S3 mount points might be costing you more than you think. Relevant Links GitHub Issue #6364: Exclude attributes

Agentic HDF5: Telemetry and Proactive Performance Advice – Matthew Larson – Call the Doctor (4/14/26)

In this session of “Call the Doctor,” Matt Larson provides a technical update on Agentic HDF5, a project designed to assist domain scientists in leveraging the power of HDF5 without needing to become experts in the library’s internal mechanics. The discussion focuses on a new telemetry system that proactively identifies performance bottlenecks and suggests optimizations

Scroll to Top