In our July 31 post, “When an HDF5 file points outside itself,” we described the use of HDF5 external raw storage in the Hugging Face incident.
Some later accounts have labeled this as an “HDF5 vulnerability.” A recent Black Hat presentation, around 27:13, refers to it as “a vulnerability in HDF5 dataset parsing” and “the HDF5 vulnerability.”
This description is inaccurate. The incident revealed a vulnerability in a service processing untrusted HDF5 files, not a flaw in HDF5 dataset parsing.
An intentional feature, not a parser failure
External raw data storage is a documented and intentional HDF5 feature. The HDF5 File Format Specification Version 1.0 defines a “Data Storage – External Data Files” message to identify files containing a dataset’s raw data. The H5Pset_external() API offers the corresponding library interface.
This behavior has existed since HDF5 1.0.0, released in November 1998.
The files involved in the incident were valid HDF5 files. The library read their metadata and accessed the specified external files as intended. There was no HDF5 memory-safety error, parser failure, undocumented behavior, or execution of code through HDF5.
The trusted service supplied the authority
The Hugging Face technical timeline describes what happened:
- An untrusted party controlled an HDF5 file.
- A trusted dataset-processing service opened that file.
- The file declared that its dataset bytes resided in local files such as /proc/self/environ.
- The service’s HDF5 library read those files using the service process’s filesystem permissions.
- The service returned the resulting bytes to the requester.
The HDF5 library does not have its own operating-system identity or privileges. It operates with the authority of its host application or service. If the process can read a file, HDF5 operations within that process can also read it.
This is a confused deputy problem. A trusted service with access to sensitive resources was misled by untrusted input to misuse that access. The dataset-processing service acted as the deputy, and HDF5 external storage was the mechanism used.
Why the distinction matters
Labeling this as an HDF5 vulnerability directs defenders to the wrong solution. Updating the HDF5 library would not address the service’s trust boundary or the authority given to its worker process.
Applications and services that process untrusted HDF5 files must decide which file-controlled capabilities they will permit. Relevant controls include:
- rejecting external raw storage unless it is explicitly required;
- inspecting external references without following them;
- restricting permitted paths to approved locations;
- running readers with minimal filesystem access;
- isolating file-processing workers from credentials and sensitive source files; and
- applying the same policy to external links, virtual dataset sources, and dynamically loaded plugins.
Clarifying this point does not diminish the seriousness of the security failure, nor does “working as designed” remove The HDF Group’s responsibility. A capability created for trusted scientific workflows in 1998 now operates in public services that accept arbitrary uploads. The security model must reflect this change.
Our HDF5 SHINES initiative addresses this gap with an HDF5 policy mechanism and an untrusted-file profile, making secondary resource access visible, controllable, and denied by default where appropriate.
Precision is important because effective mitigation depends on accurate diagnosis. HDF5 enabled the file reference, but the vulnerable service granted it authority it should not have had.
This material is based upon work supported by the U.S. National Science Foundation under Federal Award No. 2534078. Any opinions, findings, conclusions, or recommendations expressed are those of the author and do not necessarily reflect the views of the National Science Foundation.