by Scot Breitenfeld, Director of Engineering, The HDF Group
Training and deploying large models requires transferring significant volumes of low-precision tensors, including weights, gradients, activations, and KV caches. bfloat16 maintains float32’s exponent range, ensuring numerical stability with reduced storage requirements. FP8 E4M3 and E5M2 are standard for H100-class hardware, while FP6 and FP4 enable the extreme quantization supported by Blackwell hardware.
HDF5 stores all five as predefined types, making it suitable for checkpointing model weights or archiving quantized inference artifacts. However,H5Tconvert(), which transfers data between these types and, for example, float or double, uses a general-purpose loop that processes one element at a time to accommodate various exponent layouts, byte orders, and exception callbacks. While this approach is robust, it results in slow conversions; for example, converting 50 million elements takes 2.5 seconds. In training loops that checkpoint frequently, this delay becomes significant.
bfloat16 consists of the top 16 bits of a float32, so conversion should require only a cast and a shift, rather than a general-purpose loop. We implemented this optimized conversion, registered it with H5Tregister(), HDF5’s public API for such overrides, and observed a 186x speed improvement in benchmarks.
Rounding: what “shift right 16 bits” leaves out
The straightforward method to convert to bfloat16 is truncation: shift right and discard the lower 16 bits. While this is arithmetically correct, it does not match the rounding method used elsewhere in HDF5. Truncation always rounds toward zero, introducing a systematic downward bias in approximately half of all conversions, rather than random noise that averages out.
The preferred approach is to add half a ULP to the magnitude before truncating, enabling rounding rather than flooring. Ties are resolved by increasing the magnitude, following the library’s ’round-half-up’ convention for mantissa rounding. Consistency with HDF5’s general loop, which applies this method to all reduced-precision types (FP8, FP6, FP4, float16), is more important than the specific rounding method. If the fast path used a different rounding approach, checkpoint outputs could vary based on incidental factors, such as the presence of an exception callback. This is primarily a reproducibility concern, which is why this convention was chosen over round-to-nearest-even, the method used by bfloat16 hardware instructions.
Three additional areas required similar attention:
- NaN and Infinity share a boundary. If a float32 NaN has its payload bits entirely in the lower 16 bits, truncation results in a zero mantissa with an all-ones exponent, which represents Infinity rather than NaN. Since gradients may intentionally produce NaN as a signal, this case required explicit handling instead of relying on the default shift behavior.
- The empty case also required attention. In the widening direction, the code computes a backward pointer as
buf + (nelmts - 1). Ifnelmts == 0is checked after this computation, it can cause undefined behavior due to underflow. Empty tensor slices or skipped layers can trigger this scenario, andH5Tconvert()‘s public API does not automatically guard againstnelmts == 0. - Alignment and layout also matter. The general conversion framework uses a scratch buffer when data is not naturally aligned or contiguous. A raw pointer cast does not handle this automatically, so the optimized path checks for zero stride and natural alignment before proceeding. For other cases, such as strided compound-type members or unaligned buffers, it falls back to the general loop.
These issues are not apparent from simply reviewing the shift operation. They are identified through bit-for-bit comparisons with the general loop, using a million random values, deliberate tie cases, and edge-case inputs. After addressing all four areas, the float to bfloat16 conversion matched the general loop exactly, with zero mismatches in a million cases.
The double-to-bfloat16 conversion did not match as precisely, with one mismatch per million cases. This discrepancy warranted further investigation.
Chasing one mismatch in a million
Converting double to bfloat16 involves two rounding steps: first from double to float32, then from float32 to bfloat16. Double rounding can produce results that differ from a single rounding step, even if each step is correct. The key question was whether this discrepancy was random noise or a consistent bias.
Testing with fifty million doubles, randomly distributed across the full 52-bit mantissa (using a source with sufficient entropy), resulted in 388 mismatches between the two conversion methods. In every case, the fast path rounded higher than the general loop, with no mismatches in the opposite direction. A second independent run with fifty million samples produced 366 mismatches, again all in the same direction.
This outcome ruled out random noise, indicating a consistent but rare bias—approximately 1 in 129,000 conversions, compared to a 1-in-65,536 rate for the round-half-up tie bias. Aggregate analysis showed that, for 20 million real, non-adversarial values, the two rounding methods produced statistically indistinguishable results. The rare events, each about 1 ULP, contributed only 6×10⁻⁶% to the dataset mean, well below the quantization noise floor for bfloat16. While the bias is real and measurable, it is negligible in practice. Eliminating it would require abandoning the hardware double-to-float32 cast and manually implementing the general loop’s bit-level mantissa extraction, which would also forfeit the auto-vectorization benefits of the optimized path.
What the careful version costs
After addressing rounding, NaN, the empty case, alignment, and the double-rounding issue, the benefits of registering the converter were measured on an AMD Ryzen 9 9950X3D with 50 million elements, compared to the unmodified general loop:
| Conversion | General loop | Registered converter | Speedup |
|---|---|---|---|
| double to bfloat16 | 2.55 s | 0.014 s | 186x |
| float to bfloat16 | 2.40 s | 0.015 s | 159x |
| bfloat16 to double | 1.87 s | 0.019 s | 97x |
| bfloat16 to float | 1.75 s | 0.011 s | 167x |
These performance gains do not rely on specialized instructions. Both GCC 15 and Clang 21 can auto-vectorize a standard C loop over a contiguous buffer without manual intrinsics, as the conversion becomes memory-bandwidth bound once exception-callback checks and generic bit-field operations are removed. Registering the converter requires only two function calls, after which all subsequent H5Tconvert() invocations, including those within H5Dwrite() and other pipeline components, will use the optimized path automatically:
H5Tregister(H5T_PERS_HARD, "flt_bf16", H5T_NATIVE_FLOAT, H5T_FLOAT_BFLOAT16LE, conv_float_bfloat16);
H5Tregister(H5T_PERS_HARD, "bf16_flt", H5T_FLOAT_BFLOAT16LE, H5T_NATIVE_FLOAT, conv_bfloat16_float);
The conv_float_bfloat16 function must handle more than just the per-element loop. H5Tregister() first calls it with cdata->command set to H5T_CONV_INIT to determine if the conversion is supported, and with H5T_CONV_FREE during teardown. The actual element loop runs only when cdata->command is H5T_CONV_CONV. Omitting this logic will prevent the converter from functioning as intended.
The same approach, for FP8, FP6, and FP4
FP8 E4M3 and E5M2 are used for H100 training and inference, while FP6 and FP4 are Blackwell’s quantization formats. The same registration method applies, though the performance benefits differ. Widening to float uses a 256-entry lookup table, achieving 130x to 165x speedup over the general loop. Narrowing is more complex due to exponent rebias and clamping, which introduces branching and limits vectorization, resulting in a 37x to 57x improvement. These gains are significant, though still minor compared to the overall transfer or inference costs.
The GPU result that runs backward from intuition
When a tensor is generated on Blackwell hardware, cuda_fp8.h, cuda_fp6.h, and cuda_fp4.h perform native device-side conversion before data is transferred to the host, so no conversion is needed at the time of writing with H5Dwrite(). The reverse scenario is less intuitive: float32 data originating on the host must be converted to FP8 before it is usable by the GPU. There are two options: convert on the CPU and transfer a smaller data volume over PCIe, or transfer the full float32 data and convert on the GPU. Although transferring less data seems advantageous, measurements on an RTX 5060 Ti over PCIe 5.0 x8 with 50 million elements show otherwise:
| Path | Steps | Total |
|---|---|---|
Transfer float32, then convert on GPU |
6.9 ms + 0.6 ms | 7.6 ms |
| Convert on CPU first, then transfer FP8 | 42.9 ms + 1.7 ms | 44.7 ms |
Converting on the CPU is 5.9x slower, even though it transfers only a quarter of the data, because GPU-side conversion is nearly instantaneous (0.6 ms compared to 6.9 ms for the transfer). In comparison, the CPU’s branch-heavy narrowing code takes 43 ms. This overhead outweighs the benefits of reduced transfer size. When the quantized result must be returned to the host, using two CUDA streams across eight chunks allows the transfer and conversion to overlap, resulting in a full round trip of 7.92 ms, which is close to the 6.94 ms PCIe transfer limit. In GPU-resident pipelines, the PCIe link is the primary bottleneck, and device-side conversion is effectively free.
The seam is the point.
Implementing these optimizations does not require waiting for a new release or forking the library. H5Tregister() is a public API designed for situations where the caller has more knowledge about their data’s structure than a general-purpose loop can assume and is prepared to manage the associated numerical considerations. bfloat16 is an ideal candidate, as its relationship to float32 is straightforward, and any subtle divergences—such as rounding, NaN handling, empty inputs, or alignment—can be identified through bit-for-bit comparisons before reaching production checkpoints.
If your pipeline writes low-precision tensors to HDF5 at training or inference scale, this approach is valuable and should be tested as rigorously as any other process that affects every value in the file.