by Aleksandar Jelenak, The HDF Group
Virtual environments are very popular method for ordinary computer users to install software into an isolated space without assistance from their system administrators. The conda package manager is one of the most widely used tools for managing virtual environments. There is also a newer implementation with the same functionality, called mamba, but we will refer only to conda here for simplicity. Use mamba instead if you want by simply replacing conda with mamba.
We are going to install software only from the conda-forge package repository, which is a community GitHub organization that maintains a large number of software packages. We will explicitly state the conda-forge repository as the source in every command, but we recommend to do this in the conda/mamba configuration settings.
The conda-forge repository has HDF5 library as its hdf5 package and we certainly recommend installing it if you just need the library at the version and build configurations provided. But if you need a version or feature otherwise not available from a conda package repository, this blog is for you.
We will first create a virtual environment named build-example with the required packages for building HDF5 library:
conda create -n build-example -c conda-forge python=3.12 cmake ninja c-compiler openssl curl
Activate the newly created environment:
conda activate build-example
Install HDF5 Library
You should see somewhere in the terminal prompt (build-example) as an indication that the correct virtual environment is active.
We are not going to cover all the different ways to get the HDF5 library’s source code. For this example, we will download the latest HDF5 library code from its develop branch on GitHub as a ZIP archive and unzip it. This is how the folders look like:
└── hdf5-develop
├── HDF5Examples
├── bin
├── c++
├── config
├── doc
├── doxygen
├── fortran
├── hl
├── java
├── m4
├── release_docs
├── src
├── test
├── testpar
├── tools
└── utils
In the top folder, execute the following command to configure the build process:
cmake \
-S ./hdf5-develop \
-B ./build \
-G Ninja \
-DCMAKE_INSTALL_PREFIX=${CONDA_PREFIX} \
-DCMAKE_BUILD_TYPE=Release \
-DHDF5_ENABLE_SYMBOLS=OFF \
-DBUILD_SHARED_LIBS=ON \
-DHDF5_ENABLE_ALL_WARNINGS=ON \
-DHDF5_ENABLE_THREADSAFE=OFF \
-DHDF5_BUILD_HL_LIB=ON \
-DHDF5_ENABLE_PARALLEL=OFF \
-DHDF5_BUILD_CPP_LIB=OFF \
-DHDF5_BUILD_FORTRAN=OFF \
-DHDF5_BUILD_JAVA=OFF \
-DHDF5_ENABLE_ROS3_VFD=ON \
-DHDF5_ENABLE_Z_LIB_SUPPORT=ON \
-DZLIB_LIBRARY:FILEPATH=${CONDA_PREFIX}/lib/libz.1.dylib \
-DZLIB_INCLUDE_DIR:PATH=${CONDA_PREFIX}/include \
-DZLIB_USE_EXTERNAL=OFF \
-DHDF5_ENABLE_SZIP_SUPPORT=OFF
If the command finishes succesfully there should be a new folder named build with all the build artifacts. There are many build options for the HDF5 library and we are not going to discuss them here. The above command has my usual basic settings plus the Read-Only S3 (ROS3) virtual file driver. The use of the $CONDA_PREFIX environment variable makes the above command “aware” of the active virtual environment because it points to its system-like folders, listed below:
$CONDA_PREFIX ├── bin ├── conda-meta ├── doc ├── etc ├── include ├── lib ├── libexec ├── man ├── sbin ├── share └── ssl
We set the installation destinations to the bin and lib folders in $CONDA_PREFIX and link the zlib compression library installed in the virtual environment. Next, we build the library:
cmake --build ./build --config Release -j 4
And then run the tests to ensure the library is working as expected:
ctest --test-dir ./build -C Release -j 4
Depending on where the library source came from there could be some failed tests. For example, the development branch code may not always pass all of its tests. However, all tests should pass for an official release code. You can use the --stop-on-failure option to stop running tests on the first failure.
Final step, install the library and its command-line tools:
cmake --install ./build --config Release
You can check where the install artifacts went by running the commands below:
cd $CONDA_PREFIX ls bin/h5* ls lib/*hdf5* ls include/H5*
Deactivate and activate the virtual environment so the newly installed bin/h5* commands are picked up:
conda deactivate conda activate build-example
and then check the version of the h5dump command:
h5dump --version
Everything looks good so we can proceed with building h5py using this version of the HDF5 library.
Install h5py with Custom HDF5 Library
The virtual environment where the custom HDF5 library was installed must be active. Let’s check if the pip tool is available and its version:
python -m pip --version
This is a safer way to run pip because it ensures the virtual environment’s pip will be used. We are going to build h5py with the latest versions of NumPy and Cython so let’s add them to the virtual environment:
conda install numpy cython -c conda-forge
Let’s build and install h5py:
HDF5_DIR=$CONDA_PREFIX python -m pip install --no-binary=h5py --no-build-isolation h5py
The options to pip ensure building h5py from source code and using the Python packages available in the virtual environment. The latter avoids pip building h5py with different versions of the required Python packages. The HDF5_DIR environment variable points the h5py build process to the desired HDF5 library installation. Note this only applies during the build process. If you happen to have another HDF5 library in a system-wide location it may be found prior to this one and cause an error when using h5py. How to avoid this is beyond the scope here. After all, this is one of the reasons to use virtual environments.
The first and most important test is if h5py can be successfully imported. We are also going to check what is the HDF5 library version h5py was built with. It should match the version reported by the h5dump command above.
python -c "import h5py; print(h5py.version.hdf5_version)"
If the above command works and the reported library versions match—congratulations! You have successfully built and installed both HDF5 library and h5py in a conda virtual environment. Remove the build folder and don’t forget to deactivate this virtual environment when not needing it any more or just close the terminal session.