Enterprise Support Downloads

Virtual File Drivers Source Code

Download the S3 and HDFS HDF5 virtual file drivers, available only to Enterprise Support customers. This is provided as a patch, and is only necessary for those who wish to use the virtual file drivers. More information can be obtained in the README file, below.

hdf5-1.10.4-vfds.tar.gz 2.12 MB
    Contents of the Package
    hdf5-1.10.4-vfds Approximate Size Description
    COPYING .7 K License/copyright file
    hadoop-testing-bundle.tar.gz 2 MB bundle of instructions for downloading a Hadoop binary, running included scripts and/or to  setting environment variables to set up and run hadoop for testing the hdfs vfd

    This testing bundle is for internal testing or for users who do not already have and use Hadoop. Users who already have and use hadoop probably don’t need this bundle, and the README file in hdf5-1.10.4-vfds should provide whatever information they need to set up HDF5 –with-libhdfs=.

    README 2 K File with instructions for applying the patch to HDF5 1.10.4 source, building with –enable-ros3-vfd and/or –with-libhdfs= configure options, and testing to verify that it is working.
    vfds_hdf5_1_10_4.patch 1 MB Source code patch to add code for S3 and HDFS VFDs.
    Missing: Testfiles – There are 3 files for testing S3 with “make check”, which we need to include for users to put in their own S3 bucket. “Make check” will skip the tests with the files in an S# bucket if the HDF5_S3_URL environment variable is not defined.
    README

    vfds_hdf5-1.10.4.patch
    ========================

    This patch provides code for two read-only HDF5 vfd prototypes: S3 vfd and HDFS vfd. These virtual file drivers allow HDF5 to open, read, seek and close 1.) HDF5 files stored in AWS S3 buckets and 2.) HDFS files.

    Applying the patch:

    The vfds_hdf5-1.10.4.patch file should be applied to fresh HDF5 1.10.4 code as follows:
    1.) Put the patch file in the top level of the HDF5 source code or another accessible location.

    2.) cd to the top level of the HDF5 source code and run
    “patch -p 1 < <path to file>/vfds_hdf5-1.10.4.patch”.

    The operation should complete without errors or questions. Then run

    “sh ./autogen.sh”

    3.) Configure and build HDF5. The vfd provides two added configure options:

    –enable-ros3-vfd
    –with-libhdfs=<location of Hadoop install>

    Either, both or neither of these options may be specified as desired, although the second option requires a valid path to a Hadoop install directory containing lib/native/libhdfs*.

    Setup for vfd options:

    –enable-ros3-vfd

    No extra setup is required to configure and build HDF5 with this option, but testing HDF5 with the S3 vfd enabled using “make check” requires:

    1.) set the environment variable HDF5_ROS3_TEST_BUCKET_URL to a URL for an accessible AWS S3 bucket containing HDF5 files.
    2.) the proper credentials in an AWS location for access to that bucket. Check the Amazon AWS website for S3 bucket access information.

    –with-libhdfs

    If hadoop is installed, enter the path to the Hadoop install directory as described above. For users wishing to experiment with Hadoop who have not previously installed it, the hadoop-testing-bundle is included with this patch.The bundle includes a README file with details and scripts for installing and setting up hadoop for connections from HDF5. Installing Hadoop and providing the path should be sufficient to configure and build HDF5 with the option, provided java is enabled.

    Using the HDFS vfd requires the following in HADOOP_HOME/etc/hadoop: Environment variables JAVA_HOME and HADOOP_HOME are set correctly in hadoop-env.sh. The configurations in core-site.xml and hdfs-site.xml in hadoop-testing-bundle are present in the files with those named in HADOOP_HOME/etc/hadoop. The hadoop service processes are listening on the expected ports. See the README file in the hadoop-testing-bundle for details.

    Filter Plugins

    The decode only versions of these filter plugins are available on our Community HDF5 page

    hdf5plugin-1.10.5-centos7_64.tar.gz 674.4 KB
    hdf5plugin-1.10.5-centos7_64.tar.gz.md5 32 B
    hdf5plugin-1.10.5-win10_64-vs14.zip 488.4 KB
    hdf5plugin-1.10.5-win10_64-vs14.zip.md5 32 B
    hdf5plugin-1.10.5-win10_64-vs15.zip 511.5 KB
    hdf5plugin-1.10.5-win10_64-vs15.zip.md5 32 B
    hdf5plugin-1.10.5-win7_64-vs14.zip 499.5 KB
    hdf5plugin-1.10.5-win7_64-vs14.zip.md5 32 B