Latest Newsletter - #137 Published on November 15, 2013
November 2013: Book Release of "Python and HDF5"
Python users, be sure to check out "Python and HDF5: Unlocking Scientific Data" by Andrew Collette, published by O'Reilly. Andrew has implemented h5py, a powerful python interface to HDF5. This book is an excellent guide to learning h5py, with loads of exercises and real-world examples.
You should also have a look at this interview of Andrew Collette, given at the launching of the book in November 2013.
June 28, 2013: Samplify Releases APAX HDF Storage Library
June 28, 2013 -- Samplify Systems, Inc., announced the availability of its APAX HDF Storage Library at the recent International Supercomputing Conference in Leipzig, Germany. Samplify provides software and hardware solutions for solving memory, I/O, and storage bottlenecks in HPC, Big Data, cloud computing, consumer electronics and mobile devices. Its APAX technology is a universal numerical data encoder that operates on any integer or floating point data type and can achieve typical encoding rates of 3:1 to 8:1 without affecting the results of computing applications.
According to the APAX HDF Product Brief, "Using HDF’s plug-in capability, APAX HDF inserts Samplify’s APAX encoder into the write pipeline, and the APAX decoder in the read pipeline, to automatically save and access dataset chunks in APAX compressed format. Any application which already uses HDF as its storage format can take advantage of Samplify’s APAX HDF storage library WITH NO CODING REQUIRED! Unlike other plug-ins, Samplify’s APAX HDF requires no modification to solver applications."
See the APAX HDF web page on Samplify's website for more information.
A team of researchers from Lawrence Berkeley National Laboratory (Berkeley Lab) and Cray Inc. performed a trillion-particle simulation on the National Energy Research Scientific Computing Center’s (NERSC’s) Cray XE6 “Hopper” supercomputer. The experiment pushed the machine's capabilities by using more than 120,000 processors and generating approximately 350 terabytes of data. The team recently won the Best Paper award at the 2013 Cray User Group conference for their description of the simulation and its findings.
"This is the largest I/O job ever undertaken by a NERSC application. It is quite a feat when you consider that even the smallest bottleneck in a production I/O stack can degrade performance at scale," says Prabhat, a researcher in Berkeley Lab's Scientific Visualization Group and co-author of the paper. He further explained how progress made by the ExaHDF5 team over the course of the project made it possible "to demonstrate that HDF5 can scale to petascale platforms like Hopper and achieve near peak I/O rates."
ExaHDF5 is a Department of Energy funded collaboration, led by Prabhat, to develop high performance I/O and analysis strategies for future exascale computers. A primary goal of the project has been to expand the capabilities of HDF5 for petascale and future exascale platforms.
"The outcome of this work was truly ideal,” says Prabhat. “We ran a state-of-the-art simulation code at scale, which wasn’t possible before, using the best computing resources and expertise, and this effort produced a first-time science result that no one had ever seen before. Computer science researchers always hope for such an outcome, but rarely do things come together in this fashion."
April 2, 2013: "The Earth Observer" Article Covers HDF and HDF-EOS Earth Science Data Formats, HDF-EOS Website
The March-April issue of NASA's "The Earth Observer" includes an interesting article entitled "Working with NASA’s HDF and HDF-EOS Earth Science Data Formats", written by Jennifer Brennan of NASA's Goddard Space Flight Center and H. Joe Lee, MuQun Yang, Mike Folk, and Elena Pourmal of The HDF Group.
The article provides a brief overview of how the HDF and HDF-EOS formats are used in the NASA Earth Observing System (EOS) followed by an excellent description of examples available on the HDF-EOS Tools and Information Center website.
February 7, 2013: The HDF Group Board Member Recognized as 2013 "Person to Watch" in High-Performance Computing
the #1 news and information portal covering the fastest computers in the world and the
people who run them announced on January 25 that it has published its HPCwire People to Watch 2013 list, and we are
proud to congratulate The HDF Group board member Dr. William D. (Bill) Gropp for his selection to the list.
As stated in the announcement,
The annual list is comprised of an elite group of the best and brightest minds in HPC whose research, dedication and hard work will be making a difference in the HPC community and in the world with their contributions.
At the University of Illinois at Urbana-Champaign Dr. Gropp is the Paul and Cynthia Saylor Professor of Computer Science, Director of the Parallel Computing Institute, and Deputy Director for Research at the Institute for Advanced Computing Applications and Technologies. He joined the board of The HDF Group in 2011.
Dr. Gropp is also General Chair for Supercomputing 2013, the International Conference for High Performance Computing, Networking, Storage and Analysis.
About receiving this recognition Dr. Gropp remarked,
It's a great honor to be part of this year's
group of people to watch in HPC. I'll be watching to see how our predictions turn out!
November 7, 2012: ExxonMobil Upstream Research's Standards DevKit Uses HDF5DotNet
"HOUSTON, TX -- (Marketwire) -- 11/07/12 -- The Energistics Consortium announced today that ExxonMobil Upstream Research Company has provided an enhanced version of the Standards DevKit that adds support for RESQML 1.1, including managing the HDF5 file format. This enhanced version of the Standards DevKit will support RESQML, the Energistics reservoir data exchange standards as well as WITSML."
The HDF Group worked with the Energistics Consortium in 2011 to provide additional functionality in the .NET C++/CLI wrapper of the HDF5 library and supported the wrapper on Windows XP and Windows 7 for a limited period.
"RESQML is an XML-based data exchange standard that helps to address the data-incompatibility and data-integrity challenges faced by petro-technical professionals when using the multiple software technologies required along the entire subsurface workflow, for analysis, interpretation, modeling, and simulation." (Energistics Guide)
For more information about the HDF5DotNet wrapper, please visit hdf5.net.
August 2, 2012: Major Upgrade of HDF5 OPenDAP handler
A major upgrade of the HDF5 OPeNDAP handler was released at opendap.org. The upgraded version greatly improves the accessibility of NASA Earth
Sciences HDF5 data via OPeNDAP. It includes support for more HDF5 data
and much better support for the CF conventions. The enhanced CF
support will greatly improve interoperability with those clients that
July 12, 2012: DOE Primes Pump for Exascale Supercomputers
Article published in HPC Wire about The HDF Group collaboration with Whamcloud and Cray on July 10, 2012.
The HDF Group is collaborating with Cray and Whamcloud on the FastForward
program. FastForward is a jointly funded project between the Department of
Energy (DOE) and National Nuclear Security Administration (NNSA) to
accelerate the research and development (R & D) of critical technologies
needed for extreme scale computing. Exascale computing is essentially a
grand challenge to provide the next level of computational power required
to help ensure the prosperity and security of the United States.
March 23, 2012: HDF 4.2.7-patch1
A patched version of the HDF 4.2.7 source code,
HDF 4.2.7-patch1, is
now available to correct a configure issue with compilers that contain a '-' in the name.
December 19, 2011: HDF5DotNet 1.8.8 now available
HDF5DotNet 1.8.8 is now available for download. This release supports HDF5-1.8.8. See the HDF5DotNet home page for detailed information regarding this release:
October 28, 2011: NASA NPP Spacecraft Launches on Earth Observing Mission
The NPOESS Preparatory Project (NPP) spacecraft lifted off at 5:48 a.m. EDT on Oct. 28, 2011, to begin its
earth observation mission. NPP is the first of several satellites, all of whose data will be stored in HDF5.
June 1, 2011: The HDF Group joins OGC
Announcement that The HDF Group has joined the Open
Geospatial Consortium (OGC) as an Associate Member.
December 5, 2010: HDF Group's growth, move prompts news coverage
Link to an article in the Champaign, Illinois, News-Gazette.
October 12, 2010: SMAP Data Products in HDF5
SMAP (Soil Moisture Active & Passive) data products will be delivered in the HDF5 Format. SMAP is one of four Tier-1 missions recommended by the U.S. National Research Council Committee on Earth Science and Applications from Space. It will provide global measurements of soil moisture and its freeze/thaw state. These measurements will be used to enhance understanding of processes that link the water, energy and carbon cycles and extend the capabilities of weather and climate prediction models. SMAP data will also be used to quantify net carbon flux in boreal landscapes and improve flood prediction and drought monitoring capabilities.
Sony Pictures Imageworks and Industrial Light and Magic (ILM) have collaborated to create ALEMBIC, an open source exchange format that aims to become the standard for exchanging animated computer graphics scenes between content creation software packages.
February 26, 2010: HDF5 1.8 Corruption Problem
A corruption problem was found in HDF5 1.8, which affected releases 1.8.0 through 1.8.4. The problem was fixed in HDF5 1.8.4 Patch 1.
- newsletter137.html - November 15, 2013 (Release of HDF5-1.8.12, h4mapwriter 1.0.6,
h5edit 1.2.0, "Python and HDF5")
- newsletter136.html - July 30, 2013 (Release of h4toh5 2.2.2)
- newsletter135.html - June 27, 2013 (Release of H4CF Conversion Toolkit 1.0 beta)
- newsletter134.html - May 15, 2013 (Release of HDF5-1.8.11 and h4mapwriter 1.0.5)
- newsletter133.html - March 8, 2013 (Release of h5edit 1.1.0, nagg 1.5.0-beta)
- newsletter132.html - February 15, 2013 (Release of HDF 4.2.9)
- newsletter131.html - January 28, 2013 (Release of HDF5-1.8.10-patch1)
- newsletter130.html - November 15, 2012 (Release of HDF-Java Products 2.9)
- newsletter129.html - November 13, 2012 (Release of HDF5-1.8.10)
- newsletter128.html - August 15, 2012 (Release of HDF 4.2.8)
- newsletter127.html - May 15, 2012 (Release of HDF5-1.8.9)
- newsletter126.html - February 15, 2012 (Release of HDF 4.2.7 and h4toh5 2.2.1)
- newsletter125.html - December 15, 2011 (Release of HDF-Java 2.8)
- newsletter124.html - November 15, 2011 (Release of HDF5-1.8.8 and h4mapwriter 1.0.2)
- newsletter123.html - September 2, 2011 (Release of h5check 2.0.1, h5edit 1.0.0, h5augjpss 1.0.0, SC11 BOF)
- newsletter122.html - July 8, 2011 (Release of h4toh5 2.2.0, h4mapwriter 1.0.0)
- newsletter121.html - June 21, 2011 (Release of HDF 4.2.6)
- newsletter120.html - May 13, 2011 (Release of HDF5-1.8.7)
- newsletter119.html - February 22, 2011 (Release of HDF-Java Products 2.7)
- newsletter118.html - February 18, 2011 (Release of HDF5-1.8.6)
- newsletter117.html - September 30, 2010 (Release of HDF-Java Products 2.7 (Beta))
- newsletter116.html - September 3, 2010 (Release of 1.8.5-patch1; NPOESS High Level APIs)
- newsletter115.html - July 28, 2010 (1.8.5 fix for files w/non-default address/length sizes)
- newsletter114.html - June 15, 2010 (Release of HDF5 1.8.5)
- newsletter113.html - April 22, 2010 (Release of h5check 2.0)
- newsletter112.html - April 16, 2010 (Release of BioHDF pipeline prototype)
- newsletter111.html - March 5, 2010 (Release of HDF Java Products 2.6.1)
- newsletter110.html - March 2, 2010 (Release of HDF 4.2.5, H4toH5 2.1.1, 5 1.8.4 Patch 1)
- newsletter109.html - November 13, 2009 (Release of HDF5 1.8.4 and HDF5 1.6.10)
- newsletter108.html - September 23, 2009 (Patches for HDF Java Products 2.5)
- newsletter107.html - May 15, 2009 (Release of HDF5 1.8.3 and 1.6.9)
- newsletter106.html - February 11, 2009 (Release of HDF Java Products 2.5)
- newsletter105.html - February 3, 2009 (Release of HDF 4.2r4, H4 to H5 Version 2.1)
All text Issues (Link to text issues on FTP Server.)