Blog

The HDF Server allows producers of complex datasets to share their results with a wide audience base. We used it to develop the Global Fire Emissions Database (GFED) Analysis Tool, a website which guides the user through our dataset. A simple webmap interface allows users to select an area of interest and produce data visualization charts. ...

Mark Miller, Lawrence Livermore National Laboratory, Guest Blogger The HDF5 library has supported the I/O requirements of HPC codes at Lawrence Livermore National Labs (LLNL) since the late 90’s. In particular, HDF5 used in the Multiple Independent File (MIF) parallel I/O paradigm has supported LLNL code’s scalable I/O requirements and has recently been gainfully used at scales as large as 1,000,000 parallel tasks. What is the MIF Parallel I/O Paradigm? In the MIF paradigm, a computational object (an array, a mesh, etc.) is decomposed into pieces and distributed, perhaps unevenly, over parallel tasks. For I/O, the tasks are organized into groups and each group writes one file using round-robin exclusive access for the tasks in the group. Writes within groups are serialized but...

Release of HDF5-1.8.18 (Newsletter #152) - 11/16/16 Release of HDF Java Products for HDF5-1.8 (HDFView 2.13, HDF JNI 3.2.1) (Newsletter #151) - 7/25/16 Release of HDF 4.2.12 (Newsletter #150) - 6/30/16 Release of HDF5-1.10.0-patch (Bulletin) - 5/26/16 Release of HDF5-1.8.17 (Newsletter #149) - 5/13/16 Release of HDF5-1.10.0 (Newsletter #148) - 3/31/16 ...

David Pearah, The HDF Group Hello again, HDF User Community, As I mentioned in my last blog post -- HDF: The Next 30 Years (Part 2) -- we're looking for ways to better engage our users, which includes providing better tools for you to get support from the HDF Community.  We are looking for your input on three things: the HDF User Forum, Roadmap for HDF5, and Focus Groups - please take a few minutes to complete these short surveys and let us know what you think! Discussion Board and Listserv Forum Feedback:  LINK  We currently provide a listserv as the primary way that folks post questions and feedback to the community, and we would like your feedback since we're looking at web-based tools to complement or...

Dave Pearah, The HDF Group In my previous post—HDF: The Next 30 Years (Part 1)—I outlined the challenges and opportunities facing The HDF Group as an open source company. In a nutshell: Opportunity: large-scale adoption around the world in many different industries with great community-driven development (700+ projects in Github) Challenge: sufficient profit from existing business (consulting) to sustainably extend and maintain the core HDF5 platform The HDF Group is blessed with an amazingly talented + passionate + dedicated team of folks who care deeply about the HDF community, and we're all working together to determine the best path forward to sustainability, i.e. the NEXT 30 years. We want to share some of the steps that we're already taking, and -- more importantly --...

Dave Pearah, The HDF Group How can users of open source technology ensure that the open source solutions they depend on every day don’t just survive, but thrive? While on my flight home from New York, I’m reflecting on The Trading Show, which focused on tech solutions for the small but influential world of proprietary and quantitative financial trading. I participated in a panel called “Sharing is Caring,” regarding the industry’s broad use of open source technology. The panel featured a mix of companies that both provide and use open source software. Among the topics: Are cost pressures the only driving force behind the open source movement among trading firms, hedge funds and banks? How will open source solutions shape the future of...

The HDF Group’s HDF Server has been nominated for Best Use of HPC in the Cloud, and Best HPC Software Product or Technology in HPCWire’s 2016  Readers’ Choice Awards. HDF Server is a Python-based web service that enables full read/write web access to HDF data – it can be used to send and receive HDF5 data using an HTTP-based REST interface. While HDF5 provides powerful scalability and speed for complex datasets of all sizes, many HDF5 data sets used in HPC environments are extremely large and cannot easily be downloaded or moved across the internet to access data on an as-needed basis.  Users often only need to access a small subset of the data.  Using HDF Server, data can be kept in one...

David Pearah, The HDF Group Hello HDF Community! Thanks for the warm welcome into the HDF family: in my 4+ months as the new CEO, I've been blown away by your passion, diversity of interests and applications, and willingness to provide feedback on:  1. why you use HDF5?, and  2. how can HDF5 be improved? I also want to thank my predecessor Mike Folk for his invaluable and ongoing support. The HDF community is growing fast: when I last checked, there are nearly 700 HDF5 projects in GitHub! I've had the privilege of connecting via phone/web with dozens of you over the past few months. Across all of my discussions, one piece of feedback came back loud and clear: The HDF Group needs to be more engaged with its users and help foster...