mobile-logo

Uncategorized

John Readey, The HDF Group Before there was HSDS, there was h5serv. Released in 2015, h5serv was the first implementation of the HDF Rest API. Designed mainly as a way to demonstrate the RESTful interface for HDF, h5serv had a fairly simple implementation: A single threaded application that on receiving an HTTP request, made the equivalent HDF5 library call and converted the result to a JSON response which was returned to the client. Though useful for some applications, in the context of building a scalable web service, there were limitations with this approach. Since there was only one process in the h5serv application, each HTTP request had to be completely processed before handling the next one. This made it quite easy to...

October 31, 2022

Dana Robinson has been appointed as the new Director of Engineering at The HDF Group. Dana started at The HDF Group in 2009 as a software engineer until stepping into the role of interim Director of Engineering in April 2022. As the Director of Engineering, Dana will lead the team of software engineers and shape the engineering culture, including developing the roadmap for HDF5 releases. Externally, he will engage at all levels in research, development, consulting, opportunity analysis, and community outreach. A skilled software engineer and scientific data architect with over twenty-five years of experience across a wide variety of disciplines, Dana started his career at The HDF Group working on a bioinformatics project, then moved to library development. He’s worked on projects including...

October 19, 2022

When using HDF5 or HSDS you’ve likely benefited (even if you weren’t aware of it) caching features built into the software that can drastically improve performance. HSDS and h5pyd utilize caching to improve performance for service-based applications. In this post, we’ll do a quick review of how HDF5 library caching works and then dive into HSDS and h5pyd caching (with a brief discussion of web caching)....

October 17, 2022

We are excited to announce the appointment of Neil Fortner as the new Chief HDF5 Software Architect. Neil has worked for The HDF Group as a software engineer since 2008. While at The HDF Group, he focused his talents in storage and HPC on improving performance, expanding the features, and improving the maintainability of the HDF5 library. Neil has a proven history of quickly diving into and improving large, complex code bases. As the Chief HDF5 Software Architect, Neil will develop detailed architectural designs for software solutions internally and for our consulting clients. He will be a mentor within the engineering team as we continue to grow. Neil received his bachelor's in Aerospace Engineering from the University of Maryland. In his spare...

October 3, 2022

Highly Scalable Data Service principal architect John Readey covers an update to the Highly Scalable Data Service. The max request size limit per HTTP request no longer applies with the latest HSDS update. In the new version large requests are streamed back to the client as the bytes are fetched from storage. Regardless of the size of the read request, the amount of memory used by the service is limited and clients will start to see bytes coming back while the server is still processing the tail chunks in the selection. The same applies for write operations—the service will fetch some bytes from the connection, update the storage, and fetch more bytes until the entire request is complete. Learn more about...

August 22, 2022
« Previous Page — Next Page »