mobile-logo

HSDS Architecture

HSDS Schema

Map individual HDF5 objects (datasets, groups, chunks) as objects in Object Storage to store HDF5 content in S3:

  • Limit maximum storage object size
  • Support parallelism for read/write
  • Only data that is modified needs to be updated

Each chunk (heavy outlines) get persisted as a separate object:

  • Dataset is partitioned into chunks
  • Each chunk stored as an S3 object
  • Dataset meta data (type, shape, attributes, etc.) stored in a separate object (as JSON text)

Architecture

If you’re interested in further information, senior developer John Readey put on this recorded webinar.

This material is based upon work supported by NASA under award Number NNX16AL91A. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Aeronautics and Space Administration.