mobile-logo

Earth Sciences

Why HDF® Technologies?

NASA’s Earth Observing System (EOS) consists of satellites whose observations include the land surface, atmosphere, oceans, and snow and ice. The first EOS satellite, Terra, was launched in 1999, and produced about a terabyte of data per day. By 2015, 18 EOS satellites were producing 16 TB per day stored in 12 data archive centers that specialized in different aspects of the Earth system, such as snow and ice, physical oceanography, severe weather, land cover, and global precipitation. Now, totaling over 14 petabytes, the data encompasses 9,400 different data products. In 2015, 32 terabytes per day were distributed to more than 2.4 million and users worldwide.

In the mid-1990s EOS chose HDF as its data format, and HDF, in the form of HDF5, continues to be the format of choice. Other earth satellite systems, such as that of the European Space Agency, Landsat, and the Joint Polar Satellite System (JPSS) have followed suit.

Airborne applications, such as the NEON Airborne Observation Platform and Airborne Hydro Mapping also use HDF5.

HDF5 finds widespread use in most other earth sciences as well. The atmospheric science community has a format called netCDF-4, which is HDF5 with an interface and tools that are particularly suited to meteorology and related sciences.

The Bathymetry Attributed Grid (BAG) format is based on HDF5 and was developed by the bathymetry community to address the increasing resolution and metadata requirements of bathymetry data.

Likewise, there are HDF5 based standards and applications for weather radar, earthquakes and other seismology, hydrology and hydrologic modeling.

The attraction of HDF for such a wide ranging set of applications is that it does so many things well that are needed to accommodate Earth science data. As satellites and other instruments increase their ability to collect data at higher and higher resolution, they need a format that can scale easily and that can provide the ability to acquire data quickly, provide access to specific areas of interest out of large masses of information, accommodate increasingly complex and evolving metadata, and be trusted to support long-term archiving. HDF5, and HDF5-based formats such as BAG and netCDF-4, provide the best, and sometimes the only solution for many of these applications.