Ted Habermann, The HDF Group
“A strong foundation is being built for sharing data and information to create community knowledge and wisdom. This foundation includes HDF5 as the data layer with community conventions and ISO metadata facilitating use and understanding.”
We have experienced so many monumental technological shifts during the last several decades that, like the diurnal cycle of light and dark, the technology life cycle (shown below) is becoming instinctual.
It starts with a new idea, (usually aimed at new customers), that destroys existing organizational expertise and threatens the continued existence of established processes and organizations. These disruptions raise a variety of difficult questions and initiate an Era of Ferment during which established enterprises gauge the impact of the disruption in their worlds and try to adjust. The ferment creates uncertainty, high risk, considerable wasted resources, and no interoperability.
The ferment is ended when the community agrees on a dominant design and works together to make the design work. Instead of deciding what they are going to do, they work to make what they are going to do better.
An Invisible Standard
During 1993, NOAA did an assessment of HDF and concluded that the diversity of their data and users required multiple formats and that data description languages were going to decrease the future need for standard formats. These conclusions supported the prevailing attitude at the time. As is so often the case, however, the scientific community went a different direction. A more general version of HDF emerged (HDF5) and it has been selected as a foundation format in Earth Science and many other scientific and commercial disciplines. The best standards are invisible and HDF has achieved invisibility across a wide variety of tools and users.
In an upcoming blog I describe the idea of a spectrum of understanding that begins with data and progresses through information and knowledge to wisdom. We have learned a lot about interoperable data as HDF has been adopted across a variety of disciplines. Can we build on that understanding to create interoperable information? Evidence from several areas says, “Yes.”
We are all used to using hierarchical structures to organize data in a way that makes sense to us or to our communities. In some cases these are physical directories on laptops. In others, they are virtual directories supporting data access services (e.g. THREDDS). In contrast, metadata have typically been organized in records that describe individual resources, but the ISO 19115 standard introduced the ability to organize metadata in hierarchies that mirror the organization of collections of resources. Hierarchical data becomes hierarchical information.
Supporting Community Needs
Increased use of services for access to data, and mechanisms like content negotiation in web servers, have made us used to requesting and serving data in a different format than it is stored. OPeNDAP services are an excellent example of this idea developed and used in Earth Science. OPeNDAP servers hide the native format of the data behind the Data Access Protocol (DAP) used on the wire. In the metadata world, persistence layers are often relational or document databases that support indexing and rapid search while the results are provided in XML or some other representation that can be easily translated to support community needs or expectations. Again, information is transported independent of underlying data.
These examples, and others, suggest that knowledge and skills developed in the data layer can, in fact, be used to improve interoperability in the information layer. But – what about the selection event? Selection events are often times more like a series of community nudges rather than a single lightning strike. In the metadata case, these nudges come in the form of adoption of ISO TC211 metadata standards by a variety of U.S. and International Organizations including NOAA, NASA, FGDC, WMO, ANZLIC, INSPIRE, and others. The metadata selection event is clearly in progress.
A strong foundation is being built for sharing data and information to create community knowledge and wisdom. This foundation includes HDF5 as the data layer with community conventions and ISO metadata facilitating use and understanding. Interoperability lessons learned in the data layer are being extended into the information layer. Interoperable Information is coming!