Get your Bearings with HDF Compass
John Readey, The HDF Group
We’ve recently announced a new viewer application for HDF5 files: HDF Compass. In this blog post we’ll explore the motivations for providing this tool, review its features, and speculate a bit about future direction for Compass.
HDF Compass is a desktop viewer application for HDF5 and other file formats. A free and open source software product, it runs on Mac OS X, Windows, and Linux.
Compass was initially developed by Andrew Collette, a Research Scientist with IMPACT (Institute for Modeling Plasma, Atmospheres and Cosmic Dust). He has decided to work with The HDF Group to further the development of Compass. Andrew has written a very interesting blog that goes into some of the background of Compass and the challenges involved with creating GUI applications.
Yet another HDF Viewer?
If you are familiar with HDFView you may be thinking: “Why another HDF viewing application?” While there is some overlap, Compass is not intended to replace HDFView, but rather, to offer a simplified, light-weight viewer for HDF5 and other formats.
One distinction is that while HDFView supports both reading and editing HDF5 files, Compass is read-only. This tremendously reduces the complexity of the user interface. If all you need to do is make a quick check to see data values in a HDF5 file or draw a simple plot, HDF Compass is the viewer of choice. On the other hand, if you need to create datasets, edit values, or look into the more obscure details of an HDF5 file, HDFView will master the job.
To help make Compass easy to use, we’ve used a UI toolkit that allows it to conform to the user’s native platform’s look and feel. Applications that follow the platform’s look and feel (both in terms of visual appearance and how the user interacts with the application) have improved usability since the user can leverage his hard-won experience with the preferred operating system (Mac OS X, Windows, or Linux).
While the focus is on ease of use, Compass is no light-weight! It is designed to handle whatever data you care to throw at it – whether you have a file with 50,000 groups or a 50Kx50K dataset, Compass is designed to handle these gracefully. Compass reads data from the file on demand (e.g. to fill in the cells of the grid view), so it can be used even when the size of the dataset is larger than the amount of physical memory on your system.
Extendable Data Model
Compass is designed with a plugin architecture that makes it relatively easy to support additional file formats or remote resources (e.g. OPeNDAP, see: “Worried about your unlimited data plan bills? Cut them with OPeNDAP“) through the use of “plugins.” In addition, a Compass plugin can be used to provide an alternate means of displaying an existing format (say, HDF5), for a specialized domain.
As the saying goes, “Ignorance is Bliss,” and Compass is a good example of this. In computer software design, “loose coupling” refers to a design principle where the components of the system don’t have direct knowledge of each other. This approach makes it easier to extend the system and at the same time have it less prone to unintended side-effects when one component is modified.
In Compass, the UI elements don’t “know” anything about the file format, and the plugins don’t need to deal with the UI. What binds these together is the Compass Model, below. The model is an abstract representation of objects such as key/value store, arrays, images. The plugin provides implementations for these based on the file format or resource it supports and UI elements display these visually.
This approach greatly simplifies the effort in developing new plugins. To support a new file format (or alternate means of presenting a supported file format), all that is needed is for the plugin to map constructs from the file to objects in the Compass data model. Currently, Compass supports HDF5, OPeNDAP, and ASC Grid file formats. The hope is that this list will greatly expand in the future, since many file formats would be fairly easy to add as a plugin. For example, the ASC Grid Format was implemented by a summer intern who did this as his first week’s “getting started” project.
If you have a special interest in a file format that fits into the Compass Data model (which is quite inclusive!) and have ever thought it would be nice to a have a simple viewer for that format, I’d encourage you to think about leveraging Compass rather than developing a new application. Especially in the scientific area, there are thousands of different file formats that would benefit from having a viewer application. Using the Compass plugin model will make this much easier compared with writing an application from scratch.
Community Development Model
As an open source product, I’d like to encourage the community development of Compass. By leveraging the efforts of many people within the broader HDF-user base, we can create a better tool that integrates the perspectives of different users.
The code for Compass is hosted on GitHub (https://github.com/HDFGroup/hdf-compass), which removes much of the friction of coordinating software development across a diverse set of contributors. If you have an idea to improve Compass, and would like to contribute, you can submit your idea here on GitHub, post to The HDF Group forum mailing list email@example.com, or add a comment below.
As noted above, the goal is to provide a light and focused viewer in Compass, but there are many enhancements that can be made within that model. Here are some possible enhancements we are considering:
1. Search Box;
2. Sort and Filter options;
3. Clipboard (e.g. copy values to csv, paste to Excel);
4. Improved plotting.
Future plans for Compass will largely be determined by your feedback, so let your wishes be known! (Note the feedback button at the bottom of this post, or at the top under the title.)
Give HDF Compass a Spin
If you are comfortable with Python and package management, you can download the source from GitHub (https://github.com/HDFGroup/hdf-compass) and run Compass directly. (Since this is Python, there’s no compile step needed). This is also the easiest way to get the up-to-the-minute changes to the project.
For the rest of us, the easiest approach is to grab either the one-click .exe (for Windows), or the .dmg installer (Mac OS X). Please see below. If you are a penguin fan, sorry, we don’t have a Linux installer yet!
The HDF Group is committed to meeting our users’ needs and expectations for managing data in today’s fast evolving computational environment. Let us know what you think!
Binaries are here:
Mac OS X (Yosemite):
Sample files: https://support.hdfgroup.org/HDF5/examples/api18-c.html
With the binaries, there are no dependencies, everything is self-contained (including the Python interpreter).