HDF5 User Group 2021 – Daily Chat Log

Unfortunately, time stamps are set from the beginning of the meeting, which was actually started about 15 minutes before the scheduled start time. 

Tuesday, October 12, 2021

00:21:47 Gerd Heber: Joe also loves Emacs
00:31:14 Eric Winter: 😆
00:31:28 Eric Winter: “One of us! One of us!
00:42:34 Edward Hartnett: Thanks for the interesting talk! Have you tried using the new ncZarr capability recently released in netCDF?
00:45:11 Suren Byna: Can you post the link to the “Economics of Supercomputing” article?
00:56:56 Mike Smith: Could setting H5P_set_fill_time to “NEVER” help with the default fillign time?
00:57:29 Elena Pourmal: yes, this should help
01:07:37 Jean Luca Bez: Yes, as Elena mentioned, disabling filling should help. In my presentation tomorrow you’ll see a use case in which we got ~7x speedup
01:10:11 Elena Pourmal: Sunwoo, thank you for interesting talk! It is an interesting use case for THG to take a closer look and see which optimizations in the library could help.
01:12:43 Mike Smith: @Sunwoo, I rememer Gerd give a nice demo of optimising the dataset creation time in https://www.youtube.com/watch?v=JP72GWIKpMI&t=1362s
01:18:09 Sunwoo Lee: Thank you @Mike, I will definitely check the video!
01:20:14 John Readey: I’d suggest rather than many HDF libraries it might be more effective to have a single library aimed for read-only applications. Support multi-threading and S3 access and this would seem to serve the needs of many users
01:28:47 Lori Cooper: We have time for discussion. The talk time extends to 10:20, then we have a break to 10:35, but of course conversations can happen across the break time.
01:33:48 James Gallagher: +1
01:35:54 Alexander M. Martinez Mercado: Thanks
02:36:39 Matthew Dougherty: For the foreseeable future, the commercially-off-the-shelf DAOS and Infiniband RDMA are at the hardware cutting edge of data transfer and performance.

The CORE VFD would be a logical starting point in creating VFDs for DAOS and Infiniband RDMA, or perhaps using the Intel DAOS-POSIX drivers integrated into SEC2 or MPIO VFDs.

Regarding a hypothetical DAOS VFD and the Mercury/HDF5 DAOS RPC VOL, any guesses as to the performance difference? 1%, 10%, 2x, 10x, 100x

Are there plans to develop a DAOS VFD?
If not why?

Are there plans to develop an Infiniband RDMA VFD?
If not why?
02:36:56 Elena Pourmal: If you are interested in more details pluggable VFDs see https://docs.hdfgroup.org/hdf5/rfc/RFC_VFD_Plugin.docx.pdf
02:40:34 Werner Benger: This plugin handling in HDF5 – such as for VFD or filter plugins – has a big problem as it uses absolute paths instead of using a directory relative to an application installation folder. That way different application installations collide as they all try to read from the same plugin directly. Particularly when plugins need to be compiled differently (debug/release) mode, that is a problem, also with dependencies on different versions of 3rd party libraries. Applications can set the plugin path procedurally, but that does not apply to existing applications (h5ls, h5dump, …). It would be easy to solve by checking the executable’s directory as plugin path. (referring to page 13 of this presentation)
03:03:58 Elena Pourmal: https://jira.hdfgroup.org/browse/HDFFV-11276
03:08:22 Werner Benger: 👍
03:48:10 Andy Gotz: @Saba thanks for an interesting talk! If you use jupyterlab I suggest you take a look at h5web to visualise hdf5 files – https://github.com/silx-kit/jupyterlab-h5web
04:29:18 Elena Pourmal: 193 up to this date
04:29:49 Elena Pourmal: I mean 193 reports for h5diff
04:33:17 Werner Benger: HDF5 Tools: a reverse problem is “h5clear -s”, which allows to rescue many HDF5 files, but it’s not available programmatically
04:35:01 Lucas Villa Real: One thing I miss from the VOL Connector API is the ability to use “default” callbacks
04:35:15 Lucas Villa Real: For now, it’s quite difficult to write a VOL connector from scratch.
04:35:49 Lucas Villa Real: Terrific 🙂
04:36:19 Lucas Villa Real: Precisely
04:37:57 Werner Benger: Are all the presentation slides (PDF?) and recordings available online with the HUG page?
04:39:50 Lori Cooper: Presentation slides are added as we get them, and I’ll continue to bug presenters. I will start processing recordings with a goal to get them all up by Monday afternoon.
04:41:22 Werner Benger: Btw, what happened to the crash-safe journaling feature? Is it still planned or ad acta?
04:44:06 Werner Benger: okay, good step so far
04:45:07 Werner Benger: It would probably be sufficient to be able to recover the “last known consistent state” – if possible.

Wednesday, October 13, 2021

00:37:20 Werner Benger: Three questions on Async VOL:

1. Does the asynch model integrate (“speak to”) C++11 threads (std::async() ), is it bound to Posix threads (i.e. not using Windows threads), bound to MPI threads or is it completely agnostic and independent of some threading implementation?

2. Can we formulate model conditional execution, such as creating a dataset if opening failed, as part of the asynch execution queue?

3. Does it integrate with event-driven GUI applications? (e.g. waiting for a socket to be ready)
00:40:32 Marc Paterno: Another question on Async VOL:

What is the prognosis for using Async VOL to accelerate the reading speed (I
don’t care about writing speed) for other-language bindings? I am specifically
interested in h5py.
00:41:01 Richard Warren: Regarding the Async VOL: You are using the MPI_THREAD_MULTIPLE option when initializing MPI, have you been able to use darshan or other performance monitoring tools to examine the messaging use in your Async VOL applications? If so, can you provide some guidance on running those performance monitoring tool?
00:43:46 Houjun Tang: 1. Currently we use the Argobots as the thread engine, and it does not interfere with other threads an application may be using, e.g. pthread.
2. currently there is no conditional execution in the async vol level, so it won’t be fully asynchronous in this case, the dset open has to be completed and failed (need a ESwait) before calling the dset create
3. currently the dependency can only be specified for HDF5 operations
00:46:39 Houjun Tang: h5py can use async vol, but currently h5py does not use the H5*_async APIs or provide a way to wait on the operations (e.g. ESwait) to make sure the async read is finished, we are starting to look into this and will come up with a solution for h5py
00:48:42 Houjun Tang: we are still early in the MPI_THREAD_MULTIPLE vs. non-MULTIPLE performance, and only looked at the execution time as of now. Analyzing Darshan logs is on my todo list.
00:52:17 GREGORY SJAARDEMA: Does the Cache VOL automatically know whether the NvME is node-local or global, or does the user need to specify that somewhere.
00:52:29 Werner Benger: Cache VOL – can it have any benefits also for serial HDF5?
00:52:34 Gert Massa: can it be used without parallel IO?
00:54:14 GREGORY SJAARDEMA: What is the time frame for being able to use cache vol on an HDF5 from the main repository
00:54:19 John Readey: Is anyone interested in creating an async version of h5py?
00:55:37 GREGORY SJAARDEMA: By “main repository”, I actually meant “released version”
00:58:15 William Godoy: Great work and presentation.
00:58:29 Elena Pourmal: We are targeting all changes to be in for 1.13.0 which was planned for the end of the year
00:58:34 GREGORY SJAARDEMA: Nice presentations. Looking forward to using both Cache and Async soon
00:59:11 Rajeev Jain: Good talks!
01:10:12 Leigh Orf: Did I read ~ 12:1 compression rates with gzip? Seems … optimistic
01:10:38 Elena Pourmal: It was artificial data
01:13:59 Leigh Orf: Any time horizon for parallel lossy compression?
01:18:07 Leigh Orf: Thanks, that’s great news
01:28:19 Lucas Villa Real: Howard, do you have a precomputed preview dataset or do you go through the full dataset to generate the preview window? If you don’t have precomputed preview data, do you just sample data to reduce I/O? I ask this because the data load on your demo seemed to be pretty fast!
01:40:43 Mike Folk: Awesome application. I really like the viewer.
01:46:59 Lori Cooper: Yesterday’s videos should be on our YouTube account this afternoon. More organization and announcement will happen probably next week. https://www.youtube.com/channel/UCRhtsIZquL3r-zH-R-r9-tQ
03:12:36 Lori Cooper: PRESENTERS: Just a reminder to get your slides to me as soon as you can. Email lori.cooper at hdfgroup.org or use the form, link is in your email from last week. Thanks!
03:44:54 John Readey: This is they R package for HSDS: https://github.com/vjcitn/rhdf5client
04:06:25 Mike Smith: This custom sparsity implementation is also seen in bioinformatics. Many vendors ship data in HDF5 with 3 datasets representing values, indices, pointers
04:09:14 Gert Massa: I also implemented my solution for this
04:11:09 Marc Paterno: The data description sketched in Saba Sehrish’s talk shows how we implement our own solution for HEP data. In some cases, we deal with 6, 7, or 9 dimensional sparse datasets that we have “flattened” into 1-d datasets (and 6-8 index columns). We’d love to have a library solution.
04:13:49 Peter Ercius: its great to know how common the problem is! A library solution would be useful for a large number of different users and communities.
04:19:43 Lucas Villa Real: Peter, I mentioned about dynamically translating the data format using user-defined functions. The UDF platform lets you create a routine (that would convert from sparse to a regular gridded format, for instance) and associate that routine with a dataset. Each time that dataset is read that function executes and performs the translation on the fly. You get the best of the two worlds: data is stored in a compressed format and applications don’t have to be modified to learn about that sparse format.
04:21:08 Lucas Villa Real: There are some examples of how such routines look at https://github.com/lucasvr/hdf5-udf.
04:25:42 Peter Ercius: @Lucas very nice! Ill check it out.
04:36:44 Werner Benger: One dream question on HDF5: Can it move away from configure – created “config.h” files to more static configuration using C99/C++11 features such as has_include ? This would ease the build process and limit a user-specified config file to a minimum.
04:42:03 Gerd Heber: https://docs.hdfgroup.org/hdf5/develop/index.html
04:43:35 Lori Cooper: Yes we can put in redirects when it’s a little more final
04:45:41 Marc Paterno: Does THG have any relation to the keeping of the (macOS) Homebrew recipe for hdf5?

Thursday, October 14, 2021

00:52:13 Werner Benger: Can drill be used within an application to perform SQL queries on HDF5 files? Do the Java limitations apply in that case, assuming using the C++ interface then?
00:52:53 Hyokyung Lee: I can see a big performance improvement for HDF5 in 1.20.SNAPSHOT. When will Apache Drill 1.120 be officially released?
01:11:07 Werner Benger: Might it be possible to make use of the memory paging capability of a VFD such that a H5Dread() actually does not read at all, but just exposes a memory location where the file contents are paged in and out? Obviously, that would only work for uncompressed/unfiltered data, and probably require new public HDF5 APIs.
01:38:15 Mike Smith: H5Pset_fill_value() is applied to a dataset creation property list. Doesn’t this persist inside the file as part of the dataset, and can be accessed later? I’ve not tried.
01:59:52 Elena Pourmal: What do you mean by “accessed later”? Creation properties cannot be modified after dataset is created
02:00:05 Elena Pourmal: One can discover creation properties of a dataset
02:02:30 Mike Smith: There was a comment in Edward’s talk about not being able to identify what fill time had been applied if you’re given a file after it’s been created. I was speculating that you could get this from the creation property list.
02:04:26 Elena Pourmal: Probably what he meant, there is no netCDF-4 API. One still can open netCDF-4 file with HDF5 library and use HDF5 calls to discover the properties
02:10:04 Mike Smith: Ah, thanks for the clarification.
02:39:49 Elena Pourmal: Yes, it is VERY COOOL! THANK YOU!
02:45:21 Werner Benger: Mostly a question to THG: Can it be (or is it already) part of a test suite to ensure also sustained performance, not just sustained functionality?
03:06:10 Werner Benger: 1. Is this windows only? Or based on e.g. QT to be portable?
2. How does it display hard and soft links?
3. Does it read the entire file structure at once or only on-demand when needed, e.g. when opening a subgroup?
03:08:12 Werner Benger: okay, thanks, great presentation!
03:08:36 William Godoy: Yes, really nice!
03:09:33 Elena Pourmal: Very nice! Such a clean and elegant view of HDF5 file
03:14:19 Robert Seip: Thank you for your comments. Yes, I will communicate with Lori to get my info to you.
03:14:52 Lori Cooper: No problem! I’m always happy to do practice sessions.
03:18:54 Werner Benger: Just to mention, I’d like to explore H5S with our 8TB HDF5 file. It may be problematic if it scans the entire metadata on file opening, in our visualization application we added some optimization to cache the attributes of datasets in a subgroup in a single dataset. Reading just the attributes of 80.000 datasets in a group would take several minutes otherwise…
03:27:00 John Readey: Werner, I think there are some tools you can use to pack the metadata in the front (or end) of the file. Maybe Elena can provide some pointers…
03:43:27 Elena Pourmal: John, one can control aggregation of MD and read/write in pages, but there is no implementation to put all MD at the beginning or at the end. We have a split driver to store all MD in a separate file. We will need to expand on that idea. It would be very useful! Also H5Oget_info was optimized, it is faster now.
03:44:16 John Readey: Yep – I was thinking a repack option to do this would be handy
03:45:40 Werner Benger: We are actually using this split file driver for our big files, it improves performance lots. However it requires the application to explicitly support it and to open files using the split VFD.
03:52:48 Elena Pourmal: Yes, and this should be improved. We are well aware about this problem now 🙂
03:55:33 Gerd Heber: How long did it take to copy the file?
03:59:57 Hyokyung Lee: Did you set the limit in EFS throughput to save cost in your study?
04:01:32 Werner Benger: It’s running under NTFS under windows.
04:02:07 Werner Benger: @Gerd Copying 8TB? I think three days. Creating it from scratch (ie. out of 80.000 tiles) takes about two weeks. Btw., a related issue is the selection of a VFD when files are accessed via external links… currently the application needs to explicitly decide which VFD to use, but that contradicts the purpose of external links which are otherwise invisible.
04:03:27 Elena Pourmal: Could you please elaborate on external links and VFDs? It should be transparent. External files will use the same vFD
04:08:10 Werner Benger: For instance when an external link goes to a file that was created with the split file driver, or another link goes to a file that was created by the SEC2 file driver. The application would need to know which one is meant and set the VFD via H5Pset_elink_fapl(), rather than HDF5 being able to figure it out on its own, such that the application can just ignore whether a dataset is an external link or not.
04:08:26 John Readey: Gary – that’s very interesting that you used the same file for your benchmark. It would be interesting do run a test involving writing to HDF5. Of course this is impossible with the S3 VFD, but should be supported with flexfs.
04:09:05 John Readey: I imagine performance will depend on what the block size flexes uses and the nature of the application. Each hdf5 write will need to re-write the entire block.
04:10:08 Gary Planthaber: Yeah, good points John. I plan to make flexFS available to the HDF Group so we can do some joint benchmarks in more detail.
04:54:39 Robert Seip: Thank you HDF Group, 2021.
04:54:41 Ellen Johnson: Thank you to THG and to all the presenters! Great HUG!