HDF5 User Group 2021 – Daily Chat Log

Tuesday, October 12, 2021

Unfortunately, time stamps are set from the beginning of the meeting, which was actually started about 15 minutes before the scheduled start time. 

00:21:47 Gerd Heber: Joe also loves Emacs
00:31:14 Eric Winter: 😆
00:31:28 Eric Winter: “One of us! One of us!
00:42:34 Edward Hartnett: Thanks for the interesting talk! Have you tried using the new ncZarr capability recently released in netCDF?
00:45:11 Suren Byna: Can you post the link to the “Economics of Supercomputing” article?
00:56:56 Mike Smith: Could setting H5P_set_fill_time to “NEVER” help with the default fillign time?
00:57:29 Elena Pourmal: yes, this should help
01:07:37 Jean Luca Bez: Yes, as Elena mentioned, disabling filling should help. In my presentation tomorrow you’ll see a use case in which we got ~7x speedup
01:10:11 Elena Pourmal: Sunwoo, thank you for interesting talk! It is an interesting use case for THG to take a closer look and see which optimizations in the library could help.
01:12:43 Mike Smith: @Sunwoo, I rememer Gerd give a nice demo of optimising the dataset creation time in https://www.youtube.com/watch?v=JP72GWIKpMI&t=1362s
01:18:09 Sunwoo Lee: Thank you @Mike, I will definitely check the video!
01:20:14 John Readey: I’d suggest rather than many HDF libraries it might be more effective to have a single library aimed for read-only applications. Support multi-threading and S3 access and this would seem to serve the needs of many users
01:28:47 Lori Cooper: We have time for discussion. The talk time extends to 10:20, then we have a break to 10:35, but of course conversations can happen across the break time.
01:33:48 James Gallagher: +1
01:35:54 Alexander M. Martinez Mercado: Thanks
02:36:39 Matthew Dougherty: For the foreseeable future, the commercially-off-the-shelf DAOS and Infiniband RDMA are at the hardware cutting edge of data transfer and performance.

The CORE VFD would be a logical starting point in creating VFDs for DAOS and Infiniband RDMA, or perhaps using the Intel DAOS-POSIX drivers integrated into SEC2 or MPIO VFDs.

Regarding a hypothetical DAOS VFD and the Mercury/HDF5 DAOS RPC VOL, any guesses as to the performance difference? 1%, 10%, 2x, 10x, 100x

Are there plans to develop a DAOS VFD?
If not why?

Are there plans to develop an Infiniband RDMA VFD?
If not why?
02:36:56 Elena Pourmal: If you are interested in more details pluggable VFDs see https://docs.hdfgroup.org/hdf5/rfc/RFC_VFD_Plugin.docx.pdf
02:40:34 Werner Benger: This plugin handling in HDF5 – such as for VFD or filter plugins – has a big problem as it uses absolute paths instead of using a directory relative to an application installation folder. That way different application installations collide as they all try to read from the same plugin directly. Particularly when plugins need to be compiled differently (debug/release) mode, that is a problem, also with dependencies on different versions of 3rd party libraries. Applications can set the plugin path procedurally, but that does not apply to existing applications (h5ls, h5dump, …). It would be easy to solve by checking the executable’s directory as plugin path. (referring to page 13 of this presentation)
03:03:58 Elena Pourmal: https://jira.hdfgroup.org/browse/HDFFV-11276
03:08:22 Werner Benger: 👍
03:48:10 Andy Gotz: @Saba thanks for an interesting talk! If you use jupyterlab I suggest you take a look at h5web to visualise hdf5 files – https://github.com/silx-kit/jupyterlab-h5web
04:29:18 Elena Pourmal: 193 up to this date
04:29:49 Elena Pourmal: I mean 193 reports for h5diff
04:33:17 Werner Benger: HDF5 Tools: a reverse problem is “h5clear -s”, which allows to rescue many HDF5 files, but it’s not available programmatically
04:35:01 Lucas Villa Real: One thing I miss from the VOL Connector API is the ability to use “default” callbacks
04:35:15 Lucas Villa Real: For now, it’s quite difficult to write a VOL connector from scratch.
04:35:49 Lucas Villa Real: Terrific 🙂
04:36:19 Lucas Villa Real: Precisely
04:37:57 Werner Benger: Are all the presentation slides (PDF?) and recordings available online with the HUG page?
04:39:50 Lori Cooper: Presentation slides are added as we get them, and I’ll continue to bug presenters. I will start processing recordings with a goal to get them all up by Monday afternoon.
04:41:22 Werner Benger: Btw, what happened to the crash-safe journaling feature? Is it still planned or ad acta?
04:44:06 Werner Benger: okay, good step so far
04:45:07 Werner Benger: It would probably be sufficient to be able to recover the “last known consistent state” – if possible.