Performance and compatibility testing on HDF5 1.14

[markcmiller86]

No description provided.

[markcmiller86]

I've run some tests to check compatability between HDF5 versions. I created multi_ucd3d.h5 with Silo 4.11.3RC and HDF5-1.14.0 and then used browser from an old install of Silo 4.10 on CZ to read it and it read everything fine. I need to test checksumming and compression under the same conditions. But, this does indicate older silo/hdf5 combinations will read files produced by newer silo/hdf5 combinations.

[markcmiller86]

So, when Silo is run with DBSetCompatibilityMode(DB_PERF_OVER_COMPAT), the files produced are not readable at all into earlier versions of HDF5. I was expecting that maybe H5open would succeed but that maybe some objects would not be readable but H5open is not even succeeding.

[markcmiller86]

If I write ZFP compressed data with newer Silo/HDF5, the older Silo cannot read it. This is expected because the older Silo library is not set up to use ZFP and there was no generalized approach to specifying an HDF5 filter plugin in older Silo. This is something that should probably be changed.

[markcmiller86]

I discovered HDF5's libver stuff has two ways of operating...one as a file-access property (H5Pset_libver_bounds()) set during open/create time. The other is a switch on a currently open file (H5Fset_libver_bounds())

The former seems to create issues when opening a file. I was using that and wound up being unable to open files with older lib even when I set bounds to older versions.

I am now using it as a switch once a file is opened or created. So, Silo makes no call to set libver bounds unless compatability setting in Silo's global (or file) properties require it. Ordinarily, they are unset and so no calls to H5Fset_libver_bounds() are made. But, those will be made if caller specifically sets compatability mode of Silo library.

Presently, setting compatability mode on an open file from Silo's DBSetCompatibilityMode is non-operative.

[markcmiller86]

I am able to produce files using new features of HDF5 and some parts of those files are readable in old versions. I can diff files (using tools built on new version of HDF5) and they compare correctly to old versions of those same files. The files using newer features are a bit smaller (about 10%).

Files written with newer HDF5 library (1.14.0) and features

-rw------- 1 miller86 miller86  109394 Sep 17 12:09 multi_curv2d.h5
-rw------- 1 miller86 miller86 1991238 Sep 17 12:09 multi_curv3d.h5
-rw------- 1 miller86 miller86   90007 Sep 17 12:09 multi_point2d.h5
-rw------- 1 miller86 miller86  181392 Sep 17 12:09 multi_rect2d.h5
-rw------- 1 miller86 miller86 1702584 Sep 17 12:09 multi_rect3d.h5
-rw------- 1 miller86 miller86 6655274 Sep 17 12:09 multi_ucd3d.h5

Files written with older HDF5 library (1.8.22)

-rw------- 1 miller86 miller86  120384 Sep 17 12:35 multi_curv2d.h5
-rw------- 1 miller86 miller86 2050936 Sep 17 12:35 multi_curv3d.h5
-rw------- 1 miller86 miller86  100696 Sep 17 12:35 multi_point2d.h5
-rw------- 1 miller86 miller86  203808 Sep 17 12:35 multi_rect2d.h5
-rw------- 1 miller86 miller86 1773332 Sep 17 12:35 multi_rect3d.h5
-rw------- 1 miller86 miller86 6814432 Sep 17 12:35 multi_ucd3d.h5

[markcmiller86]

It is possible for a Silo caller to switch HDF5 library version bounds while the file is open (as opoposed to using the setting specified by the library when DBCreate() or DBOpen() is called using this idiom...

const void *fidvp = DBGrabDriver(dbfile);
hid_t fid = *((hid_t*)fidvp);
H5Fflush(fid); /* do this to ensure already written stuff is in the file using the old libver bounds */
H5Fset_libver_bounds(fid, H5F_LIBVER_XXX, H5F_LIBVER_XXX);
DBUngrabDriver(dbfile, fidvp);

I have played around with this and am surprised that

[markcmiller86]

@vijaysm and @iulian787 and have a look at HDFGroup/hdf5#4849