Support Zstd compression

[PallHaraldsson]

I noticed it done here: https://github.com/JuliaIO/HDF5Plugins.jl/blob/main/Project.toml

I noticed in this package Blosc is commented out, for some reason (also not in the package above/main HDF5), not sure why, I don't really care if it's supported if the best format is support that I think is Zstd (seems also important for compatibility with main format):

JLD2.jl/src/compression.jl

Line 13 in f7a946b

#UInt16(32001) => (:Blosc, :BloscCompressor, :BloscDecompressor, "BLOSC"),

I also commented on another closed PR (on pointers to null), regarding a typo (and slipped in a question). I hope people see it, not sure if they do since it's closed. Another question, do (Ptr) nulls take up space in the output stream?

[JonasIsensee]

This should be relatively easy to add.
Here's the list of registered HDF5 filter plugins (for the id)
https://confluence.hdfgroup.org/display/support/Registered+Filter+Plugins

[mkitti]

I should point out that the plugins in HDF5Plugins.jl are implemented in Julia, having been ported from C. They depend on the Codec* packages but mainly use them for C library wrapper.

For example, the Zstandard plugin implementation is found here:
https://github.com/JuliaIO/HDF5Plugins.jl/blob/main/src/H5Zzstd.jl
It uses the CodecZstd.LibZstd submodule:
https://github.com/JuliaIO/CodecZstd.jl/blob/master/src/libzstd/LibZstd.jl

Also note that we are merging the plugins directly into the HDF5.jl git tree:
JuliaIO/HDF5.jl#875
Currently they are being integrated as submodules, but we have plans to spin the individual modules out as subtree packages.

[mkitti]

Looking at how the current compression filters are implemented, I'm not sure if they are compatible with the way the HDF5 plugins work. For example, for LZ4 compression the HDF5 plugin stores the original size of the file as well as the blocksize before storing the compressed data.

https://github.com/JuliaIO/HDF5Plugins.jl/blob/main/src/H5Zlz4.jl

[JonasIsensee]

Hi @mkitti ,

it's great to see you taking an interest in this.

I'm not sure if they are compatible with the way the HDF5 plugins work

This is important. JLD2 compression tries to conform to the HDF5 standards. Thus JLD2 files using Bzip2 and Zlib compression can be decoded by HDF5.jl. This is sadly not the case for LZ4 but I was unable to figure out why, but you may be hinting in the right direction?

It'd be very neat, if one could transfer the knowledge gained from HDF5Plugins.jl to enabling the same compression algorithms in JLD2.

[milankl]

@JonasIsensee bumping this up, I still get a

ArgumentError: Unsupported Compressor
Supported Compressors are 
	JLD2.ShuffleFilter
	CodecBzip2.Bzip2Compressor
	CodecLz4.LZ4FrameCompressor
	CodecZlib.ZlibCompressor

when using compress = ZstdCompressor(). How much work is it to add this? Happy to help

[JonasIsensee]

Shouldn't be much work at all.
Here's we the compressor needs to be added

JLD2.jl/src/compression.jl

Lines 51 to 66 in ccd60eb

	const COMPRESSOR_TO_ID = Dict(
	:ZlibCompressor => UInt16(1),
	:ShuffleFilter => UInt16(2),
	:Bzip2Compressor => UInt16(307),
	#:BloscCompressor => UInt16(32001),
	:LZ4FrameCompressor => UInt16(32004),
	)
	# For loading need filter_ids as keys
	const ID_TO_DECOMPRESSOR = Dict(
	UInt16(1) => (:CodecZlib, :ZlibCompressor, :ZlibDecompressor, ""),
	UInt16(2) => (:JLD2, :ShuffleFilter, :ShuffleFilter, ""),
	UInt16(307) => (:CodecBzip2, :Bzip2Compressor, :Bzip2Decompressor, "BZIP2"),
	#UInt16(32001) => (:Blosc, :BloscCompressor, :BloscDecompressor, "BLOSC"),
	UInt16(32004) => (:CodecLz4, :LZ4FrameCompressor, :LZ4FrameDecompressor, "LZ4"),
	)

and the corresponding filter ID, you can look up at:

This should be relatively easy to add. Here's the list of registered HDF5 filter plugins (for the id) https://confluence.hdfgroup.org/display/support/Registered+Filter+Plugins

[milankl]

Great, I'll create a PR! :ZstdCompressor is the preferred name?

[mkitti]

You can use this as a reference:
https://github.com/JuliaIO/HDF5.jl/blob/master/filters/H5Zzstd/src/H5Zzstd.jl

[mkitti]

The new registered filter plugins reference page is here:

https://github.com/HDFGroup/hdf5_plugins/blob/master/docs/RegisteredFilterPlugins.md