Compression ideas

[bendavidsteel]

I have a ~3mil dataset I'm trying to create a datamap from, and final file size with inline data is very big. I know I could switch to non-inline to alleviate, but would be nice to also have smaller inline compression. I'm trying to think of ways to improve the data compression, and was wondering if you had ideas you haven't had time to implement that you could add as issues to work on in the repository.

I had a few ideas:
Rounding floats in a smart way
Ensuring RGB values are saved as integers (seems like they might be being saved as floats right now)
Allowing an option for zstd compression (zstd = smaller but more recent so less browser support) rather than gzip
Claude is suggesting brotli over gzip or zstd as a widely supported compression format

Wondering if you had thoughts on what would be some of the most fruitful avenues to go down?

[lmcinnes]

I played with various compression options some time ago, including bzip2 and brotli. Neither had good enough browser support to be consistently viable. That may have changed.

An alternative option I have been thinking about is using duckdb for all the data, and perhaps they have a good compressed format for database dumps that we could push? I haven't had time to look into all of that at the proper level of detail.

It is worth noting that usually it is the metadata (often hover-text) that consumes the most space (you'll see this if you use the non-inline-data option), so better ways to compress that will get you most of the benefit.