Why Lance instead of Icechunk?

[TomNicholas]

Hey, I just saw this project, and it looks really cool! It aligns very well with how we think about these problems at Earthmover, and your page on introducing SLAF describe the evolution of the Pangeo project pretty well (HDF to python to dask to zarr to rust).

I have a few questions:

• Have you heard of Icechunk? Your section on LanceDB might as well have been written about Icechunk. Icechunk is effectively Lance for Zarr.
• What is the relationship of SLAF as it currently stands to Zarr? and to multi-dimensional arrays in general?

[pavanramkumar]

Thanks for reaching out @TomNicholas!

I love the Pangeo / zarr v3 / Icechunk community. I think when I started thinking about SLAF,

• zarr v3 wasn't live and zarr v2 was not sufficient for the kinds of random access patterns and fine grained control of shards vs chunks I was seeking
• I wasn't sure whether zarr was the format of choice for extremely large sparse arrays as opposed to dense arrays
• I wasn't sure that zarr groups was the right place to store tabular metadata around the core array data itself (I've seen that xarray is a popular choice for this)
• I wanted to support embeddings and their CRUD lifecycle along with fast vector search. Zarr doesn't ship with HNSW or equivalent ANN implementations
• I also wanted to support B-Tree like indexes for scalar values to speed up queries on metadata and lance ships with those.

TLDR, I didn't find another open cloud-native storage format that met these needs besides lance. Building with lance has been great so far and I haven't found a strong reason to switch :)

That said, I'm really excited about zarr v3 and Icechunk. Amazing work!

[pavanramkumar]

As to your second question, I haven't thought deeply enough beyond single cell transcriptomics for SLAF, but there are design patterns here that can extend to other biological data formats like sequences and protein structures (not quite ndarrays). I think for imaging data like climate raster data and microscopy, I likely pick zarr

[pavanramkumar]

You made me really curious about what it would take to build a ndarray interface on top of lance, particularly with blobs v2. Here's a minimal forkable repo, not meant as a stand-in for zarr v3 but showing that it's possible to build an ndarray capability on lance backend: https://github.com/slaf-project/lance-array

I didn't expect the benchmarks to be ~2x faster for random reads and order of magnitude faster for batch reads using lance!

[TomNicholas]

Fascinating, thank you.

All of these things are things we have discussed recently in the Icechunk / Zarr community. You might also be interested in https://github.com/developmentseed/zarr-datafusion-search.

it's possible to build an ndarray capability on lance backend

Cool! I've been wanting to try that! But it looks like what you made there is limited to 2D arrays?

I didn't expect the benchmarks to be ~2x faster for random reads and order of magnitude faster for batch reads using lance!

Sorry, 2x faster than what?

[pavanramkumar]

Cool! I've been wanting to try that! But it looks like what you made there is limited to 2D arrays?

Yes, it's < 1 week old, currently just tinkering to find out if it's even possible and deliberately kept the scope narrow. I haven't decided if I want to build it into a feature complete equivalent of zarr yet. 2D -> ND arrays is not hard in terms of schema evolution, but covering every numpy-like slicing (for reads) and slice-assignment case (for updates) could be a big project. Also, out-of-core dask compatibility will be another big lift. With these two, though, it feels like lance-array could be somewhat close to zarr + icechunk with a different backend (lance). Happy to receive help / contributions / ideas if you're interested!

Sorry, 2x faster than what?

My bad, the benchmarks are a work in progress, but if you hop over to lance-array repo, you'll see in the README that the p95 / p99 latencies for 5x5 or larger slices are ~2x faster for lance-array than zarr for the limite configurations I tried.