Commit aaca61c
authored
refactor manifests to be lazier (#173)
Testing scaling to 10 billion points.
<!-- ELLIPSIS_HIDDEN -->
----
> [!IMPORTANT]
> Refactor Deepscatter's tile manifest handling to be lazier, improving scalability and performance for large datasets.
>
> - **Behavior**:
> - Refactor `Deeptable` and `Tile` classes to handle tile manifests lazily, improving scalability for large datasets.
> - Update `Deeptable` to use `flatManifest` instead of nested `TileManifest`.
> - Remove `points()` method from `Deeptable`.
> - **Tile Management**:
> - Modify `Tile` class to use `flatManifest` for metadata and child tile creation.
> - Add `createChildren()` method to `Tile` for lazy child creation.
> - Update `populateManifest()` and `deriveManifestInfoFromTileMetadata()` in `Tile`.
> - **Testing**:
> - Add `tix.spec.ts` to test `tixToChildren`, `tixToZxy`, and `parentTix` functions.
> - **Dependencies**:
> - Update `quadfeather` dependency in `uv.lock` to version `2.0.0rc1`.
>
> <sup>This description was created by </sup>[<img alt="Ellipsis" src="https://img.shields.io/badge/Ellipsis-blue?color=175173">](https://www.ellipsis.dev?ref=nomic-ai%2Fdeepscatter&utm_source=github&utm_medium=referral)<sup> for eec66a0. You can [customize](https://app.ellipsis.dev/nomic-ai/settings/summaries) this summary. It will automatically update as commits are pushed.</sup>
<!-- ELLIPSIS_HIDDEN -->1 parent facb850 commit aaca61c
File tree
20 files changed
+931
-317
lines changed- dev
- src
- tests
20 files changed
+931
-317
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
31 | 31 | | |
32 | 32 | | |
33 | 33 | | |
| 34 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
11 | | - | |
12 | | - | |
| 11 | + | |
| 12 | + | |
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| 10 | + | |
10 | 11 | | |
11 | 12 | | |
12 | 13 | | |
13 | 14 | | |
14 | 15 | | |
15 | 16 | | |
| 17 | + | |
16 | 18 | | |
17 | 19 | | |
18 | 20 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
90 | 90 | | |
91 | 91 | | |
92 | 92 | | |
93 | | - | |
94 | 93 | | |
95 | 94 | | |
96 | 95 | | |
| |||
175 | 174 | | |
176 | 175 | | |
177 | 176 | | |
178 | | - | |
179 | 177 | | |
180 | 178 | | |
181 | 179 | | |
| |||
0 commit comments