S3 next level: offload, backup plus also data-sharing, querying?

[hpvd]

There are several discussions/issues about utilizing S3 for nats/jetstream

• Support S3 API for JetStream #5486
• S3 Compatibility for NATS Object Store #4871

=> What do you think of taking S3 -support directly to the next level,
by making early design choices to utilize it for
offload and backup
but also for data-sharing and querying?

Data-sharing and quering (3rd party usage) could be made possible by

1. storing messages in S3 directly in the standard open tablet format
   (-> parquet, see https://duckdb.org/2025/01/22/parquet-encodings.html)

2. one could read these parquet files from multiple S3 buckets e.g. via duckdb
   https://duckdb.org/docs/guides/network_cloud_storage/s3_import.html

3. easily combine them e.g. using DuckDB
   https://duckdb.org/docs/data/multiple_files/overview.html#parquet

4. combining different schemas by position or name if needed
   https://duckdb.org/docs/data/multiple_files/combining_schemas

5. and security could be addressed when using standard parquet encryption (Parquet Modular Encryption) specification
   https://duckdb.org/docs/data/parquet/encryption.html

raw idea inspired by
https://streamnative.io/blog/stream-table-duality-and-the-vision-of-enabling-data-sharing

[ABFocke]

I really like the idea.
would be something similar to warpstream.

[hpvd]

if you like it, plz up-vote this topic :-)

[hpvd]

from #4871 (comment)

Most general approach (basic)

config:
for every subject, no matter of its use case (standard, system, KV, Object…)

connected external storage

• S3
• ...

S3-storage

• bucket name
• region
• ....

mode

• duplicate
• offload

offload threshold

• message age > [0..] seconds
• message quantity > [0..]
• subject size > [0.. ] byte

allowing to start with "0" may enable use-cases purely relying on S3 (for this subject)

[hpvd]

allowing to start with "0" may enable use-cases purely relying on S3 (for this subject)

-> for all use-cases where performance is not important but cost have to be as low as possible...

[ohanf]

allowing to start with "0" may enable use-cases purely relying on S3 (for this subject)

-> for all use-cases where performance is not important but cost have to be as low as possible...

I would not start with 0, especially if you are assuming it will be cheaper. Object storage (ie S3) is not a file system, notably you can not "append" to files once they have been uploaded. Providers also charge per operation, listing in particular can become quite expensive if you are not careful.

The overhead of writing a new file for every message would be non-viable: tons of tiny files, amplification of API operations (direct impact to cost) and writing parquet headers. Some amount of buffering would need to be done before writing, and you may want to consider "compaction" jobs even after files have been uploaded to further increase density and thus storage+cost efficiency. Using S3 for storage is becoming increasingly popular for live systems, especially in the obserability and ETL spaces, and they have all run into these problems and have similar solutions.

Look into technologies like Grafana's mimir and loki which are good Go based examples of storing live data in object storage. There is also the fledgling InfluxDB 3.0 (written in Rust) which has a similar approach. I believe there is even literature around how some Kafka connectors handle collecting messages into parquet and writing to S3 (although I don't have the link handy). I don't know how Kafka (and maybe RedPanda) does their own offloading, but that could be something to reference as well as I believe it has first class support.

[hpvd]

@ohanf many thanks for sharing this valuable knowledge incl examples and hints where to look into!
So far I have only experience using S3 for backups and file exchange...

[Material-Scientist]

allowing to start with "0" may enable use-cases purely relying on S3 (for this subject)

-> for all use-cases where performance is not important but cost have to be as low as possible...

I would not start with 0, especially if you are assuming it will be cheaper. Object storage (ie S3) is not a file system, notably you can not "append" to files once they have been uploaded. Providers also charge per operation, listing in particular can become quite expensive if you are not careful.

The overhead of writing a new file for every message would be non-viable: tons of tiny files, amplification of API operations (direct impact to cost) and writing parquet headers. Some amount of buffering would need to be done before writing, and you may want to consider "compaction" jobs even after files have been uploaded to further increase density and thus storage+cost efficiency. Using S3 for storage is becoming increasingly popular for live systems, especially in the obserability and ETL spaces, and they have all run into these problems and have similar solutions.

Look into technologies like Grafana's mimir and loki which are good Go based examples of storing live data in object storage. There is also the fledgling InfluxDB 3.0 (written in Rust) which has a similar approach. I believe there is even literature around how some Kafka connectors handle collecting messages into parquet and writing to S3 (although I don't have the link handy). I don't know how Kafka (and maybe RedPanda) does their own offloading, but that could be something to reference as well as I believe it has first class support.

💯 this

Object stores are designed for infrequent writes, given how each write creates a new object. Also, most object stores (especially minio-based) perform poorly when dealing with many small files (SeaweedFS being the exception).

A potential alternative solution is to have NATS write data to an rclone mount with S3-compatible backend. However I ran into some issues with that. Here are some of my notes:

The idea centers around the --vfs-write-back flag in rclone.

If you mount an rclone volume with an S3-compatible backend onto your host , and have NATS use that as primary directory to write data to, you can set e.g. --vfs-write-back=10m on the rclone mount, such that any data will get transferred to the S3 backend after 10min of not having been accessed. This has several important benefits:

• no change necessary on NATS side, since NATS still sees the stub-files and fetches remote data on-demand
• you don't write to S3 on every new message (i.e. messages are buffered on the local disk until the 8MB block is full, and hasn't been accessed/updated in 10min)
• if the mount dies & restarts, the data is still available locally and the writeback to S3 will happen on the next sync

Here's an example that this actually works:

NATS has 12MB on the bsky stream, but only the first 8MB block (1.blk) has been transferred to the remote S3 backend. The 2.blk has not yet been transferred because NATS is still writing to it.

The remote backend even respects NATS retention limits. E.g. if NATS deletes a local block stub file, it also gets removed from the S3 store.

E.g. here blocks 1-8 have been deleted from the remote store:

However, after a while I kept getting these errors:

69 & 72 are missing, which have errors in the logs: corrupted on transfer

That's where I stopped with this experiment. Maybe someone else knows how to resolve this, as it could enable S3-backed storage without requiring any changes to NATS.

CC sethitow