-
Notifications
You must be signed in to change notification settings - Fork 9
Open
Labels
Description
The problem: per-stream historical pruning is not available because RocksDB allows for a single TTL for the whole column family.
The solution: Add a feature to remove records in the background with the RESTful API. The feature allows users to set a default large TTL for all streams and manage stream trimming by launching background jobs with max:
- stream names (list);
- begin time;
- end time;
- records-per-second cap (max RPS).
RPS is used to avoid excessive contention. E.g., RPS set to 1800 allows cleaning for a maximum of 60 seconds every second the job runs for a 30FPS stream. The RPS is set for a job, not a stream.
Multiple jobs can work simultaneously if needed.