Skip to content

Implement manual stream trimming in Replay #241

@bwsw

Description

@bwsw

The problem: per-stream historical pruning is not available because RocksDB allows for a single TTL for the whole column family.

The solution: Add a feature to remove records in the background with the RESTful API. The feature allows users to set a default large TTL for all streams and manage stream trimming by launching background jobs with max:

  • stream names (list);
  • begin time;
  • end time;
  • records-per-second cap (max RPS).

RPS is used to avoid excessive contention. E.g., RPS set to 1800 allows cleaning for a maximum of 60 seconds every second the job runs for a 30FPS stream. The RPS is set for a job, not a stream.

Multiple jobs can work simultaneously if needed.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions