Date Tiered Compaction

[churromorales]

Description

As far as I can tell, Lucene doesn’t currently have a date‑tiered merge policy. I’m considering adding one to better support temporal queries over recent time windows.

On our larger clusters we see frequent paging of index data in and out of the cache. While most users write in temporal order—and TieredMergePolicy often produces a layout similar to what a date‑tiered approach would—it can still create segments with overlapping time ranges. I’m proposing a time‑aware merge policy, similar in spirit to the “date‑tiered compaction” described here: https://engineering.atspotify.com/2014/12/date-tiered-compaction/

I’d appreciate guidance on two points:

1. 1. Is this feasible and something the community would want?
2. Is it worthwhile, or do current features/configurations already achieve something similar?

If this is something interesting to the community, I am happy to write up a design doc and work on this task.

Thanks

[msfroh]

Have you tried using LogByteSizeMergePolicy? Unlike TieredMergePolicy, it will only merge adjacent segments, which tends to reduce time range overlap between segments.

That said, it sounds like the approach that you have in mind is much more deliberate and may offer benefit beyond what LogByteSizeMergePolicy can offer.

[churromorales]

Thank you for the pointer — this approach works better for us than TieredMergePolicy. However, we have a very large index (1PB+) in Solr, so we inevitably have far more data on disk than can fit in RAM, causing frequent major page faults.

We retain customer data for all time (10+ years), but most queries only cover the last 3 years, with a significant portion focusing on just the past 7 days. Our time dimension is a PointField, and from reviewing the code, it seems that lucene performs segment pruning when the query’s time range doesn’t overlap with a segment’s time range (given they are point fields, so you use the PointRangeQuery). This pruning provides a substantial performance improvement as it doesn't page in segments which are known not to have the data.

I’ve been considering a configuration that defines exponential time windows between the Unix epoch and now. These windows wouldn’t slide over time; instead, as time progresses, new windows would be added while older ones merge into larger intervals. And have a configuration to stop compacting our oldest segments altogether. I worked on this type of compaction in HBase which is also a LSM-Tree design and it worked out quite well for us in terms of reducing I/O considerably to do range queries between now and some point in the past.

I believe the main performance gain comes from avoiding the need to page segments in and out during recent range queries. While filtering itself is fast, it still incurs some memory overhead (which affects other queries) — and nothing beats simply not having to read unnecessary data at all. So while merging adjacent segments works, it might not lay out the data on disk as we would desire.

Also please correct me if I’m mistaken here — I’m still relatively new to Lucene/Solr/OpenSearch.

[msfroh]

Hmm... that is a neat idea! It's almost like a "LogTimeRangeMergePolicy".

I could definitely see some value in a timestamp-aware merge policy that merges older stuff into bigger segments while grouping newer stuff into smaller segments.

Another possible approach could involve rolling over and combining indexes. This approach is (I think) more common on OpenSearch/Elasticsearch. Basically, every day you roll over to a new index and the old index becomes read-only. While you might keep the daily indexes for the past seven days, maybe you combine the 7 days before that into a weekly index (and the 7 days before that, and the 7 days before that). Maybe combine some of your weekly indexes into monthly (or at least "4 weekly") indexes. In other words, solve the problem at the index level instead of at the segment level.

(Again, I also like the idea of solving it at the segment level -- I'm just pitching alternatives in case they're helpful.)