proposal[PROM-60]: Prometheus CT Storage

Signed-off-by: bwplotka <[email protected]>

Author: bwplotka

proposals/0060-ct-storage.md

@@ -0,0 +1,137 @@
+****## Native TSDB Support for Cumulative Created Timestamp (CT) (and Delta Start Timestamp (ST) on the way)
+
+* **Owners:**
+  * [`@bwplotka`](https://github.com/bwplotka)
+  * <[delta-type-WG](https://docs.google.com/document/d/1G0d_cLHkgrnWhXYG9oXEmjy2qp6GLSX2kxYiurLYUSQ/edit) members?>
+
+* **Implementation Status:** `Partially implemented`
+
+* **Related Issues and PRs:**
+  * [WAL](https://github.com/prometheus/prometheus/issues/14218), [PRW2](https://github.com/prometheus/prometheus/issues/14220), [CT Meta](https://github.com/prometheus/prometheus/issues/14217).
+  * [appender](https://github.com/prometheus/prometheus/pull/17104)
+  * [initial attempt for ct per sample](https://github.com/prometheus/prometheus/pull/16046)
+  * [rw2 proto change for ct per sample](https://github.com/prometheus/prometheus/pull/17036)
+
+* **Other docs or links:**
+  * [PROM-29 (Created Timestamp)](https://github.com/prometheus/proposals/blob/main/proposals/0029-created-timestamp.md)
+  * [Delta type proposal](https://github.com/prometheus/proposals/pull/48), [Delta WG](https://docs.google.com/document/d/1G0d_cLHkgrnWhXYG9oXEmjy2qp6GLSX2kxYiurLYUSQ/edit)
+
+> TL;DR: We propose to extend Prometheus TSDB storage sample definition to include an extra int64 that will represent the cumulative created timestamp (CT) and, for the future delta temporality ([PROM-48](https://github.com/prometheus/proposals/pull/48)), a delta start timestamp (ST).
+> Once implemented, wee propose to deprecate the `created-timestamps-zero-injection` experimental feature.
+
+## Why
+
+The main goal of this proposal is to make sure [PROM-29's created timestamp (CT)](0029-created-timestamp.md) information is reliably and efficiently stored in Prometheus TSDB, so:
+
+* Written via TSDB Appender interfaces.
+* Query-able via TSDB Querier interfaces.
+* Persistent in WAL.
+* Watch-able (WAL) by Remote Writer.
+* (eventually) Persistent in TSDB block storage.
+
+To do it reliably, we propose to extend TSDB storage to "natively" support CT as something you can attach to a sample and use later on.
+Native CT support in Prometheus TSDB would unblock the practical use of CT information for:
+
+* Remote storages (Remote Write 2.0) (e.g. Otel, Chronosphere, Google)
+* PromQL and other read APIs (including federation) (e.g. increased cumulative based operation accuracy)
+
+Furthermore, it would unblock future Prometheus features for wider range of monitoring cases like:
+
+* Delta temporality support
+* UpAndDown counter (i.e. not monotonic counters) e.g. StatsD
+
+On top of that this allows to simplify some existing features e.g. detecting (exponential) native histogram resets (instead of reset hints)
+
+### Background: CT feature
+
+[PROM-29](0029-created-timestamp.md) introduced the "created timestamp" (CT) concept for Prometheus cumulative metrics. Semantically, CT represents the time when "counting" (from 0) started.
+In other words, CT is the time when the counter "instance" was created.
+
+Conceptually, CT extends the Prometheus data model for cumulative monotonic counters as follows:
+
+* (new) int64 Timestamp (CT): When counting started.
+* float64 or [Histogram](https://github.com/prometheus/prometheus/blob/d7e9a2ffb0f0ee0b6835cda6952d12ceee1371d0/model/histogram/histogram.go#L50) Value (V): The current value of the count, since the CT time.
+* int64 Timestamp (T): When this value was observed.
+* Labels: Unique identity of a series.
+  * This includes special metadata labels like: `__name__`, `__type__`, `__unit__`
+* Exemplars
+* Metadata
+
+Since the CT concept introduction in Prometheus we:
+
+* Extended Prometheus protobuf scrape format to include CT per each cumulative sample (TODO link).
+* Proposed (for OM 2) text format changes for CT scraping (improvement over existing OM1 `_created` lines) (TODO link).
+* Expanded Scrape parser interface to return `CreatedTimestamp` per sample (aka per line).
+* Optimized Protobuf and OpenMetrics parsers for CT use (TODO links).
+* Implemented an opt-in, experimental [`created-timestamps-zero-injection`](https://prometheus.io/docs/prometheus/latest/feature_flags/#created-timestamps-zero-injection) feature flag that injects fake sample (V: 0, T: CT).
+* Included CT in Remote Write 2 specification (TODO link).
+
+### Background: Delta temporality
+
+See the details, motivations and discussions about the delta temporality in [PROM-48](https://github.com/prometheus/proposals/pull/48).
+
+The core TL;DR relevant for this proposal is that the delta temporality counter sample can be conceptually seen as a "mini-cumulative counters". Essentially delta is a single-sample (value) cumulative counter for a period between (inclusive) start(ST)/create(CT) timestamp and a (end)timestamp.
+
+In other words, `increase(<counter>[5m])` produces a single delta sample for a `[t-5m, t]` period (V: `increase(<counter>[5m])`, CT/ST: `now()-5m`, T: `now()`).
+
+This proves that it's worth considering delta when desiging a CT feature support.
+
+### Background: CT (cumulative) vs ST (delta)
+
+[Previous section](#background-delta-temporality) argues that conceptually the Cumulative Created Timestamp (CT) and Delta Start Timestamp (ST) are essentially the same thing. This is why typically they are stored in the same "field" in other system APIs and storages (e.g. start time in OpenTelemetry TODO link).
+
+The notable difference when this special timestamp is used for cumulatives vs delta samples is the dynamicity **characteristics** of this timestamp.
+
+* For the cumulatives we expect CT to change on every new counter restart, so:
+  * Average: in the order of ~weeks/months for stable workloads, ~days/weeks for more dynamic environments (Kubernetes).
+  * Best case: it never changes (infinite count) e.g days_since_X_total.
+  * Worse case: it changes for every sample.
+* For the delta we expect CT to change for every sample.
+
+### Pitfalls of the current solution(s)
+
+* The `created-timestamps-zero-injection` feature allows some CT use cases, but it's limited in practice:
+  * It's stateful, which means it can't be used effectively across the ecosystem. Essentially you can't miss a single sample (and/or you have to process all samples since 0) to find CT information per sample. For example:
+    * Remote Write ingestion would need to be persistent and stateful, which blocks horizontal scalability of receiving.
+    * It limits effectiveness of using CT for PromQL operations like `rate`, `resets` etc.
+    * It makes "rolloup" (write time recording rules that pre-calculate rates) difficult to implement.
+  * Given immutability invariant (e.g. Prometheus), you can't effectively inject CT at a later time (out of order writes are sometimes possible, but expensive, especially for a single sample to be written in the past per series).
+  * It's prone to OOO false positives (we ignore this error for CTs now in Prometheus).
+  * It's producing an artificial sample, which looks like it was scraped.
+* We can't implement delta temporarily effectively.
+
+## Goals
+
+* [MUST] Prometheus can reliably store, query, ingest and export cumulative created timestamp (CT) information (long term plan for [PROM-29](https://github.com/prometheus/proposals/blob/main/proposals/0029-created-timestamp.md#:~:text=For%20those%20reasons%2C%20created%20timestamps%20will%20also%20be%20stored%20as%20metadata%20per%20series%2C%20following%20the%20similar%20logic%20used%20for%20the%20zero%2Dinjection.))
+* [SHOULD] Prometheus can reliably store, query, ingest and export delta start time information. This unblocks [PROM-48 delta proposal](https://github.com/prometheus/proposals/pull/48). Notably adding delta feature later on should ideally not require another complex storage design or implementation.
+* [SHOULD] Overhead of the solution should be minimal--initial overhead target set to maximum of 10% CPU, 10% of memory and 15% of disk space.
+
+## Non-Goals
+
+In this propose we don't want to:
+
+* Expand details on delta temporality. For this proposal it's enough to assume about delta, what's described in the [Background: Delta Temporality](#background-delta-temporality).
+* Expand or motivate the non-monotonic counter feature.
+
+## How
+
+TODO:
+* Describe touch points
+* Section for TSDB interfaces
+* Section for WAL changes + benchmark
+* Section for TSDB + benchmark
+* Section for PRW2 changes
+* Bonus: Should we rename CT to ST?
+* Expand on potential plan
+
+## Alternatives
+
+1. This is why not solution Z...
+
+## Action Plan
+
+The tasks to do in order to migrate to the new idea.
+
+* [ ] Task one
+
+* [ ] Task two