Prometheusremotewrite stops receiving data

[pavolloffay]

Component(s)

receiver/prometheusremotewrite

What happened?

Description

The

Steps to Reproduce

1. start prometheus
2. strart OTEL col with enabled prom RW and debug exporter
3. configure prometheus to RW to the collector

The prometheus config

    convertClassicHistogramsToNHCB: true
    enableFeatures:
      - otlp-write-receiver
      - metadata-wal-records
      - native-histograms
    remoteWrite:
      - messageVersion: V2.0
        sendExemplars: true
        sendNativeHistograms: true
        url: http://otel-collector.observability.svc.cluster.local:9090/api/v1/write

After some time collector logs contain only

{"level":"info","ts":"2025-10-17T15:44:56.308Z","msg":"","resource":{"service.instance.id":"4e53c5a4-fe85-4ca0-a554-27111e9df3cf","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"debug","otelcol.component.kind":"exporter","otelcol.signal":"metrics"}
{"level":"info","ts":"2025-10-17T15:44:56.757Z","msg":"Metrics","resource":{"service.instance.id":"4e53c5a4-fe85-4ca0-a554-27111e9df3cf","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"debug","otelcol.component.kind":"exporter","otelcol.signal":"metrics","resource metrics":0,"metrics":0,"data points":0}
{"level":"info","ts":"2025-10-17T15:44:56.757Z","msg":"","resource":{"service.instance.id":"4e53c5a4-fe85-4ca0-a554-27111e9df3cf","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"debug","otelcol.component.kind":"exporter","otelcol.signal":"metrics"}
{"level":"info","ts":"2025-10-17T15:44:56.757Z","msg":"Metrics","resource":{"service.instance.id":"4e53c5a4-fe85-4ca0-a554-27111e9df3cf","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"debug","otelcol.component.kind":"exporter","otelcol.signal":"metrics","resource metrics":0,"metrics":0,"data points":0}
{"level":"info","ts":"2025-10-17T15:44:56.757Z","msg":"","resource":{"service.instance.id":"4e53c5a4-fe85-4ca0-a554-27111e9df3cf","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"debug","otelcol.component.kind":"exporter","otelcol.signal":"metrics"}
{"level":"info","ts":"2025-10-17T15:44:57.136Z","msg":"Metrics","resource":{"service.instance.id":"4e53c5a4-fe85-4ca0-a554-27111e9df3cf","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"debug","otelcol.component.kind":"exporter","otelcol.signal":"metrics","resource metrics":0,"metrics":0,"data points":0}
{"level":"info","ts":"2025-10-17T15:44:57.136Z","msg":"","resource":{"service.instance.id":"4e53c5a4-fe85-4ca0-a554-27111e9df3cf","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"debug","otelcol.component.kind":"exporter","otelcol.signal":"metrics"}
{"level":"info","ts":"2025-10-17T15:44:57.159Z","msg":"Metrics","resource":{"service.instance.id":"4e53c5a4-fe85-4ca0-a554-27111e9df3cf","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"debug","otelcol.component.kind":"exporter","otelcol.signal":"metrics","resource metrics":0,"metrics":0,"data points":0}
{"level":"info","ts":"2025-10-17T15:44:57.159Z","msg":"","resource":{"service.instance.id":"4e53c5a4-fe85-4ca0-a554-27111e9df3cf","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"debug","otelcol.component.kind":"exporter","otelcol.signal":"metrics"}
{"level":"info","ts":"2025-10-17T15:44:57.230Z","msg":"Metrics","resource":{"service.instance.id":"4e53c5a4-fe85-4ca0-a554-27111e9df3cf","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"debug","otelcol.component.kind":"exporter","otelcol.signal":"metrics","resource metrics":0,"metrics":0,"data points":0}
{"level":"info","ts":"2025-10-17T15:44:57.230Z","msg":"","resource":{"service.instance.id":"4e53c5a4-fe85-4ca0-a554-27111e9df3cf","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"debug","otelcol.component.kind":"exporter","otelcol.signal":"metrics"}
{"level":"info","ts":"2025-10-17T15:44:57.234Z","msg":"Metrics","resource":{"service.instance.id":"4e53c5a4-fe85-4ca0-a554-27111e9df3cf","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"debug","otelcol.component.kind":"exporter","otelcol.signal":"metrics","resource metrics":0,"metrics":0,"data points":0}
{"level":"info","ts":"2025-10-17T15:44:57.251Z","msg":"","resource":{"service.instance.id":"4e53c5a4-fe85-4ca0-a554-27111e9df3cf","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"debug","otelcol.component.kind":"exporter","otelcol.signal":"metrics"}
{"level":"info","ts":"2025-10-17T15:44:57.756Z","msg":"Metrics","resource":{"service.instance.id":"4e53c5a4-fe85-4ca0-a554-27111e9df3cf","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"debug","otelcol.component.kind":"exporter","otelcol.signal":"metrics","resource metrics":0,"metrics":0,"data points":0}
{"level":"info","ts":"2025-10-17T15:44:57.756Z","msg":"","resource":{"service.instance.id":"4e53c5a4-fe85-4ca0-a554-27111e9df3cf","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"debug","otelcol.component.kind":"exporter","otelcol.signal":"metrics"}
{"level":"info","ts":"2025-10-17T15:44:57.748Z","msg":"Metrics","resource":{"service.instance.id":"4e53c5a4-fe85-4ca0-a554-27111e9df3cf","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"debug","otelcol.component.kind":"exporter","otelcol.signal":"metrics","resource metrics":0,"metrics":0,"data points":0}
{"level":"info","ts":"2025-10-17T15:44:57.762Z","msg":"","resource":{"service.instance.id":"4e53c5a4-fe85-4ca0-a554-27111e9df3cf","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"debug","otelcol.component.kind":"exporter","otelcol.signal":"metrics"}

Prometheus shows this logs

time=2025-10-16T13:32:17.522Z level=ERROR source=queue_manager.go:1670 msg="we got 2xx status code from the Receiver yet statistics indicate some data was not written; investigation needed" component=remote remote_name=867337 url=http://otel-collector.observability.svc.cluster.local:9090/api/v1/write failedSampleCount=110 failedHistogramCount=11 failedExemplarCount=0

Expected Result

Actual Result

Collector version

0.137.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

config:
    receivers:
      prometheusremotewrite:
        endpoint: 0.0.0.0:9090

    processors:

    exporters:
      debug:
        # basic, normal, detailed
        verbosity: normal

    service:
      telemetry:
        logs:
          level: "INFO"
          development: true
          encoding: "json"
        metrics:
      pipelines:
        metrics:
          receivers: [prometheusremotewrite]
          processors: []
          exporters:
            - debug

Log output

{"level":"info","ts":"2025-10-17T15:44:56.308Z","msg":"","resource":{"service.instance.id":"4e53c5a4-fe85-4ca0-a554-27111e9df3cf","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"debug","otelcol.component.kind":"exporter","otelcol.signal":"metrics"}
{"level":"info","ts":"2025-10-17T15:44:56.757Z","msg":"Metrics","resource":{"service.instance.id":"4e53c5a4-fe85-4ca0-a554-27111e9df3cf","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"debug","otelcol.component.kind":"exporter","otelcol.signal":"metrics","resource metrics":0,"metrics":0,"data points":0}
{"level":"info","ts":"2025-10-17T15:44:56.757Z","msg":"","resource":{"service.instance.id":"4e53c5a4-fe85-4ca0-a554-27111e9df3cf","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"debug","otelcol.component.kind":"exporter","otelcol.signal":"metrics"}
{"level":"info","ts":"2025-10-17T15:44:56.757Z","msg":"Metrics","resource":{"service.instance.id":"4e53c5a4-fe85-4ca0-a554-27111e9df3cf","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"debug","otelcol.component.kind":"exporter","otelcol.signal":"metrics","resource metrics":0,"metrics":0,"data points":0}
{"level":"info","ts":"2025-10-17T15:44:56.757Z","msg":"","resource":{"service.instance.id":"4e53c5a4-fe85-4ca0-a554-27111e9df3cf","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"debug","otelcol.component.kind":"exporter","otelcol.signal":"metrics"}
{"level":"info","ts":"2025-10-17T15:44:57.136Z","msg":"Metrics","resource":{"service.instance.id":"4e53c5a4-fe85-4ca0-a554-27111e9df3cf","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"debug","otelcol.component.kind":"exporter","otelcol.signal":"metrics","resource metrics":0,"metrics":0,"data points":0}
{"level":"info","ts":"2025-10-17T15:44:57.136Z","msg":"","resource":{"service.instance.id":"4e53c5a4-fe85-4ca0-a554-27111e9df3cf","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"debug","otelcol.component.kind":"exporter","otelcol.signal":"metrics"}
{"level":"info","ts":"2025-10-17T15:44:57.159Z","msg":"Metrics","resource":{"service.instance.id":"4e53c5a4-fe85-4ca0-a554-27111e9df3cf","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"debug","otelcol.component.kind":"exporter","otelcol.signal":"metrics","resource metrics":0,"metrics":0,"data points":0}
{"level":"info","ts":"2025-10-17T15:44:57.159Z","msg":"","resource":{"service.instance.id":"4e53c5a4-fe85-4ca0-a554-27111e9df3cf","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"debug","otelcol.component.kind":"exporter","otelcol.signal":"metrics"}
{"level":"info","ts":"2025-10-17T15:44:57.230Z","msg":"Metrics","resource":{"service.instance.id":"4e53c5a4-fe85-4ca0-a554-27111e9df3cf","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"debug","otelcol.component.kind":"exporter","otelcol.signal":"metrics","resource metrics":0,"metrics":0,"data points":0}
{"level":"info","ts":"2025-10-17T15:44:57.230Z","msg":"","resource":{"service.instance.id":"4e53c5a4-fe85-4ca0-a554-27111e9df3cf","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"debug","otelcol.component.kind":"exporter","otelcol.signal":"metrics"}
{"level":"info","ts":"2025-10-17T15:44:57.234Z","msg":"Metrics","resource":{"service.instance.id":"4e53c5a4-fe85-4ca0-a554-27111e9df3cf","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"debug","otelcol.component.kind":"exporter","otelcol.signal":"metrics","resource metrics":0,"metrics":0,"data points":0}
{"level":"info","ts":"2025-10-17T15:44:57.251Z","msg":"","resource":{"service.instance.id":"4e53c5a4-fe85-4ca0-a554-27111e9df3cf","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"debug","otelcol.component.kind":"exporter","otelcol.signal":"metrics"}
{"level":"info","ts":"2025-10-17T15:44:57.756Z","msg":"Metrics","resource":{"service.instance.id":"4e53c5a4-fe85-4ca0-a554-27111e9df3cf","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"debug","otelcol.component.kind":"exporter","otelcol.signal":"metrics","resource metrics":0,"metrics":0,"data points":0}
{"level":"info","ts":"2025-10-17T15:44:57.756Z","msg":"","resource":{"service.instance.id":"4e53c5a4-fe85-4ca0-a554-27111e9df3cf","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"debug","otelcol.component.kind":"exporter","otelcol.signal":"metrics"}
{"level":"info","ts":"2025-10-17T15:44:57.748Z","msg":"Metrics","resource":{"service.instance.id":"4e53c5a4-fe85-4ca0-a554-27111e9df3cf","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"debug","otelcol.component.kind":"exporter","otelcol.signal":"metrics","resource metrics":0,"metrics":0,"data points":0}
{"level":"info","ts":"2025-10-17T15:44:57.762Z","msg":"","resource":{"service.instance.id":"4e53c5a4-fe85-4ca0-a554-27111e9df3cf","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"debug","otelcol.component.kind":"exporter","otelcol.signal":"metrics"}

Additional context

No response

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[github-actions]

Pinging code owners:

• receiver/prometheusremotewrite: @dashpole @ArthurSens @perebaj

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[WhiteDiamondz]

I have faced a similar problem where metrics are no longer received exactly after 5minutes from the first serie received.

Using the debug exporter I was able to confirm that they are no longer exported and seem to be dropped in the receiver's stage. Same behavior with different prometheus versions.

Prometheus config for the remote_write:

remote_write:
  - url: https://collector.example.com/api/v1/write
    protobuf_message: io.prometheus.write.v2.Request
    tls_config:
      ca_file: /etc/ssl/certs/ca-certificates.crt
      key_file: /data/certs/example.key.pem
      cert_file: /data/certs/example.cert.pem
    name: edge-prometheus

[constanca-m]

This definitely seems like an issue in the receiver. Please @dashpole @ArthurSens @perebaj, can you take a look at this? I am not sure if the issue itself will be on the receiver, but I am not familiar enough with this component to know

[ethuleau]

I think it is related to #42159