x509 Certificate Exporter Throttled – No Data in Grafana

[ssingh0158]

Hi Team,

x509 certificate exporter in k8s environment is experiencing severe client-side throttling when querying Kubernetes secrets, resulting in no data being displayed on the Grafana dashboard.

Issue Details:

Exporter logs show repeated messages like:

Waited for 2m8.271862912s due to client-side throttling, not priority and fairness, request: GET:https://100.65.0.1:443/api/v1/namespaces/app-def/secrets
Waited for 2m8.271862912s due to client-side throttling, not priority and fairness, request: GET:https://100.65.0.1:443/api/v1/namespaces/app/secrets

Grafana dashboards relying on cert metrics are blank or stale.

[cr7258]

@ssingh0158 You can adjust the parameters for burstQueries and queriesPerSecond based on the number of secrets being monitored.

[obeyler]

I think I've got the same issue no data in Grafana:

time=2025-06-23T13:30:06.625Z level=INFO msg="Discovered and parsed certificates" certificates=725 failures=0
time=2025-06-23T13:30:06.626Z level=INFO msg="Listening on" address=[::]:9793
time=2025-06-23T13:30:06.626Z level=INFO msg="TLS is disabled." http2=false address=[::]:9793
I0623 14:11:05.961439       1 request.go:729] Waited for 1.048117319s due to client-side throttling, not priority and fairness, request: GET:https://10.43.0.1:443/api/v1/namespaces
I0623 14:11:16.161335       1 request.go:729] Waited for 1.198736396s due to client-side throttling, not priority and fairness, request: GET:https://10.43.0.1:443/api/v1/namespaces/website-vcollastchancefixes/secrets
I0623 15:29:05.962474       1 request.go:729] Waited for 1.05465958s due to client-side throttling, not priority and fairness, request: GET:https://10.43.0.1:443/api/v1/namespaces
I0623 15:29:16.161306       1 request.go:729] Waited for 1.198678027s due to client-side throttling, not priority and fairness, request: GET:https://10.43.0.1:443/api/v1/namespaces/website-vca65-overview/secrets

Even if I activate the burstQueries and queriesPerSecond by the helm there are no data on grafana dashboard

 secretsExporter:
     kubeApiRateLimits:
         enabled: true

On the /metrics of exporter I can see that the metrics are present

...
x509_cert_not_before{issuer_C="US",issuer_CN="ISRG Root X1",issuer_O="Internet Security Research Group",secret_key="tls.crt",secret_name="website-tkb-staging.xx.net-tls",secret_namespace="website-tkb-card-preset",serial_number="184083759606652600789093070426744763640",subject_C="US",subject_CN="R11",subject_O="Let's Encrypt"} 1.710288e+09
x509_cert_not_before{issuer_C="US",issuer_CN="ISRG Root X1",issuer_O="Internet Security Research Group",secret_key="tls.crt",secret_name="website-tkb-staging.xx.net-tls",secret_namespace="website-tkb-carousel-videos",serial_number="100566080436859127493189557861323994576",subject_C="US",subject_CN="R10",subject_O="Let's Encrypt"} 1.710288e+09
x509_cert_not_before{issuer_C="US",issuer_CN="ISRG Root X1",issuer_O="Internet Security Research Group",secret_key="tls.crt",secret_name="website-tkb-staging.xx.net-tls",secret_namespace="website-tkb-ci-test",serial_number="184083759606652600789093070426744763640",subject_C="US",subject_CN="R11",subject_O="Let's Encrypt"} 1.710288e+09
...