caller=client.go:360 logs_config=integrations component=client host=logs-prod3.grafana.net: final error sending batch: server returned HTTP status 400 Bad Request (400): has 16 label names; limit 15\"}"

[marcellodesales]

🧧 Problem: Agent can't forward logs to granafa.net: has 16 label names; limit 15

• All DaemonSet pods can't forward logs to granafa.net from the hosts

ts=2022-04-23T20:02:20.168115648Z caller=client.go:360 level=error component=logs 
logs_config=integrations component=client host=logs-prod3.grafana.net 
msg="final error sending batch" status=400 error="server returned HTTP status 400 Bad Request (400): 
{\"code\":400,\"status\":\"error\",\"message\":\"entry for stream 
'{app_super_cash_platform=\\\"observability\\\", cloud=\\\"aws\\\", cluster=\\\"cloud\\\", 
container=\\\"grafana-agent-logs\\\", controller_revision_hash=\\\"664bbd7cf8\\\", 
env=\\\"prd\\\", filename=\\\"/var/log/pods/supercash-observability-platform-aws-sae1-prdt-prd-prd_grafana-agent-logs-9r2tz_3363202b-c199-4b84-bc5a-dfcb0b0225d6/grafana-agent-logs/0.log\\\", 
job=\\\"supercash-observability-platform-aws-sae1-prdt-prd-prd/grafana-agent-logs-9r2tz\\\", 
namespace=\\\"supercash-observability-platform-aws-sae1-prdt-prd-prd\\\", 
pod=\\\"grafana-agent-logs-9r2tz\\\", pod_template_generation=\\\"1\\\", 
product=\\\"grafana-cloud\\\", region=\\\"sae1\\\", segment=\\\"prd\\\", stream=\\\"stderr\\\", 
type=\\\"prdt\\\"}' has 16 label names; limit 15\"}"

❓ Question

• How to increase the limit of the agent by 1???
  • I can't change the original labels as they are used for specific purposes...

[ivankovnatsky]

Also bumped into this after installing istio. Istio's sidecar proxy added another labels to a pod:

│               security.istio.io/tlsMode=istio                                                                                                                              
│               service.istio.io/canonical-name=kubernetes-dashboard                                                                                                         
│               service.istio.io/canonical-revision=2.4.0  

[rfratto]

Hi! This repository is unfortunately just dedicated to Grafana Agent as an open source project, and we don't offer assistance with Grafana Cloud here; you'd have to open up a support ticket in the Grafana portal to get in touch with someone for this.

[ivankovnatsky]

@marcellodesales I fixed it by droplabel in promtail configs. I don't think you need this dimenstion/label: pod_template_generation. Reference https://grafana.com/docs/loki/latest/clients/promtail/stages/labeldrop/.

[marcellodesales]

@ivankovnatsky I'm sorry I missed your msg... where exactly can I find this config?

[ivankovnatsky]

Sure, we use terraform and helm provider to deploy promtail:

resource "helm_release" "promtail" {
  repository = "https://grafana.github.io/helm-charts"
  name       = "promtail"
  chart      = "promtail"
  namespace  = kubernetes_namespace.monitoring.metadata[0].name
  version    = "2.2.0"
  values     = [file("./promtail-values.yaml")]
}

# promtail-values.yaml:
...
pipelineStages:
  - cri: {}
  - json:
      source: output
      expressions:
        level: level
        fields: fields
        timestamp: timestamp
  - labeldrop:
      - stream
      - profile
      - product
      - pod_template_generation
      - security_istio_io_tlsMode
      - service_istio_io_canonical_name
      - service_istio_io_canonical_revision

I think you can adapt this to your promtail deployment.

[xcrypt0surf3r]

@ivankovnatsky sounds good... if anyone else is interested, here's a link to the config promtail-values.yaml

[xcrypt0surf3r]

Hope to verify soon... Thank you!

[sjk7524068]

@xcrypt0surf3r This promtail-values link works for me