feat: /metrics

[eoghanlawless]

implementing /metrics endpoint to expose prometheus metrics

[adorney99]

not that familiar with promhttp syntax but lgtm if metrics are getting exported

[gcgirish]

1. Who is consuming these metrics?
2. Who defines what metrics should be collected and how are these metrics getting reported at system wide dashboard (if there is one).
3. How is this feature tested?
4. Is there a larger EMF wide feature that is dealing with scraping these metrics across various subsystems?

[eoghanlawless]

1. Who is consuming these metrics?
2. Who defines what metrics should be collected and how are these metrics getting reported at system wide dashboard (if there is one).
3. How is this feature tested?
4. Is there a larger EMF wide feature that is dealing with scraping these metrics across various subsystems?

1. We're creating a Grafana dashboard to monitor these metrics to help with scale testing.
2. We currently define the metrics and will add to them when needed, these metrics are exported to an orchestrator wide Grafana service (where we're creating a dashboard).
3. This feature has been manually tested E2E on a Coder environment by querying our exported metrics from the Grafana service (and manually scraping our /metrics endpoint).
4. I'm not sure what you mean here, but each service that wants to export metrics to Grafana needs to create a Prometheus ServiceMonitor resource that targets their metrics endpoint through their service.

[togashidm]

a few comments