Installation fails on DO managed k8s #3939
Description
Hi, I followed the setup guide here and I'm getting these errors https://us5.datadoghq.com/signup/agent#kubernetes:
2024-10-13 15:55:55 UTC | CORE | ERROR | (pkg/collector/worker/check_logger.go:71 in Error) | check:cilium | Error running check: [{"message":"HTTPConnectionPool(host='10.138.32.241', port=9962): Max retries exceeded with url: /metrics (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1cdeea14d0>: Failed to establish a new connection: [Errno 111] Connection refused'))","traceback":"Traceback (most recent call last):\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/urllib3/connection.py\", line 196, in _new_conn\n sock = connection.create_connection(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/urllib3/util/connection.py\", line 85, in create_connection\n raise err\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/urllib3/util/connection.py\", line 73, in create_connection\n sock.connect(sa)\nConnectionRefusedError: [Errno 111] Connection refused\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/urllib3/connectionpool.py\", line 789, in urlopen\n response = self._make_request(\n ^^^^^^^^^^^^^^^^^^^\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/urllib3/connectionpool.py\", line 495, in _make_request\n conn.request(\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/urllib3/connection.py\", line 398, in request\n self.endheaders()\n File \"/opt/datadog-agent/embedded/lib/python3.11/http/client.py\", line 1293, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/opt/datadog-agent/embedded/lib/python3.11/http/client.py\", line 1052, in _send_output\n self.send(msg)\n File \"/opt/datadog-agent/embedded/lib/python3.11/http/client.py\", line 990, in send\n self.connect()\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/urllib3/connection.py\", line 236, in connect\n self.sock = self._new_conn()\n ^^^^^^^^^^^^^^^^\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/urllib3/connection.py\", line 211, in _new_conn\n raise NewConnectionError(\nurllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f1cdeea14d0>: Failed to establish a new connection: [Errno 111] Connection refused\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/requests/adapters.py\", line 667, in send\n resp = conn.urlopen(\n ^^^^^^^^^^^^^\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/urllib3/connectionpool.py\", line 843, in urlopen\n retries = retries.increment(\n ^^^^^^^^^^^^^^^^^^\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/urllib3/util/retry.py\", line 519, in increment\n raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nurllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='10.138.32.241', port=9962): Max retries exceeded with url: /metrics (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1cdeea14d0>: Failed to establish a new connection: [Errno 111] Connection refused'))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/base/checks/base.py\", line 1313, in run\n self.check(instance)\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/base/checks/openmetrics/base_check.py\", line 142, in check\n self.process(scraper_config)\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/base/checks/openmetrics/mixins.py\", line 578, in process\n for metric in self.scrape_metrics(scraper_config):\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/base/checks/openmetrics/mixins.py\", line 505, in scrape_metrics\n response = self.poll(scraper_config)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/base/checks/openmetrics/mixins.py\", line 842, in poll\n response = self.send_request(endpoint, scraper_config, headers)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/base/checks/openmetrics/mixins.py\", line 868, in send_request\n return http_handler.get(endpoint, stream=True, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/base/utils/http.py\", line 352, in get\n return self._request('get', url, options)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/base/utils/http.py\", line 416, in _request\n response = self.make_request_aia_chasing(request_method, method, url, new_options, persist)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/base/utils/http.py\", line 422, in make_request_aia_chasing\n response = request_method(url, **new_options)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/requests/api.py\", line 73, in get\n return request(\"get\", url, params=params, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/requests/api.py\", line 59, in request\n return session.request(method=method, url=url, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/requests/sessions.py\", line 589, in request\n resp = self.send(prep, **send_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/requests/sessions.py\", line 703, in send\n r = adapter.send(request, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/requests/adapters.py\", line 700, in send\n raise ConnectionError(e, request=request)\nrequests.exceptions.ConnectionError: HTTPConnectionPool(host='10.138.32.241', port=9962): Max retries exceeded with url: /metrics (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1cdeea14d0>: Failed to establish a new connection: [Errno 111] Connection refused'))\n"}]
2024-10-13 15:56:05 UTC | CORE | WARN | (pkg/logs/client/http/destination.go:336 in unconditionalSend) | failed to post http payload. code=403 host=contlcycle-intake.datadoghq.com response={"errors":[{"status":"403","title":"Forbidden","detail":"Api key is either forbidden or blacklisted"}]}
2024-10-13 15:56:05 UTC | CORE | WARN | (pkg/logs/client/http/destination.go:245 in sendAndRetry) | Could not send payload: client error
2024-10-13 15:56:10 UTC | CORE | ERROR | (pkg/collector/worker/check_logger.go:71 in Error) | check:cilium | Error running check: [{"message":"HTTPConnectionPool(host='10.138.32.241', port=9962): Max retries exceeded with url: /metrics (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1cdee8e810>: Failed to establish a new connection: [Errno 111] Connection refused'))","traceback":"Traceback (most recent call last):\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/urllib3/connection.py\", line 196, in _new_conn\n sock = connection.create_connection(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/urllib3/util/connection.py\", line 85, in create_connection\n raise err\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/urllib3/util/connection.py\", line 73, in create_connection\n sock.connect(sa)\nConnectionRefusedError: [Errno 111] Connection refused\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/urllib3/connectionpool.py\", line 789, in urlopen\n response = self._make_request(\n ^^^^^^^^^^^^^^^^^^^\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/urllib3/connectionpool.py\", line 495, in _make_request\n conn.request(\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/urllib3/connection.py\", line 398, in request\n self.endheaders()\n File \"/opt/datadog-agent/embedded/lib/python3.11/http/client.py\", line 1293, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/opt/datadog-agent/embedded/lib/python3.11/http/client.py\", line 1052, in _send_output\n self.send(msg)\n File \"/opt/datadog-agent/embedded/lib/python3.11/http/client.py\", line 990, in send\n self.connect()\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/urllib3/connection.py\", line 236, in connect\n self.sock = self._new_conn()\n ^^^^^^^^^^^^^^^^\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/urllib3/connection.py\", line 211, in _new_conn\n raise NewConnectionError(\nurllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f1cdee8e810>: Failed to establish a new connection: [Errno 111] Connection refused\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/requests/adapters.py\", line 667, in send\n resp = conn.urlopen(\n ^^^^^^^^^^^^^\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/urllib3/connectionpool.py\", line 843, in urlopen\n retries = retries.increment(\n ^^^^^^^^^^^^^^^^^^\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/urllib3/util/retry.py\", line 519, in increment\n raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nurllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='10.138.32.241', port=9962): Max retries exceeded with url: /metrics (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1cdee8e810>: Failed to establish a new connection: [Errno 111] Connection refused'))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/base/checks/base.py\", line 1313, in run\n self.check(instance)\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/base/checks/openmetrics/base_check.py\", line 142, in check\n self.process(scraper_config)\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/base/checks/openmetrics/mixins.py\", line 578, in process\n for metric in self.scrape_metrics(scraper_config):\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/base/checks/openmetrics/mixins.py\", line 505, in scrape_metrics\n response = self.poll(scraper_config)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/base/checks/openmetrics/mixins.py\", line 842, in poll\n response = self.send_request(endpoint, scraper_config, headers)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/base/checks/openmetrics/mixins.py\", line 868, in send_request\n return http_handler.get(endpoint, stream=True, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/base/utils/http.py\", line 352, in get\n return self._request('get', url, options)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/base/utils/http.py\", line 416, in _request\n response = self.make_request_aia_chasing(request_method, method, url, new_options, persist)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/base/utils/http.py\", line 422, in make_request_aia_chasing\n response = request_method(url, **new_options)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/requests/api.py\", line 73, in get\n return request(\"get\", url, params=params, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/requests/api.py\", line 59, in request\n return session.request(method=method, url=url, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/requests/sessions.py\", line 589, in request\n resp = self.send(prep, **send_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/requests/sessions.py\", line 703, in send\n r = adapter.send(request, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/opt/datadog-agent/embedded/lib/python3.11/site-packages/requests/adapters.py\", line 700, in send\n raise ConnectionError(e, request=request)\nrequests.exceptions.ConnectionError: HTTPConnectionPool(host='10.138.32.241', port=9962): Max retries exceeded with url: /metrics (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1cdee8e810>: Failed to establish a new connection: [Errno 111] Connection refused'))\n"}]
I use DD for work profesionally and was going to use it for a hobby project on Digital Ocean, but I'm not sure if it's unsupported? Anyway, I will probably be uninstalling it since I was hoping it would be easier than using Grafana's OS solutions (since those are entirely free), but it doesn't look like it.