[systemd] opening of /run/systemd/private causes errors in system journal

[prophile]

Output of the info page

$ sudo systemctl status datadog-agent.service
● datadog-agent.service - Datadog Agent
   Loaded: loaded (/lib/systemd/system/datadog-agent.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-10-22 17:09:56 BST; 53min ago
 Main PID: 1977 (agent)
    Tasks: 19 (limit: 4915)
   CGroup: /system.slice/datadog-agent.service
           └─1977 /opt/datadog-agent/bin/agent/agent run -p /opt/datadog-agent/run/agent.pid

Oct 22 17:40:00 beckham.adm.thread.com agent[1977]: 2019-10-22 17:40:00 BST | CORE | INFO | (pkg/serializer/serializer.go:274 in SendMetadata) | Sent host metadata payload, size (ra
Oct 22 17:40:00 beckham.adm.thread.com agent[1977]: 2019-10-22 17:40:00 BST | CORE | INFO | (pkg/serializer/serializer.go:294 in SendJSONToV1Intake) | Sent processes metadata payloa
Oct 22 17:40:01 beckham.adm.thread.com agent[1977]: 2019-10-22 17:40:01 BST | CORE | INFO | (pkg/metadata/host/host.go:152 in getNetworkMeta) | could not get network metadata: could
Oct 22 17:40:02 beckham.adm.thread.com agent[1977]: 2019-10-22 17:40:02 BST | CORE | INFO | (pkg/serializer/serializer.go:274 in SendMetadata) | Sent host metadata payload, size (ra
Oct 22 17:45:00 beckham.adm.thread.com agent[1977]: 2019-10-22 17:45:00 BST | CORE | INFO | (pkg/serializer/serializer.go:294 in SendJSONToV1Intake) | Sent processes metadata payloa
Oct 22 17:50:00 beckham.adm.thread.com agent[1977]: 2019-10-22 17:50:00 BST | CORE | INFO | (pkg/serializer/serializer.go:274 in SendMetadata) | Sent host metadata payload, size (ra
Oct 22 17:50:00 beckham.adm.thread.com agent[1977]: 2019-10-22 17:50:00 BST | CORE | INFO | (pkg/serializer/serializer.go:294 in SendJSONToV1Intake) | Sent processes metadata payloa
Oct 22 17:55:00 beckham.adm.thread.com agent[1977]: 2019-10-22 17:55:00 BST | CORE | INFO | (pkg/serializer/serializer.go:294 in SendJSONToV1Intake) | Sent processes metadata payloa
Oct 22 18:00:00 beckham.adm.thread.com agent[1977]: 2019-10-22 18:00:00 BST | CORE | INFO | (pkg/serializer/serializer.go:274 in SendMetadata) | Sent host metadata payload, size (ra
Oct 22 18:00:00 beckham.adm.thread.com agent[1977]: 2019-10-22 18:00:00 BST | CORE | INFO | (pkg/serializer/serializer.go:294 in SendJSONToV1Intake) | Sent processes metadata payloa

Additional environment details (Operating System, Cloud provider, etc):

Debian Stretch, dedicated hosts. Agent version is 6.41.1, from commit fa227f0, serialisation version 4.12.0, Go version go1.12.9; representing an install of the Debian package version 1:6.14.1-1. systemd is systemd 232; the YAML of the check is:

init_config:

instances:
  - unit_names:
      - ssh.service
      - [redacted].service

With the latter service absent the behaviour is the same.

Steps to reproduce the issue:

1. $ sudo -u dd-agent -- datadog-agent check systemd
2. Behold the journal: $ sudo journalctl --since '2m ago'

Describe the results you received:

Logs of the following format:

systemd[1]: Incoming private connection from unprivileged client, refusing: Operation not permitted

Describe the results you expected:

The sound of silence.

Additional information you deem important (e.g. issue happens only occasionally):

This happens every time the agent tries to communicate with systemd. It is triggered by this line:

https://github.com/DataDog/datadog-agent/blob/6.14.1/pkg/collector/corechecks/systemd/systemd.go#L236

A connection to /run/systemd/private will always be refused when running as a uid other than root; see this discussion for a similar issue with the Prometheus project. The agent correctly then detects the error and falls back to using dbus.

The failing connection does, however, cause systemd to emit security messages about the denied connection which are currently flooding our log aggregation :)

Configuring an alternative path such as /dev/null to cause the connection to fail more quietly removes the dbus fallback which causes the check not to work. Lennart Poettering has said that /run/systemd/private shouldn't generally be considered a public API; the commit history suggests that this is in order to access the host systemd from within a Docker container, but perhaps that behaviour could be suppressed when not running within a container, since it cannot work unless running as root anyway?