clustering: `--cluster.advertise-address` implicitly expects an IP address and does not let one use a hostname

[lelutin]

What did you do?

I followed the documentation at https://prometheus.io/docs/alerting/latest/high_availability/#configuration and set --cluster.advertise-address to be hostname:443 (where hostname really is the fqdn of the machine).

What did you expect to see?

I would expect alertmanager to use that hostname to advertise to other peers.

I'm trying to configure two alertmanager instances in different PoPs (for better availability on major network outages). Thus, I want them to talk using TLS for the gossip. I could use IP addresses directly but it feels annoying and brittle. I cannot let alertmanager autodiscover the advertise address since there is no private IP on those hosts.

Note that I've already mentioned in #1271 that the documentation is wrong and needs fixing. But this issue is more about the fact that I think requiring an IP address for that option should get changed to permit using a hostname

What did you see instead? Under which circumstances?

The service fails to start and issues a log message about failing to parse an address on hostname

I dug around in the source code and found out that the option gets passed on to net.ParseIP here, which causes the parsing errors: https://github.com/prometheus/alertmanager/blob/main/cluster/advertise.go#L38-L42

To fix this we would just need to add some call to resolve the option to an IP in the same function if the option does not look like an IP address.

System information

Debian trixie

Alertmanager version

alertmanager, version 0.28.1+ds (branch: debian/sid, revision: 0.28.1+ds-1)
  build user:       team+pkg-go@tracker.debian.org
  build date:       20250309-13:50:43
  go version:       go1.24.1
  platform:         linux/amd64
  tags:             unknown

installed from debian archive

Alertmanager configuration file

Prometheus version

Prometheus configuration file

Logs

~~~
Mar 25 16:43:06 prometheus-04 prometheus-alertmanager[436770]: time=2026-03-25T16:43:06.017Z level=INFO source=main.go:191 msg="Starting Alertmanager" version="(version=0.28.1+ds, branch=debian/sid, revision=0.28.1+ds-1)"
Mar 25 16:43:06 prometheus-04 prometheus-alertmanager[436770]: time=2026-03-25T16:43:06.017Z level=INFO source=main.go:192 msg="Build context" build_context="(go=go1.24.1, platform=linux/amd64, user=team+pkg-go@tracker.debian.org, date=20250309-13:50:43, tags=unknown)"
Mar 25 16:43:06 prometheus-04 prometheus-alertmanager[436770]: time=2026-03-25T16:43:06.017Z level=WARN source=cluster.go:178 msg="couldn't deduce an advertise address: failed to parse advertise addr 'prometheus-04.torproject.org'" component=cluster
Mar 25 16:43:06 prometheus-04 prometheus-alertmanager[436770]: time=2026-03-25T16:43:06.018Z level=INFO source=cluster.go:241 msg="using TLS for gossip" component=cluster
Mar 25 16:43:06 prometheus-04 prometheus-alertmanager[436770]: time=2026-03-25T16:43:06.019Z level=ERROR source=main.go:259 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: failed to parse advertise address \"prometheus-04.torproject.org\""
~~~

[TheMeier]

I agree that the documentation shoul be improved. But I am not so sure resolving hostnames to IPs is the right way to go, which IP do you choose if multiple are returned?

[lelutin]

@TheMeier I can see the kind of issues that would lead to in the code. It could be something like "pick the first one" (which tends to be semi-random in DNS responses) and then issue a warning about the muliple entries. That surprising behavior could be specified in the docs, saying something along the lines of if a DNS hostname is specified it is expected to resolve to only one IP address. If more than one IP is returned by the DNS record, the first IP in the response will be used which might lead to unexpected behavior

or, the additional IPs could be kept around and used to fallback to in case of connection troubles..

most systems that implement some form of clustering make it possible to use DNS hostnames so it is a bit of a surprise that it's not permitted in this case

[anarcat]

i would also argue, simply: garbage-in, garbage-out. if you think that having a round-robin, multiple addresses there makes no sense, you could error out during resolution entirely, but i think it actually makes a lot of sense here.

i would even argue this is necessary if you are (like we are) in a dual-stack IPv4/IPv6 environment, where you necessarily return two IP addresses and (amazingly!) expect a fallback between the two.

there's actually a standard for this behaviour, as far as IPv6 is concerned, called Happy Eyeballs and it would be nice if alertmanager could support this, and that essentially requires accepting a hostname there. there is also a "happy eyeballs v2" that generalizes this behaviour more broadly to HTTP/2, HTTP/3 and RR records, but it might be a little further than what we need here.

(essentially, if you take the approach that RR records are acceptable here, what you're doing is creating a "hierarchical mesh" of alertmanagers clusters, where you can have multiple machines on each side of the cluster... i think it's an interesting concept, but certainly not one that's currently possible to implement, i think...)

[TheMeier]

I don't know very much abbout gossip, but I think it usually encodes IP (and port?) in the peer names. So happy eyeballs might also not be so easy.
Also please go easy on me. I was just trying to point out that there is more to consider here.

[anarcat]

Also please go easy on me. I was just trying to point out that there is more to consider here.

Oh yeah, of course! Sorry if we came across as aggressive here, we're merely trying to point out our experience with the HA stuff was a little obscure and difficult to figure out, and possible ways to improve it! :) This is a wishlist, really, no pressure whatsoever. Things are working great right now and I'm pretty excited we managed to setup HA! Thank you for all you do on Prometheus!!