Skip to content

Add host to etcd cluster #59

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Add host to etcd cluster #59

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions home/.chezmoi.yaml.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,8 @@ data:
docker:
doRegion: nyc1
domain: "{{ $domain }}"
etcd:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we remove this? See comments below

Copy link
Contributor Author

@enggnr enggnr Jul 7, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we are to use the public_services_domain and the etcd cluster member available at etcd.{{ service_domain }} has to be the only member in the cluster. The host where the script runs will be the 2nd member. If there are more members in the cluster, then this variable is needed. This is because, cluster configuration is a 2 step process. More details below.

The initial_cluster_url is the value returned by etcdctl on adding a new host to the cluster. The command returns 3 values, which can be set as environment variables and a new member bootstrapped. This member will join the cluster using the information in these variables.

The value of initial_cluster_urls is set to the value of ETCD_INITIAL_CLUSTER (one of the 3 values returned).

An example is below, taken from here.

etcdctl member add infra3 --peer-urls=http://10.0.1.13:2380
added member 9bf1b35fc7761a23 to cluster

ETCD_NAME="infra3"
ETCD_INITIAL_CLUSTER="infra0=http://10.0.1.10:2380,infra1=http://10.0.1.11:2380,infra2=http://10.0.1.12:2380,infra3=http://10.0.1.13:2380"
ETCD_INITIAL_CLUSTER_STATE=existing

Start the new etcd member:
export ETCD_NAME="infra3"
$ export ETCD_INITIAL_CLUSTER="infra0=http://10.0.1.10:2380,infra1=http://10.0.1.11:2380,infra2=http://10.0.1.12:2380,infra3=http://10.0.1.13:2380"
$ export ETCD_INITIAL_CLUSTER_STATE=existing
$ etcd --listen-client-urls http://10.0.1.13:2379 --advertise-client-urls http://10.0.1.13:2379 --listen-peer-urls http://10.0.1.13:2380 --initial-advertise-peer-urls http://10.0.1.13:2380 --data-dir %data_dir%

Note: The ETCD_INITIAL_CLUSTER value includes the name and advertised peer URLs of existing members and the new member. This is used by the new member to verify the configuration.

While it is possible to use a discovery service (an internal one) to add a member (after the cluster is up and running), using runtime configuration is the recommended way.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See other comments below. We need to use the CloudFlare API to get a list of all of the subdomains for the PUBLIC_SERVICES_DOMAIN and then piece together the network like that. We can store data in TXT records, for instance and assume the master node of etcd for instance is available at etcd.megabyte.space.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @enggnr --- can we pull the DNS records for the PUBLIC_SERVICES_DOMAIN and return all the DNS entries that look something like etcd.{{ .host.hostname }}.{{ .host.domain }}? This should give all the existing nodes which we can then use to populate everything.

ETCD_NAME should just equal the {{ .host.hostname }}

Can you take a swing at this implementation? I think the cfcli should be able to provide all the functionality we need to get this working.

initial_cluster_urls: "node1=http://10.18.0.2:2380,etcd3=http://10.18.0.4:2380"
headless: {{ $headless }}
home: "{{ .chezmoi.homeDir }}"
homeParentFolder: "{{ if eq .chezmoi.os "linux" }}/home{{ else if eq .chezmoi.os "darwin" }}/Users{{ else }}C:\Users{{ end }}"
Expand Down
38 changes: 38 additions & 0 deletions home/.chezmoiscripts/universal/run_onchange_before_11-join-etcd-cluster.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
{{- if (eq .host.distro.family "linux") -}}
#!/usr/bin/env bash
# @file etcd Clustering
# @brief Adds the host to `etcd` cluster
# @description
# This script adds the host to `etcd` cluster provided as input, if `etcd` is installed. Static clustering
# mechanism is used to join the host to an existing etcd cluster. The other mechanisms that are based on
# service discovery is not supported by this script.
#
# Static mechanism needs the IP address(es) of all the participating members to bootstrap a node/member.
# This is the only input needed to be able to add the host as a member of the cluster. The format of this field
# is available in the [inputs file](https://github.com/megabyte-labs/install.doctor/blob/master/home/.chezmoi.yaml.tmpl).
#
# ## Configuration Variables
#
# The following chart details the input variable(s) that are used to join the host to `etcd` cluster:
#
# | Variable | Description |
# |------------------------------|------------------------------------------------------------|
# | `etcd.initial_cluster_urls` | Appropriately formatted initial cluster configuration |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we replace this with an address generated based on the domain address that gets populated in .chezmoi.yaml.tmpl? That way we can provision etcd even before we know the final IP address of the etcd server. Also, what is the node part for? Shouldn't we just need a server to connect to? Also, we should be using https. And where did you get the two IP addresses from listed above?

I think we should run this over CloudFlare tunnels which I'll be setting up so you can assume that etcd.{{ service_domain }} points to the server. Just not sure what the node1 is for.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The script is adding a member to an etcd cluster. If we are bootstrapping a new cluster, in which the host will be a member, then a discover mechanism can be used. In that case, the variable can be removed. This will let the cluster be created before the list of members and their addresses are known. This is different from adding a member to a cluster.

The node part is the name of the member in the cluster. I am not sure why there is a 'mapping' between the name and its advertise URLs. The value of advertise-peer-urls holds this 'map' of all the members that are part of the cluster and it changes when the membership changes.

The IPs are just examples. I knew this would need changes, so I put it in to show what is expected.

Yes, we should be using HTTPS. Certificates can be passed as inputs when starting a member to encrypt communication between peers and with clients. It is also possible to have the members automatically generate certificates for communication among peers. Certificates can be used for authentication as well.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @enggnr -- we need to come up with a clever plan to use the PUBLIC_SERVICES_DOMAIN's DNS record on CloudFlare to figure out all the other peers, masters and then either provision the master node or join to it with a peer.

Each network provisioned by Install Doctor will use the PUBLIC_SERVICES_DOMAIN to classify which network to join to.


{{ includeTemplate "universal/profile-before" }}
{{ includeTemplate "universal/logg-before" }}

### Join host to `etcd` cluster
if command -v etcd > /dev/null; then
logg info 'Adding host to the `etcd` cluster'
{{ if and .etcd.initial_cluster_urls (gt (len .etcd.initial_cluster_urls) 0) }}
etcd --name {{ .chezmoi.hostname }} \
--initial-advertise-peer-urls http://$(hostname -i):2380 --listen-peer-urls http://$(hostname -i):2380 \
--advertise-client-urls http://$(hostname -i):2379 --listen-client-urls http://$(hostname -i):2379 \
--initial-cluster "{{ .etcd.initial_cluster_urls }}" \
--initial-cluster-state existing
{{ end }}
else
logg info 'etcd is not installed or it is not available in the PATH'
fi
{{ end -}}