From 4f87dc6a592638cb6effcf6befa2dd334d826613 Mon Sep 17 00:00:00 2001 From: Alex Privalov Date: Sat, 30 Oct 2021 15:00:23 -0600 Subject: [PATCH 1/6] Optional Worker keys --- README.md | 24 ++++++++++++++---------- tasks/worker/configure.yml | 10 ++++++---- 2 files changed, 20 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index 1ae1579..76ef8eb 100644 --- a/README.md +++ b/README.md @@ -82,7 +82,7 @@ but exist for when control over related behaviour is needed. See examples for a * `concourse_service_enabled`: Optional. Default: "yes". Manage a `systemd` service for a Concourse `web` and/or `worker` instance. * `concourse_service_start`: Optional. Default: "yes". Start the `systemd` service(s) for Concourse `web` and/or `worker`. * `concourse_log_level`: Optional. The minimum level of logs to see. [debug|info|error|fatal] -* `concourse_env_file`: Optional. A file containing environment variables which is fed into the `EnvironmentFile` attribute of the +* `concourse_env_file`: Optional. A file containing environment variables which is fed into the `EnvironmentFile` attribute of the `systemd` service unit file. This is useful for configuration managed outside of the playbook. If the configuration should be managed by the playbook, `concourse_web_env` and/or `concourse_worker_env` is the better choice. @@ -95,10 +95,10 @@ but exist for when control over related behaviour is needed. See examples for a * `concourse_tls_bind_port`: Optional. The port on which to listen for HTTPS traffic. * `concourse_tls_certificate`: Optional. The content of the TLS certificate to use for HTTPS termination. * `concourse_tls_certificate_path`: Optional. The remote file path of the TLS certificate to use for HTTPS termination. - Normally, only `concourse_tls_certificate` needs to be defined. + Normally, only `concourse_tls_certificate` needs to be defined. * `concourse_tls_key`: Optional. Optional. The content of the TLS key to use for HTTPS termination. * `concourse_tls_key_path`: Optional. The remote file path of the TLS key to use for HTTPS termination. - Normally, only `concourse_tls_key` needs to be defined. + Normally, only `concourse_tls_key` needs to be defined. * `concourse_peer_address`: Optional. The URL at which this ATC can be reached from other ATCs in the cluster. * `concourse_external_url`: Optional. The URL at which any ATC can be reached from the outside. * `concourse_web_launcher_path`: Optional. The path to the script that launches the Concourse web process. @@ -109,13 +109,13 @@ but exist for when control over related behaviour is needed. See examples for a * `concourse_session_signing_key`: Required. The session signing key. * `concourse_session_signing_key_path`: Optional. The path to the session signing key file. * `concourse_encryption_key`: Optional. A 16 or 32 length key used to encrypt sensitive data before storing - it in the database -* `concourse_old_encryption_key`: Optional. An encryption key previously used. If provided without a new key, + it in the database +* `concourse_old_encryption_key`: Optional. An encryption key previously used. If provided without a new key, data is encrypted. If provided with a new key, data is re-encrypted. * `concourse_host_key`: Required. The host key. * `concourse_authorized_worker_keys`: Required. Concatenated authorized worker keys. * `concourse_auth_duration`: Optional. The length of time for which tokens are valid. -* `concourse_resource_checking_interval`: Optional. Interval on which to check for new versions of resources. +* `concourse_resource_checking_interval`: Optional. Interval on which to check for new versions of resources. * `concourse_base_resource_type_defaults`: Optional. A hash of cluster-wide defaults for resource types. * `concourse_base_resource_type_defaults_file`: Optional. The path to the resource type defaults file. * `concourse_web_options`: Optional. Other non-managed options to pass to `concourse`. @@ -126,7 +126,7 @@ but exist for when control over related behaviour is needed. See examples for a * `concourse_postgres_host`: Optional. The Postgres host to connect to. * `concourse_postgres_port`: Optional. The Postgres port to connect to. * `concourse_postgres_socket`: Optional. The path to a Unix domain socket to connect to. -* `concourse_postgres_user`: Optional. The Postgres user to sign in as. +* `concourse_postgres_user`: Optional. The Postgres user to sign in as. * `concourse_postgres_password`: Optional. The Postgres user's password. * `concourse_postgres_ssl_mode`: Optional. Whether or not to use SSL with the Postgres connection. * `concourse_postgres_ca_cert`: Optional. The Postgres CA cert file location. @@ -137,7 +137,7 @@ but exist for when control over related behaviour is needed. See examples for a #### Web Local Authentication Variables -* `concourse_local_users`: Optional. A list of concourse user credentials that are added as local users. +* `concourse_local_users`: Optional. A list of concourse user credentials that are added as local users. Entries are objects having `name` and `password` fields (see example). Passwords can be plain text or bcrypted. * `concourse_main_team_local_users`: Optional. List of whitelisted local concourse users (of the supplied local user list). @@ -166,8 +166,10 @@ Unsupported. Do it yer dang self by supplying `concourse web` command options wi * `concourse_tsa_public_key_path`: Optional. The path to the tsa public key file. * `concourse_tsa_worker_key_path`: Optional. The path to the worker private key file. * `concourse_tsa_host`: Required. The value of the `--tsa-host` option. -* `concourse_tsa_public_key`: Required. The tsa public key. -* `concourse_tsa_worker_key`: Required. The tsa worker private key. +* `concourse_tsa_public_key`: Optional. The tsa public key. +* `concourse_tsa_worker_key`: Optional. The tsa worker private key. + + **Note**: if a key is not provided, then it is expected that the corresponding path (i.e. `concourse_tsa_(public|worker)_key_path`) is given for Concourse configuration, and the file with a proper value is provisioned during the bootstrap. * `concourse_worker_tag`: Optional. The value of the `--tag` option. * `concourse_baggageclaim_driver`: Optional. The driver to use for managing volumes. * `concourse_worker_options`: Optional. Other non-managed options to pass to `concourse`. @@ -183,6 +185,7 @@ Unsupported. Do it yer dang self by supplying `concourse web` command options wi ## Example Playbook +```yaml - hosts: atc roles: - role: troykinsella.concourse @@ -211,6 +214,7 @@ Unsupported. Do it yer dang self by supplying `concourse web` command options wi CONCOURSE_GARDEN_NETWORK_POOL: 10.254.0.0/16 CONCOURSE_GARDEN_MAX_CONTAINERS: 512 CONCOURSE_GARDEN_DOCKER_REGISTRY: https://docker.my-private-registry.org +``` ## Testing diff --git a/tasks/worker/configure.yml b/tasks/worker/configure.yml index 9d58329..b6f1320 100644 --- a/tasks/worker/configure.yml +++ b/tasks/worker/configure.yml @@ -1,5 +1,5 @@ --- -- name: copy tsa public key +- name: configure/worker - copy tsa public key copy: content: "{{ concourse_tsa_public_key }}" dest: "{{ concourse_tsa_public_key_path }}" @@ -8,9 +8,10 @@ mode: "{{ concourse_etc_files_mode }}" become: yes notify: - - restart concourse worker + - restart concourse worker + when: concourse_tsa_public_key is defined and concourse_tsa_public_key|length -- name: copy worker key +- name: configure/worker - copy worker key copy: content: "{{ concourse_tsa_worker_key }}" dest: "{{ concourse_tsa_worker_key_path }}" @@ -19,4 +20,5 @@ mode: "{{ concourse_etc_files_mode }}" become: yes notify: - - restart concourse worker + - restart concourse worker + when: concourse_tsa_worker_key is defined and concourse_tsa_worker_key|length From c9f8ea7a5f15074be9dc3d95a970de3c92a975a7 Mon Sep 17 00:00:00 2001 From: Alex Privalov Date: Sat, 30 Oct 2021 15:55:42 -0600 Subject: [PATCH 2/6] btrfs troubleshooting --- tasks/worker/filesystem.yml | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/tasks/worker/filesystem.yml b/tasks/worker/filesystem.yml index 00ea468..7b95c67 100644 --- a/tasks/worker/filesystem.yml +++ b/tasks/worker/filesystem.yml @@ -1,25 +1,24 @@ --- - - name: probe btrfs module modprobe: name: btrfs state: present become: yes - when: concourse_baggageclaim_driver == 'btrfs' + when: concourse_baggageclaim_driver == "btrfs" - name: install btrfs-progs package package: - name: btrfs-progs - state: present + name: btrfs-progs + state: present become: yes - when: concourse_baggageclaim_driver == 'btrfs' + when: concourse_baggageclaim_driver == "btrfs" - name: probe overlay module modprobe: name: overlay state: present become: yes - when: concourse_baggageclaim_driver == 'overlay' + when: concourse_baggageclaim_driver == "overlay" - name: unmount work volume mount: From 0f2033393ade78d5052d43e461276f220be6deb0 Mon Sep 17 00:00:00 2001 From: Alex Privalov Date: Sat, 30 Oct 2021 16:03:39 -0600 Subject: [PATCH 3/6] btrfs troubleshooting --- tasks/main.yml | 4 ++++ tasks/worker/filesystem.yml | 5 +++++ 2 files changed, 9 insertions(+) diff --git a/tasks/main.yml b/tasks/main.yml index 378b1cd..a0457f0 100644 --- a/tasks/main.yml +++ b/tasks/main.yml @@ -1,4 +1,8 @@ --- +- name: main - print extra info + ansible.builtin.debug: + msg: + - "concourse_baggageclaim_driver: {{ concourse_baggageclaim_driver }}" - include: user.yml when: concourse_manage_user diff --git a/tasks/worker/filesystem.yml b/tasks/worker/filesystem.yml index 7b95c67..f5da8aa 100644 --- a/tasks/worker/filesystem.yml +++ b/tasks/worker/filesystem.yml @@ -1,4 +1,9 @@ --- +- name: worker/filesystem - print extra info + ansible.builtin.debug: + msg: + - "concourse_baggageclaim_driver: {{ concourse_baggageclaim_driver }}" + - name: probe btrfs module modprobe: name: btrfs From 2f768aa20d3bcdbbc105f0ba27e45573e1d055c7 Mon Sep 17 00:00:00 2001 From: Alex Privalov Date: Sat, 30 Oct 2021 16:13:12 -0600 Subject: [PATCH 4/6] install btrfs-progs even if volume management is off --- tasks/main.yml | 8 ++------ tasks/worker/filesystem.yml | 8 +++----- tasks/worker/install.yml | 9 ++++----- 3 files changed, 9 insertions(+), 16 deletions(-) diff --git a/tasks/main.yml b/tasks/main.yml index a0457f0..4c2257f 100644 --- a/tasks/main.yml +++ b/tasks/main.yml @@ -1,8 +1,4 @@ --- -- name: main - print extra info - ansible.builtin.debug: - msg: - - "concourse_baggageclaim_driver: {{ concourse_baggageclaim_driver }}" - include: user.yml when: concourse_manage_user @@ -14,5 +10,5 @@ - shell: /bin/true when: concourse_force_restart notify: - - restart concourse web - - restart concourse worker + - restart concourse web + - restart concourse worker diff --git a/tasks/worker/filesystem.yml b/tasks/worker/filesystem.yml index f5da8aa..7e061ae 100644 --- a/tasks/worker/filesystem.yml +++ b/tasks/worker/filesystem.yml @@ -1,9 +1,4 @@ --- -- name: worker/filesystem - print extra info - ansible.builtin.debug: - msg: - - "concourse_baggageclaim_driver: {{ concourse_baggageclaim_driver }}" - - name: probe btrfs module modprobe: name: btrfs @@ -30,6 +25,7 @@ path: "{{ concourse_work_volume_mount_path }}" state: unmounted become: yes + when: concourse_manage_work_volume - name: create work volume filesystem filesystem: @@ -38,6 +34,7 @@ force: "{{ concourse_work_volume_fs_force_create }}" resizefs: "{{ concourse_work_volume_fs_resize }}" become: yes + when: concourse_manage_work_volume - name: mount work volume mount: @@ -47,3 +44,4 @@ opts: "{{ concourse_work_volume_mount_opts }}" state: mounted become: yes + when: concourse_manage_work_volume diff --git a/tasks/worker/install.yml b/tasks/worker/install.yml index badb4c3..2c1ab43 100644 --- a/tasks/worker/install.yml +++ b/tasks/worker/install.yml @@ -9,7 +9,6 @@ become: yes - include: filesystem.yml - when: concourse_manage_work_volume - name: create worker launch script template: @@ -21,7 +20,7 @@ force: yes become: yes notify: - - restart concourse worker + - restart concourse worker - name: create worker land script template: @@ -52,7 +51,7 @@ become: yes register: concourse_worker_service notify: - - restart concourse worker - - systemd daemon reload + - restart concourse worker + - systemd daemon reload tags: - - no-test + - no-test From 128ccc86f00b4d022c074c757dfade4e66a2361d Mon Sep 17 00:00:00 2001 From: Alex Privalov Date: Sat, 30 Oct 2021 19:33:13 -0600 Subject: [PATCH 5/6] Optional Web keys --- README.md | 14 ++++++++------ tasks/web/configure.yml | 27 +++++++++++++++------------ 2 files changed, 23 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index 76ef8eb..97b53df 100644 --- a/README.md +++ b/README.md @@ -92,6 +92,14 @@ but exist for when control over related behaviour is needed. See examples for a * `concourse_web`: Optional. Set to "yes" to install the Concourse ATC. * `concourse_bind_ip`: Optional. The IP address on which to listen to web traffic. * `concourse_bind_port`: Optional. The port on which to listen for HTTP traffic. +* `concourse_host_key`: Optional. The host key. +* `concourse_authorized_worker_keys`: Optional. Concatenated authorized worker keys. +* `concourse_session_signing_key`: Optional. The session signing key. + + **Note**: if the keys are not provided, then it is expected that the corresponding paths (i.e. `concourse_(host|session_signing|authorized_worker)_key_path`) are given for Concourse configuration, and the files with proper values are provisioned during the bootstrap. +* `concourse_authorized_worker_keys_path`: Optional. The path to the authorized worker keys file. +* `concourse_host_key_path`: Optional. The path to the host key file. +* `concourse_session_signing_key_path`: Optional. The path to the session signing key file. * `concourse_tls_bind_port`: Optional. The port on which to listen for HTTPS traffic. * `concourse_tls_certificate`: Optional. The content of the TLS certificate to use for HTTPS termination. * `concourse_tls_certificate_path`: Optional. The remote file path of the TLS certificate to use for HTTPS termination. @@ -104,16 +112,10 @@ but exist for when control over related behaviour is needed. See examples for a * `concourse_web_launcher_path`: Optional. The path to the script that launches the Concourse web process. * `concourse_web_launcher_mode`: Optional. The file mode of the web launcher script. * `concourse_cli_artifacts_dir`: Optional. The value of the `--cli-artifacts-dir` option. -* `concourse_authorized_worker_keys_path`: Optional. The path to the authorized worker keys file. -* `concourse_host_key_path`: Optional. The path to the host key file. -* `concourse_session_signing_key`: Required. The session signing key. -* `concourse_session_signing_key_path`: Optional. The path to the session signing key file. * `concourse_encryption_key`: Optional. A 16 or 32 length key used to encrypt sensitive data before storing it in the database * `concourse_old_encryption_key`: Optional. An encryption key previously used. If provided without a new key, data is encrypted. If provided with a new key, data is re-encrypted. -* `concourse_host_key`: Required. The host key. -* `concourse_authorized_worker_keys`: Required. Concatenated authorized worker keys. * `concourse_auth_duration`: Optional. The length of time for which tokens are valid. * `concourse_resource_checking_interval`: Optional. Interval on which to check for new versions of resources. * `concourse_base_resource_type_defaults`: Optional. A hash of cluster-wide defaults for resource types. diff --git a/tasks/web/configure.yml b/tasks/web/configure.yml index f6fe1a0..81d2530 100644 --- a/tasks/web/configure.yml +++ b/tasks/web/configure.yml @@ -1,5 +1,5 @@ --- -- name: copy host key +- name: configure/web - copy host key copy: content: "{{ concourse_host_key }}" dest: "{{ concourse_host_key_path }}" @@ -8,9 +8,10 @@ mode: "{{ concourse_etc_files_mode }}" become: yes notify: - - restart concourse web + - restart concourse web + when: concourse_host_key is defined and concourse_host_key|length -- name: copy session signing key +- name: configure/web - copy session signing key copy: content: "{{ concourse_session_signing_key }}" dest: "{{ concourse_session_signing_key_path }}" @@ -19,9 +20,10 @@ mode: "{{ concourse_etc_files_mode }}" become: yes notify: - - restart concourse web + - restart concourse web + when: concourse_session_signing_key is defined and concourse_session_signing_key|length -- name: copy authorized worker keys +- name: configure/web - copy authorized worker keys template: src: authorized_worker_keys.j2 dest: "{{ concourse_authorized_worker_keys_path }}" @@ -30,9 +32,10 @@ mode: "{{ concourse_etc_files_mode }}" become: yes notify: - - restart concourse web + - restart concourse web + when: concourse_authorized_worker_keys is defined and concourse_authorized_worker_keys|length -- name: tls certificate | concourse +- name: configure/web - tls certificate | concourse copy: content: "{{ concourse_tls_certificate }}" dest: "{{ concourse_tls_certificate_path }}" @@ -42,9 +45,9 @@ become: yes when: concourse_tls_certificate is defined notify: - - restart concourse web + - restart concourse web -- name: tls key | concourse +- name: configure/web - tls key | concourse copy: content: "{{ concourse_tls_key }}" dest: "{{ concourse_tls_key_path }}" @@ -54,9 +57,9 @@ become: yes when: concourse_tls_key is defined notify: - - restart concourse web + - restart concourse web -- name: copy resource type defaults +- name: configure/web - copy resource type defaults copy: content: "{{ concourse_base_resource_type_defaults | to_nice_yaml }}" dest: "{{ concourse_base_resource_type_defaults_file }}" @@ -66,4 +69,4 @@ become: yes when: concourse_base_resource_type_defaults is defined notify: - - restart concourse web + - restart concourse web From a700a48a98ce8d4462a4cf357bab4d7f06063151 Mon Sep 17 00:00:00 2001 From: Alex Privalov Date: Sun, 31 Oct 2021 10:40:51 -0600 Subject: [PATCH 6/6] sensetive params README warning --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 97b53df..fadb8d8 100644 --- a/README.md +++ b/README.md @@ -47,6 +47,8 @@ to the concourse binary at launch time. Run `concourse web -h` or `concourse wor Note: The vast majority of variables have sensible defaults and normally need not be defined, but exist for when control over related behaviour is needed. See examples for a minimal configuration set. +**IMPORTANT**: do NOT use Ansible variables with sensitive keys, passwords and secrets in production, i.e. `concourse_encryption_key`, `concourse_github_client_secret`, `concourse_postgres_password`. As they will be leaked and exposed both in the logs and in the process' command string. Instead set the corresponding environment variables in `concourse_web_env`, i.e. `CONCOURSE_ENCRYPTION_KEY`, `CONCOURSE_GITHUB_CLIENT_SECRET`, `CONCOURSE_POSTGRES_PASSWORD`, etc. + ### Maintenance Variables * `concourse_force_restart`: Optional. Default: "no". Triggers a restart of the web and/or worker services regardless as to whether or not configuration has changed.