CLOUDP-311484 - add support for arm and ibm machines (#330)

# Summary

Related TD:
[Link](https://docs.google.com/document/d/1wzwtRTaivKCnCpOobeZRMccC2u8ygO2DNE3DQghTvuU/edit?tab=t.0#bookmark=id.5jh2f6g25k1x)

This pull request adds support for running end-to-end (E2E) tests and
building images on IBM Power (ppc64le), IBM Z (s390x), and ARM (arm64)
architectures. It introduces new Evergreen build variants, tasks, and
setup logic to enable cross-platform testing and image creation.
Additionally, it removes the legacy agent image build dependencies and
updates scripts to use more generic container authentication. There are
also improvements to how multi-architecture images are defined in the
build info.

**Multi-architecture E2E test and build support:**

* Added new Evergreen build variants, task groups, and setup logic for
IBM Power (`ppc64le`), IBM Z (`s390x`), and ARM (`arm64`) to enable E2E
smoke tests and image builds on these platforms. This includes new setup
groups, test image build tasks, and test group definitions for each
architecture.
[[1]](diffhunk://#diff-9a7744843f307953832ec106e8d49be546037475db24ec6873e5fec6249c236dR74-R80)
[[2]](diffhunk://#diff-9a7744843f307953832ec106e8d49be546037475db24ec6873e5fec6249c236dR1183-R1198)
[[3]](diffhunk://#diff-9a7744843f307953832ec106e8d49be546037475db24ec6873e5fec6249c236dR1462-R1573)
[[4]](diffhunk://#diff-9a7744843f307953832ec106e8d49be546037475db24ec6873e5fec6249c236dR1787-R1804)
[[5]](diffhunk://#diff-9a7744843f307953832ec106e8d49be546037475db24ec6873e5fec6249c236dR409-R423)
[[6]](diffhunk://#diff-ad8722e626fc7bc08be6765b8268550446b1fb934c1a7eb6a5766d6446f92ad1R532-R543)
* Updated `build_info.json` to specify that images for
`mongodb-kubernetes` and `mongodb-kubernetes-init-database` are built
for `linux/amd64`, `linux/arm64`, `linux/s390x`, and `linux/ppc64le`
across all release channels.
[[1]](diffhunk://#diff-ac3b10589f332066db566cb08c01afea746d5f16e704beef946107a0c5de5225L8-R31)
[[2]](diffhunk://#diff-ac3b10589f332066db566cb08c01afea746d5f16e704beef946107a0c5de5225L50-R80)

**Evergreen pipeline and function improvements:**

* Added new setup functions for installing required tools (`jq`,
minikube, etc.) and improved host setup logic for IBM architectures,
including conditional logic for Docker/Podman and AWS CLI installation.
[[1]](diffhunk://#diff-ad8722e626fc7bc08be6765b8268550446b1fb934c1a7eb6a5766d6446f92ad1R54-R69)
[[2]](diffhunk://#diff-ad8722e626fc7bc08be6765b8268550446b1fb934c1a7eb6a5766d6446f92ad1R118)
[[3]](diffhunk://#diff-ad8722e626fc7bc08be6765b8268550446b1fb934c1a7eb6a5766d6446f92ad1L115-L121)
[[4]](diffhunk://#diff-ad8722e626fc7bc08be6765b8268550446b1fb934c1a7eb6a5766d6446f92ad1R265-R293)
* Refactored E2E setup and teardown calls to use a wrapper script
(`run_python.sh`) for better environment handling.
[[1]](diffhunk://#diff-ad8722e626fc7bc08be6765b8268550446b1fb934c1a7eb6a5766d6446f92ad1L330-R355)
[[2]](diffhunk://#diff-ad8722e626fc7bc08be6765b8268550446b1fb934c1a7eb6a5766d6446f92ad1L343-R367)
* Added a manual Evergreen task and build variant to allow manual ECR
agent image releases for testing.
[[1]](diffhunk://#diff-9a7744843f307953832ec106e8d49be546037475db24ec6873e5fec6249c236dR388-R399)
[[2]](diffhunk://#diff-9a7744843f307953832ec106e8d49be546037475db24ec6873e5fec6249c236dR1977-R1986)

**Cleanup and modernization:**

* Updated references from `configure_docker_auth.sh` to a more generic
`configure_container_auth.sh` in both Evergreen and Makefile to support
Podman as well as Docker.
[[1]](diffhunk://#diff-ad8722e626fc7bc08be6765b8268550446b1fb934c1a7eb6a5766d6446f92ad1L228-R234)
[[2]](diffhunk://#diff-76ed074a9305c04054cdebb9e9aad2d818052b07091de1f20cad0bbac34ffb52L150-R150)

These changes collectively enable robust multi-architecture support in
both CI and release pipelines, modernize setup scripts, and remove
obsolete dependencies.

## Proof of Work

- green ci (no changes to currently running e2e patches)
- green ci on master merges (simulated
[here](https://spruce.mongodb.com/version/68a4bd9f74629000078b930e/tasks?sorts=STATUS%3AASC%3BBASE_STATUS%3ADESC)
which runs multi arch tests on master merges)
- manually released the agents to ecr as multi-arch
```
❯ docker buildx imagetools inspect 268558157000.dkr.ecr.us-east-1.amazonaws.com/dev/mongodb-agent-ubi:108.0.4.8770-1
Name:      268558157000.dkr.ecr.us-east-1.amazonaws.com/dev/mongodb-agent-ubi:108.0.4.8770-1
MediaType: application/vnd.docker.distribution.manifest.list.v2+json
Digest:    sha256:6c30ad0325aeac074538976094e0a5b8a69bdb819fc95496c6f52acefd028188

Manifests:
  Name:      268558157000.dkr.ecr.us-east-1.amazonaws.com/dev/mongodb-agent-ubi:108.0.4.8770-1@sha256:38494a7b753ce8b9965332e12c195d56f143aa2b4505352237c6cdf757916475
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm64

  Name:      268558157000.dkr.ecr.us-east-1.amazonaws.com/dev/mongodb-agent-ubi:108.0.4.8770-1@sha256:bb28cdbd9b9cc97cb2a8168c7c00897a93b96c02e51a8611361829a1349da2f5
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/amd64

  Name:      268558157000.dkr.ecr.us-east-1.amazonaws.com/dev/mongodb-agent-ubi:108.0.4.8770-1@sha256:0ca9e958a6ac0a86541861bdfb39ae930f5b4dd588f9a8df1c9289579b020aa8
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/s390x

  Name:      268558157000.dkr.ecr.us-east-1.amazonaws.com/dev/mongodb-agent-ubi:108.0.4.8770-1@sha256:bdeb9b00a666500b336a4dc33a34f40992ee5ff1d57fbff62a6511bcd4cacad3
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/ppc64le
```

## Checklist

- [x] Have you linked a jira ticket and/or is the ticket in the title?
- [x] Have you checked whether your jira ticket required DOCSP changes?
- [x] Have you added changelog file?
    - use `skip-changelog` label if not needed
- refer to [Changelog files and Release
Notes](https://github.com/mongodb/mongodb-kubernetes/blob/master/CONTRIBUTING.md#changelog-files-and-release-notes)
section in CONTRIBUTING.md for more details

---------

Co-authored-by: Lucian Tosa <[email protected]>
Co-authored-by: Julien Benhaim <[email protected]>
Co-authored-by: Maciej Karaś <[email protected]>
Co-authored-by: Maciej Karaś <[email protected]>
Co-authored-by: mms-build-account <[email protected]>
Co-authored-by: Evergreen <[email protected]>
Co-authored-by: Vivek Singh <[email protected]>

Author: 7 people

.evergreen-functions.yml

@@ -51,10 +51,22 @@ functions:
 
   ### Setup Functions ###
 
+  setup_jq: &setup_jq
+    command: subprocess.exec
+    type: setup
+    params:
+      <<: *e2e_include_expansions_in_env
+      add_to_path:
+        - ${workdir}/bin
+      working_dir: src/github.com/mongodb/mongodb-kubernetes
+      binary: scripts/evergreen/setup_jq.sh
+
   setup_context: &setup_context # Running the first switch is important to fill the workdir and other important initial env vars
     command: shell.exec
     type: setup
     params:
+      add_to_path:
+        - ${workdir}/bin
       shell: bash
       working_dir: src/github.com/mongodb/mongodb-kubernetes
       <<: *e2e_include_expansions_in_env
@@ -103,6 +115,7 @@ functions:
       type: setup
       params:
         command: "git config --global user.email '[email protected]'"
+    - *setup_jq # we need jq in the context
     - *setup_context
 
   setup_kubectl: &setup_kubectl
@@ -112,13 +125,6 @@ functions:
       working_dir: src/github.com/mongodb/mongodb-kubernetes
       binary: scripts/evergreen/setup_kubectl.sh
 
-  setup_jq: &setup_jq
-    command: subprocess.exec
-    type: setup
-    params:
-      working_dir: src/github.com/mongodb/mongodb-kubernetes
-      binary: scripts/evergreen/setup_jq.sh
-
   setup_shellcheck:
     command: subprocess.exec
     type: setup
@@ -225,7 +231,7 @@ functions:
       working_dir: src/github.com/mongodb/mongodb-kubernetes
       add_to_path:
         - ${workdir}/bin
-      binary: scripts/dev/configure_docker_auth.sh
+      binary: scripts/dev/configure_container_auth.sh
 
   setup_evg_host: &setup_evg_host
     command: subprocess.exec
@@ -256,16 +262,35 @@ functions:
   # Configures docker authentication to ECR and RH registries.
   setup_building_host:
     - *switch_context
+    - *python_venv
     - *setup_aws
-    - *configure_docker_auth
     - *setup_evg_host
-    - *python_venv
+    - *configure_docker_auth
 
-  prune_docker_resources:
+  # This differs for normal evg_host as we require minikube instead of kind for
+  # IBM machines also install aws cli via pip instead and use podman
+  setup_building_host_minikube:
+    - *switch_context
     - command: subprocess.exec
       type: setup
       params:
-        command: "docker system prune -a -f"
+        working_dir: src/github.com/mongodb/mongodb-kubernetes
+        add_to_path:
+          - ${workdir}/bin
+        command: scripts/evergreen/setup_minikube_host.sh
+
+  prune_docker_resources:
+    - command: shell.exec
+      type: setup
+      params:
+        shell: bash
+        script: |
+          if command -v docker >/dev/null 2>&1; then
+            echo "Docker found, pruning docker resources..."
+            docker system prune -a -f
+          else
+            echo "Docker not found, skipping docker resource pruning"
+          fi
 
   # the task configures the set of tools necessary for any task working with K8 cluster:
   # installs kubectl, jq, kind (if necessary), configures docker authentication
@@ -327,8 +352,7 @@ functions:
         shell: bash
         working_dir: src/github.com/mongodb/mongodb-kubernetes
         script: |
-          source .generated/context.export.env
-          scripts/evergreen/e2e/setup_cloud_qa.py create
+          scripts/dev/run_python.sh scripts/evergreen/e2e/setup_cloud_qa.py create
     # The additional switch is needed, since we now have created the needed OM exports.
     - *switch_context
 
@@ -340,8 +364,7 @@ functions:
         shell: bash
         working_dir: src/github.com/mongodb/mongodb-kubernetes
         script: |
-          source .generated/context.export.env
-          scripts/evergreen/e2e/setup_cloud_qa.py delete
+          scripts/dev/run_python.sh scripts/evergreen/e2e/setup_cloud_qa.py delete
 
   dump_diagnostic_information_from_all_namespaces:
     - command: subprocess.exec
@@ -506,17 +529,20 @@ functions:
           - ${workdir}/bin
           - ${workdir}
 
-  pipeline:
+  build_test_image_ibm:
     - *switch_context
-    - command: shell.exec
-      type: setup
+    - command: subprocess.exec
       params:
         shell: bash
-        script: |
-          # Docker Hub workaround
-          # docker buildx needs the moby/buildkit image when setting up a builder so we pull it from our mirror
-          docker buildx create --driver=docker-container --driver-opt=image=268558157000.dkr.ecr.eu-west-1.amazonaws.com/docker-hub-mirrors/moby/buildkit:buildx-stable-1 --use
-          docker buildx inspect --bootstrap
+        working_dir: src/github.com/mongodb/mongodb-kubernetes
+        include_expansions_in_env:
+          - version_id
+        add_to_path:
+          - ${workdir}/bin
+        binary: scripts/evergreen/e2e/build_tests_image_ibm.sh
+
+  pipeline:
+    - *switch_context
     - command: subprocess.exec
       retry_on_failure: true
       type: setup

.evergreen.yml

@@ -71,6 +71,13 @@ variables:
       - func: download_kube_tools
       - func: setup_building_host
 
+  - &setup_group_ibm
+    setup_group_can_fail_task: true
+    setup_group:
+      - func: clone
+      - func: setup_building_host_minikube
+      - func: build_multi_cluster_binary
+
   - &setup_group_multi_cluster
     setup_group_can_fail_task: true
     setup_group:
@@ -379,6 +386,18 @@ tasks:
           image_name: agent
           all_agents: "--all-agents"
 
+  - name: release_all_currently_used_agents_on_ecr
+    # this enables us to run this manually (patch) and release all agent versions to ECR to verify
+    # Dockerfile, script changes etc.
+    allowed_requesters: [ "patch" ]
+    commands:
+      - func: clone
+      - func: setup_building_host
+      - func: pipeline
+        vars:
+          image_name: agent
+          all_agents: "--current-agents"
+
   - name: build_test_image
     commands:
       - func: clone
@@ -388,6 +407,21 @@ tasks:
         vars:
           image_name: meko-tests
 
+  - name: build_test_image_ibm
+    commands:
+      - func: clone
+      - command: subprocess.exec
+        type: setup
+        params:
+          env:
+            SKIP_MINIKUBE_SETUP: "true"
+          working_dir: src/github.com/mongodb/mongodb-kubernetes
+          add_to_path:
+            - ${workdir}/bin
+          command: scripts/evergreen/setup_minikube_host.sh
+      - func: build_multi_cluster_binary
+      - func: build_test_image_ibm
+
   - name: build_mco_test_image
     commands:
       - func: clone
@@ -1147,6 +1181,22 @@ task_groups:
       - e2e_om_ops_manager_backup
     <<: *teardown_group
 
+  - name: e2e_smoke_arm_task_group
+    max_hosts: -1
+    <<: *setup_group
+    <<: *setup_and_teardown_task_cloudqa
+    tasks:
+      - e2e_replica_set
+    <<: *teardown_group
+
+  - name: e2e_smoke_ibm_task_group
+    max_hosts: -1
+    <<: *setup_group_ibm
+    <<: *setup_and_teardown_task_cloudqa
+    tasks:
+      - e2e_replica_set
+    <<: *teardown_group
+
   - name: e2e_ops_manager_kind_5_0_only_task_group
     max_hosts: -1
     <<: *setup_group
@@ -1410,6 +1460,118 @@ buildvariants:
     tasks:
       - name: e2e_smoke_task_group
 
+  - name: e2e_smoke_ibm_power
+    display_name: e2e_smoke_ibm_power
+    tags: [ "e2e_test_suite", "e2e_smoke_release_test_suite" ]
+    run_on:
+      - rhel9-power-small
+      - rhel9-power-large
+    allowed_requesters: [ "patch", "github_tag" ]
+    depends_on:
+      - name: build_operator_ubi
+        variant: init_test_run
+      - name: build_init_database_image_ubi
+        variant: init_test_run
+      - name: build_database_image_ubi
+        variant: init_test_run
+      - name: build_init_appdb_images_ubi
+        variant: init_test_run
+      - name: build_init_om_images_ubi
+        variant: init_test_run
+      - name: build_test_image_ibm
+        variant: init_test_run_ibm_power
+    tasks:
+      - name: e2e_smoke_ibm_task_group
+
+  - name: e2e_smoke_ibm_z
+    display_name: e2e_smoke_ibm_z
+    tags: [ "e2e_test_suite", "e2e_smoke_release_test_suite" ]
+    run_on:
+      - rhel9-zseries-small
+      - rhel9-zseries-large
+    allowed_requesters: [ "patch", "github_tag" ]
+    depends_on:
+      - name: build_operator_ubi
+        variant: init_test_run
+      - name: build_init_database_image_ubi
+        variant: init_test_run
+      - name: build_database_image_ubi
+        variant: init_test_run
+      - name: build_init_appdb_images_ubi
+        variant: init_test_run
+      - name: build_init_om_images_ubi
+        variant: init_test_run
+      - name: build_test_image_ibm
+        variant: init_test_run_ibm_z
+    tasks:
+      - name: e2e_smoke_ibm_task_group
+
+  - name: e2e_smoke_arm
+    display_name: e2e_smoke_arm
+    tags: [ "e2e_test_suite", "e2e_smoke_release_test_suite" ]
+    run_on:
+      - ubuntu2204-arm64-large
+    allowed_requesters: [ "patch", "github_tag" ]
+    <<: *base_no_om_image_dependency
+    tasks:
+      - name: e2e_smoke_arm_task_group
+
+  - name: e2e_static_smoke_arm
+    display_name: e2e_smoke_arm
+    tags: [ "e2e_test_suite", "e2e_smoke_release_test_suite" ]
+    run_on:
+      - ubuntu2204-arm64-large
+    allowed_requesters: [ "patch", "github_tag" ]
+    <<: *base_no_om_image_dependency
+    tasks:
+      - name: e2e_smoke_arm_task_group
+
+  - name: e2e_static_smoke_ibm_z
+    display_name: e2e_static_smoke_ibm_z
+    tags: [ "e2e_test_suite", "e2e_smoke_release_test_suite" ]
+    run_on:
+      - rhel9-zseries-small
+      - rhel9-zseries-large
+    allowed_requesters: [ "patch", "github_tag" ]
+    depends_on:
+      - name: build_operator_ubi
+        variant: init_test_run
+      - name: build_init_database_image_ubi
+        variant: init_test_run
+      - name: build_database_image_ubi
+        variant: init_test_run
+      - name: build_init_appdb_images_ubi
+        variant: init_test_run
+      - name: build_init_om_images_ubi
+        variant: init_test_run
+      - name: build_test_image_ibm
+        variant: init_test_run_ibm_z
+    tasks:
+      - name: e2e_smoke_ibm_task_group
+
+  - name: e2e_static_smoke_ibm_power
+    display_name: e2e_static_smoke_ibm_power
+    tags: [ "e2e_test_suite", "e2e_smoke_release_test_suite" ]
+    run_on:
+      - rhel9-power-small
+      - rhel9-power-large
+    allowed_requesters: [ "patch", "github_tag" ]
+    depends_on:
+      - name: build_operator_ubi
+        variant: init_test_run
+      - name: build_init_database_image_ubi
+        variant: init_test_run
+      - name: build_database_image_ubi
+        variant: init_test_run
+      - name: build_init_appdb_images_ubi
+        variant: init_test_run
+      - name: build_init_om_images_ubi
+        variant: init_test_run
+      - name: build_test_image_ibm
+        variant: init_test_run_ibm_power
+    tasks:
+      - name: e2e_smoke_ibm_task_group
+
   - name: e2e_static_smoke
     display_name: e2e_static_smoke
     tags: [ "e2e_test_suite", "e2e_smoke_release_test_suite" ]
@@ -1623,6 +1785,24 @@ buildvariants:
       - name: build_upgrade_hook_image
       - name: prepare_aws
 
+  - name: init_test_run_ibm_power
+    display_name: init_test_run_ibm_power
+    max_hosts: -1
+    run_on:
+      - rhel9-power-small
+      - rhel9-power-large
+    tasks:
+      - name: build_test_image_ibm
+
+  - name: init_test_run_ibm_z
+    display_name: init_test_run_ibm_z
+    max_hosts: -1
+    run_on:
+      - rhel9-zseries-small
+      - rhel9-zseries-large
+    tasks:
+      - name: build_test_image_ibm
+
   - name: run_pre_commit
     priority: 70
     display_name: run_pre_commit
@@ -1795,6 +1975,16 @@ buildvariants:
     tasks:
       - name: release_all_agents_on_ecr
 
+    # Only called manually, It's used for testing the task release_agents_on_ecr in case the release.json
+    # has not changed, and you still want to push the images to ecr.
+  - name: manual_ecr_release_agent_currently_used
+    display_name: manual_ecr_release_agent_currently_used
+    tags: [ "release_all_agents_manually" ]
+    run_on:
+      - ubuntu2204-large
+    tasks:
+      - name: release_all_currently_used_agents_on_ecr
+
     # These variants are used to test the code snippets and each one can be used in patches
     # Prerelease is especially used when the repo is tagged
     # More details in the TD: https://docs.google.com/document/d/1fuTxfRtP8QPtn7sKYxQM_AGcD6xycTZH8svngGxyKhc/edit?tab=t.0#bookmark=id.e8uva0393mbe

Makefile

@@ -147,7 +147,7 @@ ac:
 # in parallel and both call 'aws_login') then Docker login may return an error "Error saving credentials:..The
 # specified item already exists in the keychain". Seems this allows to ignore the error
 aws_login:
-	@ scripts/dev/configure_docker_auth.sh
+	@ scripts/dev/configure_container_auth.sh
 
 # cleans up aws resources, including s3 buckets which are older than 5 hours
 aws_cleanup:

README.md

@@ -57,4 +57,3 @@ Migration from [MongoDB Community Operator](https://github.com/mongodb/mongodb-k
 See our detailed migration guides:
 - [Migrating from MongoDB Community Operator](docs/migration/community-operator-migration.md)
 - [Migrating from MongoDB Enterprise Kubernetes Operator](https://www.mongodb.com/docs/kubernetes/current/tutorial/migrate-to-mck/)
-