CLOUDP-338084 - removing and refactoring agent matrix from pipeline.py and atomic_pipeline.py (#346)

# Summary
## Why we do this
- since we don't do an agent matrix release anymore, there is no need to
release all the agents we see on `release.json`. Instead we should only
release the agent if PCT adds a new agent. That happens during OM and CM
bumps, the new detection script should handle this and release the
images

## What changes
- adding a detection script that detects agent changes between `local`
vs `origin/master` for `release.json` and uses that as a base to do the
release
- streamline evergreen.yml and remove matrix builds/releases
- streamline agent builds on pipeline.py

**Evergreen configuration cleanup and simplification:**

* Removed obsolete tasks and buildvariants related to agent image
releases, such as `release_agent_operator_release`,
`release_agents_on_ecr_conditional`, and `init_release_agents_on_ecr`,
to streamline the release process.

**Pipeline logic refactoring:**

* Refactored the `build_agent_default_case` function in `pipeline.py` to
use the new `detect_ops_manager_changes` function for determining which
agent versions to build, and eliminated the separate
`build_agent_on_agent_bump` logic.
* Simplified agent in `pipeline.py` to match `atomic_pipeline.py`
* Updated the image builder function mapping so that both `"agent"` and
`"agent-pct"` use the unified `build_agent_default_case` function.

## Proof of Work

- no agent needed to be released - patch:
[Link](https://spruce.mongodb.com/task/mongodb_kubernetes_init_test_run_build_agent_images_ubi_patch_0786a90d4034657a11c9289e917c62691bc2500f_689da41126d25300077a269a_25_08_14_08_53_40/logs?execution=0)
```
[2025/08/14 10:56:13.642] === Detecting OM Mapping Changes (Local vs Base) ===
[2025/08/14 10:56:13.643] INFO     2025-08-14 08:56:13,642 [atomic_pipeline]  No changes detected, skipping agent build
[2025/08/14 10:56:13.725] Finished command 'subprocess.exec' in function 'pipeline' (step 3.5 of 3) in 4.209554597s.
```
- manual changing release.json ([the changeset, its a manual rsynced
patch](https://evergreen.mongodb.com/filediff/689e037a1e4cc8000785a0fd/?file_name=release.json&patch_number=0&commit_number=4)),
leading to an agent release/build. Note; the task is not passing because
I am releasing a non existant image
-[Link](https://spruce.mongodb.com/task/mongodb_kubernetes_init_test_run_build_agent_images_ubi_patch_f0df6f42547f5602c5c4ab120d1d7e3d0db05797_689e037a1e4cc8000785a0fd_25_08_14_15_40_44/logs?execution=0)
```
[2025/08/14 17:42:50.558] INFO     2025-08-14 15:42:50,558 [atomic_pipeline]  ======= Agent versions to build [('13.30.0.9590-1', '100.12.2')] =======
[2025/08/14 17:42:50.558] INFO     2025-08-14 15:42:50,558 [atomic_pipeline]  ======= Building Agent ('13.30.0.9590-
```
- cm bump worked with this changeset
[link](#311) + [the
related
patch](https://spruce.mongodb.com/task/mongodb_kubernetes_init_test_run_build_agent_images_ubi_patch_f0df6f42547f5602c5c4ab120d1d7e3d0db05797_689dcf5b0e81480007929ddc_25_08_14_11_58_22/logs?execution=0)
- this caused in init_test_run agent build to release the agent to ecr
as release.json has changed
```
[2025/08/14 13:59:35.068] === Detecting OM Mapping Changes (Local vs Base) ===

[2025/08/14 13:59:35.068] INFO     2025-08-14 11:59:35,067 [atomic_pipeline]  Building Agent versions: [('13.38.0.9654-1', '100.12.2')]

[2025/08/14 13:59:35.068] INFO     2025-08-14 11:59:35,068 [atomic_pipeline]  Running with factor of None

[2025/08/14 13:59:35.068] INFO     2025-08-14 11:59:35,068 [atomic_pipeline]  ======= Agent versions to build [('13.38.0.9654-1', '100.12.2')] =======

[2025/08/14 13:59:35.068] INFO     2025-08-14 11:59:35,068 [atomic_pipeline]  ======= Building Agent ('13.38.0.9654-1', '100.12.2') (0/1)
```

- we have a dedicated variant that can also release all agents:
[link](https://parsley.mongodb.com/evergreen/mongodb_kubernetes_manual_ecr_release_agent_release_all_agents_on_ecr_patch_0786a90d4034657a11c9289e917c62691bc2500f_689dea314fcada00075dd3c4_25_08_14_13_52_51/0/task?bookmarks=0,6038)

### Example Cases
**A new OM/CM bump workflow**
- publish_om/cm and release_agent variants are getting triggered
- detection script detects a change in release.json
- release the new agent

## Checklist

- [x] Have you linked a jira ticket and/or is the ticket in the title?
- [x] Have you checked whether your jira ticket required DOCSP changes?
- [x] Have you added changelog file?
    - use `skip-changelog` label if not needed
- refer to [Changelog files and Release
Notes](https://github.com/mongodb/mongodb-kubernetes/blob/master/CONTRIBUTING.md#changelog-files-and-release-notes)
section in CONTRIBUTING.md for more details

Author: nammn

.evergreen-functions.yml

@@ -517,30 +517,14 @@ functions:
           # docker buildx needs the moby/buildkit image when setting up a builder so we pull it from our mirror
           docker buildx create --driver=docker-container --driver-opt=image=268558157000.dkr.ecr.eu-west-1.amazonaws.com/docker-hub-mirrors/moby/buildkit:buildx-stable-1 --use
           docker buildx inspect --bootstrap
-    - command: ec2.assume_role
-      display_name: Assume IAM role with permissions to pull Kondukto API token
-      params:
-        role_arn: ${kondukto_role_arn}
-    - command: shell.exec
-      display_name: Pull Kondukto API token from AWS Secrets Manager and write it to file
-      params:
-        silent: true
-        shell: bash
-        include_expansions_in_env: [AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN]
-        script: |
-          set -e
-          # use AWS CLI to get the Kondukto API token from AWS Secrets Manager
-          kondukto_token=$(aws secretsmanager get-secret-value --secret-id "kondukto-token" --region "us-east-1" --query 'SecretString' --output text)
-          # write the KONDUKTO_TOKEN environment variable to Silkbomb environment file
-          echo "KONDUKTO_TOKEN=$kondukto_token" > ${workdir}/silkbomb.env
     - command: subprocess.exec
       retry_on_failure: true
       type: setup
       params:
         shell: bash
         <<: *e2e_include_expansions_in_env
         working_dir: src/github.com/mongodb/mongodb-kubernetes
-        binary: scripts/dev/run_python.sh scripts/release/pipeline_main.py --parallel ${image_name}
+        binary: scripts/dev/run_python.sh scripts/release/pipeline_main.py --parallel ${image_name} ${all_agents}
 
   # TODO: CLOUDP-335471 ; once all image builds are made with the new atomic pipeline, remove the following function
   legacy_pipeline:

.evergreen.yml

@@ -61,14 +61,14 @@ variables:
         variant: init_test_run
       - name: build_test_image
         variant: init_test_run
-      - name: build_agent_images_ubi
-        variant: init_test_run
       - name: build_readiness_probe_image
         variant: init_test_run
       - name: build_upgrade_hook_image
         variant: init_test_run
       - name: build_mco_test_image
         variant: init_test_run
+      - name: build_agent_images_ubi
+        variant: init_test_run
 
   - &setup_group
     setup_group_can_fail_task: true
@@ -347,19 +347,6 @@ tasks:
           image_name: init-ops-manager
           include_tags: release
 
-  - name: release_agent_operator_release
-    tags: [ "image_release" ]
-    allowed_requesters: [ "patch", "github_tag" ]
-    commands:
-      - func: clone
-      - func: setup_building_host
-      - func: quay_login
-      - func: setup_docker_sbom
-      - func: legacy_pipeline
-        vars:
-          image_name: agent
-          include_tags: release
-
   # pct only triggers this variant once a new agent image is out
   - name: release_agent
     # this enables us to run this variant either manually (patch) which pct does or during an OM bump (github_pr)
@@ -371,8 +358,7 @@ tasks:
       - func: setup_docker_sbom
       - func: legacy_pipeline
         vars:
-          image_name: agent-pct
-          include_tags: release
+          image_name: agent
 
   - name: run_precommit_and_push
     tags: ["patch-run"]
@@ -392,48 +378,17 @@ tasks:
           working_dir: src/github.com/mongodb/mongodb-kubernetes
           binary: scripts/evergreen/precommit_bump.sh
 
-  # Pct only triggers this variant once a new agent image is out
-  # these releases the agent with the operator suffix (not patch id) on ecr to allow for digest pinning to pass.
-  # For this to work, we rely on skip_tags which is used to determine whether
-  # we want to release on quay or not, in this case - ecr instead.
-  # We rely on the init_database from ecr for the agent x operator images.
-  # This runs on agent releases that are not concurrent with operator releases.
-  - name: release_agents_on_ecr_conditional
-    commands:
-      - func: clone
-      - func: run_task_conditionally
-        vars:
-          condition_script: scripts/evergreen/should_release_agents_on_ecr.sh
-          variant: init_release_agents_on_ecr
-          task: release_agents_on_ecr
-
-  - name: release_agents_on_ecr
-    # this enables us to run this variant either manually (patch) which pct does or during an OM bump (github_pr)
-    allowed_requesters: [ "patch", "github_pr" ]
-    priority: 70
-    commands:
-      - func: clone
-      - func: setup_building_host
-      - func: legacy_pipeline
-        vars:
-          image_name: agent-pct
-          skip_tags: release
-
   - name: release_all_agents_on_ecr
-    # this enables us to run this manually (patch) and release all agent versions to ECR
-    # it's needed during operator new version release process - e2e tests (especially olm tests)
-    # will look for agent with new operator version suffix, but during PR checks we only build
-    # agent versions for most recent major OM versions and the tests will fail. Before running the PR
-    # we have to manually release all agents to ECR by triggering this patch
+    # this enables us to run this manually (patch) and release all agent versions to ECR to verify
+    # Dockerfile, script changes etc.
     allowed_requesters: [ "patch" ]
     commands:
       - func: clone
       - func: setup_building_host
-      - func: legacy_pipeline
+      - func: pipeline
         vars:
-          image_name: agent-pct
-          skip_tags: release
-          all_agents: true
+          image_name: agent
+          all_agents: "--all-agents"
 
   - name: build_test_image
     commands:
@@ -1334,8 +1289,7 @@ buildvariants:
         variant: init_test_run
       - name: build_init_database_image_ubi
         variant: init_test_run
-      - name: build_agent_images_ubi
-        variant: init_test_run
+
     tasks:
       - name: e2e_custom_domain_task_group
 
@@ -1369,8 +1323,7 @@ buildvariants:
         variant: init_test_run
       - name: build_init_database_image_ubi
         variant: init_test_run
-      - name: build_agent_images_ubi
-        variant: init_test_run
+
     run_on:
       - ubuntu2204-small
     tasks:
@@ -1594,6 +1547,8 @@ buildvariants:
         variant: init_test_run
       - name: prepare_and_upload_openshift_bundles_for_e2e
         variant: init_tests_with_olm
+      - name: build_agent_images_ubi
+        variant: init_test_run
     tasks:
       - name: e2e_kind_olm_group
 
@@ -1619,6 +1574,7 @@ buildvariants:
         variant: init_test_run
       - name: build_agent_images_ubi
         variant: init_test_run
+
     tasks:
       - name: e2e_kind_olm_group
 
@@ -1683,18 +1639,6 @@ buildvariants:
       - name: build_upgrade_hook_image
       - name: prepare_aws
 
-  - name: init_release_agents_on_ecr
-    display_name: init_release_agents_on_ecr
-    # this enables us to run this variant either manually (patch) which pct does or during an OM bump (github_pr)
-    allowed_requesters: [ "patch", "github_pr" ]
-    tags: [ "release_agents_on_ecr" ]
-    # We want that to run first and finish asap. Digest pinning depends on this to succeed.
-    priority: 70
-    run_on:
-      - ubuntu2204-large
-    tasks:
-      - name: release_agents_on_ecr_conditional
-
   - name: run_pre_commit
     priority: 70
     display_name: run_pre_commit
@@ -1722,8 +1666,7 @@ buildvariants:
         variant: init_test_run
       - name: build_init_om_images_ubi
         variant: init_test_run
-      - name: build_agent_images_ubi
-        variant: init_test_run
+
     run_on:
       - ubuntu2204-small
     tasks:
@@ -1809,13 +1752,6 @@ buildvariants:
       - name: release_init_database
       - name: release_init_ops_manager
       - name: release_database
-      # Once we release the operator, we will also release the init databases, we require them to be out first
-      # such that we can reference them and retrieve those binaries.
-      # Since we immediately run daily rebuild after creating the image, we can ensure that the init_database is out
-      # such that the agent image build can use it.
-      - name: release_agent_operator_release
-        depends_on:
-          - name: release_init_database
 
   - name: preflight_release_images
     display_name: preflight_release_images
@@ -1847,13 +1783,13 @@ buildvariants:
 
     # It will be called by pct while bumping the agent cloud manager image
   - name: release_agent
-    display_name: (Static Containers) Release Agent matrix
+    display_name: release_agent
     tags: [ "release_agent" ]
     run_on:
       - release-ubuntu2204-large # This is required for CISA attestation https://jira.mongodb.org/browse/DEVPROD-17780
     depends_on:
-      - variant: init_release_agents_on_ecr
-        name: '*'
+      - variant: init_test_run
+        name: build_agent_images_ubi # this ensures the agent gets released to ECR as well
       - variant: e2e_multi_cluster_kind
         name: '*'
       - variant: e2e_static_multi_cluster_2_clusters

inventories/agent.yaml

@@ -3,7 +3,7 @@ vars:
   s3_bucket: s3://enterprise-operator-dockerfiles/dockerfiles/mongodb-agent
 
 images:
-  - name: mongodb-agent-ubi
+  - name: mongodb-agent
     vars:
       context: .
       template_context: docker/mongodb-agent

lib/sonar/template.py

@@ -13,7 +13,7 @@ def render(path: str, template_name: str, parameters: Dict[str, str]) -> str:
     """
     env = jinja2.Environment(loader=jinja2.FileSystemLoader(path), undefined=jinja2.StrictUndefined)
 
-    template = "Dockerfile"
+    template = "Dockerfile.old"
     if template_name is not None:
         template = "Dockerfile.{}".format(template_name)