Separate MongoDB Multi-Cluster tests and encapsulate shared functionalities for reuse.

[filipcirtog]

Summary

Overview:
In order to replicate E2E test coverage from MongoDBMulti to MongoDB within a multicluster topology, we are transitioning to library-based approach. This PR marks the first step in this transition by focusing on the implementation of MongoDBMultiCluster using the shared library pattern. Future implementations for the MongoDB resource will adopt this same methodology.

File structure:
../multicluster/shared/example-test.py → Contains the core test logic shared across implementations.
../multicluster/mongodbmulticluster/mongodbmulticluster-example-test.py → Includes test fixtures and pytest calls to invoke methods from the shared library.

Exceptions to the Library-Based Approach:
Some tests do not utilize the shared library due to the increased complexity it would introduce. Unlike the multicluster folder, which includes 40 resource-specific tests leveraging the shared library approach, these tests are unique within their respective folders:
multicluster_om/multicluster_mongodbmulticluster_om_appdb_no_mesh.py
multicluster_appdb/multicluster_mongodbmulticluster_appdb_s3_based_backup_restore.py

Proof of Work

Shared library:

• The shared library contains the actual test logic and can be reused. This library does not include fixtures, as fixtures are resource-specific (MongoDB vs MongoDBMulti).

Test implementation

• For each test present in the shared library, a pytest is implemented in the MongoDBMultiCluster folder.

Naming convention for distinction:

• Fixtures and evergreen tests related to MongoDBMultiCluster are prefixed/named mongodbmulticluster for differentiation.

Checklist

• Have you linked a jira ticket and/or is the ticket in the title?
• Have you checked whether your jira ticket required DOCSP changes?
• Have you added changelog file?
  • use skip-changelog label if not needed
  • refer to Changelog files and Release Notes section in CONTRIBUTING.md for more details

[github-actions]

⚠️ (this preview might not be accurate if the PR is not rebased on current master branch)

MCK 1.6.1 Release Notes

[m1kola]

I think we should only modify tests that test MongoDBMultiCluster CRD. This one seems to be only about Ops Manager. I don't think we should touch this test.

Or am I missing something?

[filipcirtog]

There are some tests which use MongoDBMulti in a different way. For example, they aren't centered on testing MongoDBMulti but rather as a supporting component. These tests use MongoDBMulti as a resource to test functionalities such as backup and restore workflows, Ops Manager integration, or multi-cluster configurations without focusing on MongoDBMulti itself. I’ve attached such tests below (description using AI) :

multicluster/multi_cluster_reconcile_races.py - This test validates the MongoDB Kubernetes Operator's ability to reconcile resources reliably across multi-cluster environments while detecting data races and ensuring telemetry correctness.

multicluster_om/multicluster_om_appdb_no_mesh.py - This test validates the Multi-Cluster MongoDB Kubernetes Operator when deployed without Service Mesh, focusing on its ability to manage MongoDBMulti resources with TLS security, S3 backups, external app database monitoring, external connectivity, data restoration via restores, and telemetry validation.

multicluster_appdb/multicluster_appdb_s3_based_backup_restore.py - This test validates the MongoDB Kubernetes Operator in a multi-cluster environment with S3-backed backups and restores, ensuring Ops Manager is properly configured with cluster-level TLS, external connectivity, and custom DNS settings while verifying metadata stores, restore functionality, and backup consistency for MongoDBMulti resources.

While leaving them alone and just replacing MongoDBMultiCluster with MongoDB once the feature branch is merged is one possibility, I would consider replicating them as well in order to be on the safe side. As a result, it’s safer to leave these tests untouched (aside from changes such as renaming and library inclusion for multicluster/multi_cluster_reconcile_races.py) while creating new, identical tests for MongoDB. This approach also makes it easier to delete them once the resource is deprecated.

Additionally, multicluster_om/multicluster_om_appdb_no_mesh.py and multicluster_appdb/multicluster_appdb_s3_based_backup_restore.py will not adopt the library-based approach, as these tests are unique within their respective folders (being the only tests utilizing MongoDBMulti) and will simply be replicated as is.

[m1kola]

• multicluster/multi_cluster_reconcile_races.py - For this test I think it makes sense to add MongoDB-based multi-cluster RS as an additional resource into that test (in the feature branch) and not convert it into using a library.

• multicluster_om/multicluster_om_appdb_no_mesh.py and multicluster_appdb/multicluster_appdb_s3_based_backup_restore.py - I agree that it is better to be safe and have a respective tests for MongoDB-based multi-cluster RS. As an option we can still have shared folders (one in multicluster_om and one in multicluster_appdb) and add a new file on the same level as existing test for MongoDB-based multi-cluster RS. But I'm fine with just copying them.

[m1kola]

Just curious - why replace self with KubernetesTester? Also - would super() be better/worse here?

[filipcirtog]

thanks for highlighting this. i have reverted such changes now.

[m1kola]

Wondering if something like this make it less boilerplate for class-based tests?

class TestWebhookValidation(testhelper.TestWebhookValidation):
    pass

Or even maybe (as a more generic for both class and function based tests)

TestWebhookValidation = pytest.mark.e2e_mongodbmulticluster_multi_cluster_validation(testhelper.TestWebhookValidation)

Or does it not work because of fixtures?

[filipcirtog]

i would say this does not work since there is no way to inject the other fixtures/arguments

[m1kola]

I think we will also need an equivalent test for MongoDB-based multi-cluster RS. So would it make sense to also make it shared?

[filipcirtog]

i have made a shared library for this as well. thank you!

[Julien-Ben]

The PR looks good overall ! I haven't identified any significant issues.
One nit: since we are repeating mongodbmulticluster many times, it may be better to abbreviate it. As long as we find another suitable abbreviation for the future test functions.

I will let external reviewers take a look and not approve for the moment, since it is a significant change to our test structure.

[anandsyncs]

Thanks for explaining the crux on the call.

I am fine with these changes considering that the shared test approach will be revisited when MongoDBMultiCluster is removed.

[lsierant]

The overall approach LGTM, but due to sheer volume it wasn't able to verify all the diffs.

Let's ensure we have the same number of e2e tests running and their running time is similar to the "previous" runs. This way we'll ensure we're running the same tests after extracting them to the shared library code.

[m1kola]

I've just converted this PR into a draft to prevent accidental merge. After a sync with PM we've decided to review the bigger picture and for now we want to hold the merge.