diff --git a/content/conformance-ee/_index.en.md b/content/conformance-ee/_index.en.md new file mode 100644 index 000000000..8141d8f6b --- /dev/null +++ b/content/conformance-ee/_index.en.md @@ -0,0 +1,6 @@ ++++ +title = "Conformance EE Docs" +sitemapexclude = true ++++ + +Conformance EE is a Ginkgo v2-based end-to-end conformance test framework for Kubermatic Enterprise Edition, automatically generating, running, and reporting thousands of test scenarios across cloud providers. diff --git a/content/conformance-ee/main/_index.en.md b/content/conformance-ee/main/_index.en.md new file mode 100644 index 000000000..f0de578a1 --- /dev/null +++ b/content/conformance-ee/main/_index.en.md @@ -0,0 +1,49 @@ ++++ +title = "Kubermatic Conformance EE" +date = 2026-03-17T10:07:15+02:00 +weight = 7 +description = "Learn how to use Kubermatic Conformance EE to automatically generate, run, and report end-to-end conformance test scenarios for Kubermatic Kubernetes Platform clusters." ++++ + +## What is Conformance EE? + +[Conformance EE](https://github.com/kubermatic/conformance-ee) is a project by Kubermatic that provides an automated end-to-end conformance test framework for **Kubermatic Kubernetes Platform (KKP) Enterprise Edition**. It automatically generates, runs, and reports thousands of test scenarios by combinatorially applying configuration modifiers to cloud provider clusters and machine deployments. + +## Motivation and Background + +Validating Kubernetes cluster configurations across multiple cloud providers, OS distributions, Kubernetes versions, and feature combinations is a complex and error-prone task when done manually. The number of possible configuration permutations grows exponentially, making exhaustive manual testing impractical. + +Conformance EE solves this by: + +- **Combinatorially generating test scenarios** from cluster modifiers (CNI, proxy mode, expose strategy, etc.) and machine modifiers (CPU, memory, disk, OS distribution, etc.) +- **Deduplicating clusters** using SHA-256 hashing to avoid creating redundant clusters when different modifier combinations produce the same effective configuration +- **Parallelizing test execution** across Ginkgo nodes with concurrent worker pools +- **Providing live visibility** into test progress via Kubernetes ConfigMap reporting and JUnit XML output +- **Offering an interactive TUI** for configuring and launching test runs directly from the terminal + +## Key Features + +- **Scenario Matrix Generation**: Combinatorial generation of thousands of test scenarios from provider-discovered settings and user configuration +- **SHA-256 Deduplication**: Prevents duplicate cluster creation by hashing sanitized cluster specs +- **Parallel Execution**: 100 concurrent workers across Ginkgo nodes with max 4 clusters created in parallel +- **Multi-Provider Support**: Currently supports KubeVirt with extensible provider architecture +- **Interactive Terminal UI**: Built with Bubble Tea for configuration and execution management +- **In-Cluster Deployment**: Runs as Kubernetes Jobs with ConfigMap-based live reporting +- **JUnit XML Reports**: Compatible with all major CI systems (Jenkins, GitLab CI, GitHub Actions) +- **YAML Configuration**: Flexible configuration system for providers, releases, distributions, resources, and timeouts + +## Table of Content + +{{% children depth=5 %}} +{{% /children %}} + +## Further Information + +- [Conformance EE GitHub Repository](https://github.com/kubermatic/conformance-ee) +- [Kubermatic Kubernetes Platform](https://www.kubermatic.com/products/kubermatic-kubernetes-platform/) + +Visit [kubermatic.com](https://www.kubermatic.com/) for further information. + +{{% notice tip %}} +For latest updates follow us on Twitter [@Kubermatic](https://twitter.com/Kubermatic) +{{% /notice %}} diff --git a/content/conformance-ee/main/architecture/_index.en.md b/content/conformance-ee/main/architecture/_index.en.md new file mode 100644 index 000000000..24d630fbc --- /dev/null +++ b/content/conformance-ee/main/architecture/_index.en.md @@ -0,0 +1,161 @@ ++++ +title = "Architecture" +date = 2026-03-17T10:07:15+02:00 +weight = 5 ++++ + +Conformance EE is a Ginkgo v2-based test framework that follows a three-phase pattern: **Build**, **Execute**, and **Report**. It uses combinatorial scenario generation, SHA-256 deduplication, and parallel execution to efficiently validate KKP cluster configurations. + +## High-Level Architecture + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ Ginkgo Test Suite │ +│ provider/kubevirt/provider_suite_test.go │ +│ provider/kubevirt/provider_kubevirt_test.go │ +└──────────────────────┬──────────────────────────────────────────┘ + │ calls + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ build.GetTableEntries() │ +│ Combinatorially generates all test scenarios │ +│ Deduplicates using SHA-256 of sanitized cluster specs │ +└──────────────────────┬──────────────────────────────────────────┘ + │ uses + ┌────────────┼────────────┐ + ▼ ▼ ▼ + ┌──────────┐ ┌──────────┐ ┌──────────────┐ + │settings/ │ │options/ │ │build/ │ + │cluster.go│ │options.go│ │provider/ │ + │kubevirt │ │ │ │kubevirt.go │ + └──────────┘ └──────────┘ └──────────────┘ + + Parallel Ginkgo nodes + ┌─────────────────────────────────┐ + │ Node 1 Node 2 ... Node N │ + │ cluster/ utils/ │ + │ ensure.go provider_suite_ │ + │ update.go utils.go │ + └─────────────────────────────────┘ + + Reporting + ┌─────────────────────────────────┐ + │ utils/junit_reporter.go │ + │ utils/reports_configmap.go │ + └─────────────────────────────────┘ +``` + +## Three-Phase Pattern + +### 1. Build Phase + +The `build.GetTableEntries()` function generates the complete scenario matrix before any test spec runs: + +1. **Discover Infrastructure** — Calls the provider's `DiscoverDefaultDatacenterSettings()` to enumerate VPCs, subnets, storage classes, instance types, etc. from the live infrastructure cluster. +2. **Generate Cluster Specs** — Combinatorially applies cluster modifiers (CNI, proxy mode, expose strategy, etc.) and datacenter modifiers (network, storage class). +3. **Deduplicate Cluster Specs** — Serializes each sanitized `ClusterSpec` to JSON, hashes with SHA-256, and uses a `(dcHash[:6], specHash[:6], k8sVersion)` key to identify truly distinct clusters. +4. **Generate Machine Specs** — For each unique cluster, combinatorially applies machine modifiers (CPU, memory, disk, OS distribution, image source, DNS policy, eviction strategy, etc.). +5. **Deduplicate Machine Specs** — Same SHA-256 approach applied to machine specs. + +The result is a `map[string]*Scenario` where each key is a cluster dedup key and each `Scenario` contains the cluster spec plus all machines to test against it. + +### 2. Execute Phase + +Test scenarios are executed in parallel across Ginkgo nodes: + +- **Suite Setup** (`SynchronizedBeforeSuite`): Expensive one-time operations (project creation, cluster setup) run only on the first parallel Ginkgo node. Results are shared via `[]byte` to all nodes. +- **Spec Execution**: Each Ginkgo node picks up specs from the scenario table and runs the full machine deployment lifecycle. +- **Suite Teardown** (`SynchronizedAfterSuite`): Cluster deletion and report publishing run only on the first node after all others finish. + +### 3. Report Phase + +- **JUnit XML**: Per-spec JUnit XML files written to the reports directory, consumable by any CI system. +- **ConfigMap Live Reporting**: After each spec, results are patched into a shared Kubernetes ConfigMap using JSON merge patches for live visibility into test progress. + +## Components + +### Scenario Generator (`build/`) + +The scenario generator is the core engine that produces the test matrix. It uses 100 concurrent workers to parallelize cluster and machine spec generation: + +``` +Discover Infrastructure + ↓ (VPCs, subnets, storage classes, instance types) +Generate Cluster Specs + ↓ (apply 28 cluster modifiers across 17 groups) +Deduplicate Cluster Specs + ↓ (SHA-256 hash: dcNameHash[:6] + specHash[:6] + k8sVersion) +For Each Unique Cluster: + Generate Machine Specs + ↓ (apply machine modifiers: CPU, memory, disk, OS, DNS, etc.) + Deduplicate Machine Specs + ↓ (SHA-256 hash of sanitized spec + machineName) +Return Map[clusterDedupKey]*Scenario +``` + +### Cluster Lifecycle Manager (`cluster/`) + +Handles the full lifecycle of KKP clusters: + +- **Creation**: Create cluster resource, wait for reconciliation (10 min), validate control plane health, add cleanup finalizers, run smoke tests +- **Upgrade**: Patch Kubernetes version, wait for reconciliation, re-run health checks +- **Deletion**: Delete with 25 min timeout, wait for cleanup finalizer-based PV/LB deletion + +### Machine Deployment Manager (`utils/`) + +Manages machine deployments within user clusters: + +- **Setup**: Create `MachineDeployment`, attach OSP annotations, wait for node references, label nodes, wait for Ready state and pod readiness +- **Update**: Patch kubelet version, wait for rollout, verify new nodes + +### Interactive TUI (`ui/`) + +A Bubble Tea-based terminal interface that guides users through: + +1. Environment selection (local or existing cluster) +2. Provider selection (currently KubeVirt) +3. Kubeconfig and credential configuration +4. Kubernetes version, distribution, datacenter, and modifier selection +5. Test execution and live monitoring + +### Deployment Engine (`deploy/`) + +Creates Kubernetes resources for in-cluster test execution: + +- Kubernetes client setup from kubeconfig +- Namespace and RBAC (cluster-admin) creation +- ConfigMap for provider configuration +- Secret for kubeconfig credentials +- Job definition with mounted config and credentials + +## Directory Structure + +``` +conformance-ee/ +├── cmd/ # CLI entry point +│ └── main.go +├── deploy/ # Kubernetes deployment utilities +│ └── kubernetes.go +├── tests/ # Core test framework +│ ├── build/ # Scenario generation engine +│ │ ├── build.go # GetTableEntries(), worker pools +│ │ ├── scenario.go # Entry point for scenario generation +│ │ ├── types.go # Core types (Scenario, clusterJob) +│ │ ├── labels.go # Ginkgo label helpers +│ │ ├── provider.go # Provider-agnostic building +│ │ └── provider/ # Provider implementations +│ ├── cluster/ # Cluster lifecycle +│ │ ├── ensure.go # Creation and health validation +│ │ └── update.go # Version upgrade logic +│ ├── options/ # Configuration loading +│ │ └── options.go # YAML config and runtime options +│ ├── settings/ # Modifier definitions +│ │ ├── types.go # Core modifier types +│ │ ├── cluster.go # 28 cluster modifiers +│ │ └── kubevirt.go # KubeVirt machine/DC modifiers +│ ├── utils/ # Test utilities and reporters +│ └── provider/kubevirt/ # Ginkgo test suite +├── ui/ # Interactive TUI +├── Dockerfile.ginkgo # Multi-stage Docker build +└── go.mod # Go module dependencies +``` diff --git a/content/conformance-ee/main/compatibility-matrix/_index.en.md b/content/conformance-ee/main/compatibility-matrix/_index.en.md new file mode 100644 index 000000000..24f04aa04 --- /dev/null +++ b/content/conformance-ee/main/compatibility-matrix/_index.en.md @@ -0,0 +1,38 @@ ++++ +title = "Compatibility Matrix" +date = 2026-03-17T10:07:15+02:00 +weight = 35 ++++ + +This matrix reflects the tested combinations of Conformance EE with Kubermatic Kubernetes Platform and Kubernetes versions. + +## Component Compatibility + +| Conformance EE | KKP Version | Kubernetes | Go | Ginkgo | +|----------------|-------------|------------|----|--------| +| main | v2.30 | v1.31, v1.32 | 1.25 | v2.28.1 | + +## Supported Cloud Providers + +| Provider | Status | Notes | +|----------|--------|-------| +| KubeVirt | Supported | Full infrastructure discovery and scenario generation | + +## Supported OS Distributions + +| Distribution | Versions | KubeVirt | +|-------------|----------|----------| +| Ubuntu | 20.04, 22.04 | ✔️ | +| RHEL | 8, 9 | ✔️ | +| Flatcar | 3374.2.2 | ✔️ | +| Rocky Linux | 8, 9 | ✔️ | + +## Key Dependencies + +| Dependency | Version | +|------------|---------| +| Kubermatic SDK | v2.30.0 | +| k8s.io/client-go | v0.35.2 | +| KubeVirt API | v1.3.1 | +| Bubble Tea | v1.3.10 | +| Gomega | v1.39.0 | diff --git a/content/conformance-ee/main/configuration/_index.en.md b/content/conformance-ee/main/configuration/_index.en.md new file mode 100644 index 000000000..5ba13e7d1 --- /dev/null +++ b/content/conformance-ee/main/configuration/_index.en.md @@ -0,0 +1,12 @@ ++++ +title = "Configuration" +date = 2026-03-17T10:07:15+02:00 +weight = 20 ++++ + +Conformance EE is configured via a YAML file pointed to by the `CONFORMANCE_TESTER_CONFIG_FILE` environment variable. If not set, built-in defaults are used. + +## Table of Content + +{{% children depth=5 %}} +{{% /children %}} diff --git a/content/conformance-ee/main/configuration/config-file/_index.en.md b/content/conformance-ee/main/configuration/config-file/_index.en.md new file mode 100644 index 000000000..d6d96470e --- /dev/null +++ b/content/conformance-ee/main/configuration/config-file/_index.en.md @@ -0,0 +1,197 @@ ++++ +title = "Configuration File" +date = 2026-03-17T10:07:15+02:00 +weight = 10 ++++ + +The primary configuration is a YAML file that controls which providers, Kubernetes versions, OS distributions, and resource sizes are included in the test matrix. + +## Full Configuration Reference + +```yaml +# Cloud providers to test (must match provider name) +providers: + - kubevirt + +# Kubernetes versions to test +releases: + - "1.31" + - "1.32" + +# OS distributions to include +enableDistributions: + - ubuntu + - flatcar + +# OS distributions to exclude (takes precedence over enableDistributions) +excludeDistributions: [] + +# Fine-grained modifier filtering +included: + datacenterDescriptions: [] + clusterDescriptions: + - "with cni plugin set to canal" + machineDescriptions: [] + +excluded: + clusterDescriptions: + - "with mla-logging enabled" + +# Resource sizes for the combinatorial matrix +resources: + cpu: [2, 4] + memory: ["4Gi", "8Gi"] + diskSize: ["20Gi", "50Gi"] + +# OS image sources per distribution and version +imageSources: + ubuntu: + "22.04": "docker://quay.io/kubermatic-virt-disks/ubuntu:22.04" + flatcar: + "3374.2.2": "docker://quay.io/kubermatic-virt-disks/flatcar:3374.2.2" + +# Operating System Profile annotations +ospAnnotations: + "k8c.io/operating-system-profile": "osp-ubuntu" + +# Custom DNS nameservers +nameservers: + - "8.8.8.8" + - "8.8.4.4" + +# Container runtime settings applied to all nodes +nodeSettings: + insecureRegistries: + - "registry.local:5000" + registryMirrors: + - "https://mirror.gcr.io" + pauseImage: "registry.k8s.io/pause:3.9" + +# Reporting +resultsFile: /reports/results.json +reportsRoot: /reports + +# Timeouts +controlPlaneReadyWaitTimeout: 10m +nodeReadyTimeout: 20m +customTestTimeout: 10m +nodeCount: 1 +``` + +## Configuration Sections + +### Providers + +```yaml +providers: + - kubevirt +``` + +Specifies which cloud providers to test. Each provider must have a corresponding implementation in the `build/provider/` directory. + +| Provider | Status | +|----------|--------| +| `kubevirt` | Supported | + +### Releases + +```yaml +releases: + - "1.31" + - "1.32" +``` + +Kubernetes versions to include in the test matrix. Each version is combined with every provider, distribution, and modifier combination. + +### Distributions + +```yaml +enableDistributions: + - ubuntu + - flatcar +excludeDistributions: [] +``` + +Controls which OS distributions are tested. `excludeDistributions` takes precedence over `enableDistributions`. + +### Included/Excluded Modifiers + +```yaml +included: + datacenterDescriptions: [] + clusterDescriptions: + - "with cni plugin set to canal" + machineDescriptions: [] +excluded: + clusterDescriptions: + - "with mla-logging enabled" +``` + +Filters the generated scenario matrix by modifier descriptions. Use this to focus tests on specific configurations or exclude known-problematic combinations. + +- **included**: If non-empty, only scenarios matching at least one included description are kept. +- **excluded**: Scenarios matching any excluded description are removed (applied after includes). + +### Resources + +```yaml +resources: + cpu: [2, 4] + memory: ["4Gi", "8Gi"] + diskSize: ["20Gi", "50Gi"] +``` + +Resource sizes for machine deployments. Each combination of CPU, memory, and disk size produces a separate machine modifier in the test matrix. + +### Image Sources + +```yaml +imageSources: + ubuntu: + "22.04": "docker://quay.io/kubermatic-virt-disks/ubuntu:22.04" + flatcar: + "3374.2.2": "docker://quay.io/kubermatic-virt-disks/flatcar:3374.2.2" +``` + +Maps OS distribution names and versions to container image URIs used for KubeVirt virtual machine disks. + +#### Default Image Sources + +When no `imageSources` are configured, the following defaults are used: + +| Distribution | Version | Source | +|--------------|----------|-----------------------------------------------------| +| ubuntu | 20.04 | `docker://quay.io/kubermatic-virt-disks/ubuntu:20.04` | +| ubuntu | 22.04 | `docker://quay.io/kubermatic-virt-disks/ubuntu:22.04` | +| rhel | 8 | `docker://quay.io/kubermatic-virt-disks/rhel:8` | +| rhel | 9 | `docker://quay.io/kubermatic-virt-disks/rhel:9` | +| flatcar | 3374.2.2 | `docker://quay.io/kubermatic-virt-disks/flatcar:3374.2.2` | +| rockylinux | 8 | `docker://quay.io/kubermatic-virt-disks/rocky:8` | +| rockylinux | 9 | `docker://quay.io/kubermatic-virt-disks/rocky:9` | + +### Node Settings + +```yaml +nodeSettings: + insecureRegistries: + - "registry.local:5000" + registryMirrors: + - "https://mirror.gcr.io" + pauseImage: "registry.k8s.io/pause:3.9" +``` + +Container runtime settings applied to all worker nodes in created clusters. + +### Timeouts + +```yaml +controlPlaneReadyWaitTimeout: 10m +nodeReadyTimeout: 20m +customTestTimeout: 10m +``` + +| Timeout | Default | Description | +|---------|---------|-------------| +| `controlPlaneReadyWaitTimeout` | `10m` | Time to wait for the control plane to become healthy | +| `nodeReadyTimeout` | `20m` | Time to wait for worker nodes to reach Ready state | +| `customTestTimeout` | `10m` | Timeout for individual custom test cases | diff --git a/content/conformance-ee/main/configuration/runtime-flags/_index.en.md b/content/conformance-ee/main/configuration/runtime-flags/_index.en.md new file mode 100644 index 000000000..fe9cd9fbd --- /dev/null +++ b/content/conformance-ee/main/configuration/runtime-flags/_index.en.md @@ -0,0 +1,80 @@ ++++ +title = "Runtime Flags" +date = 2026-03-17T10:07:15+02:00 +weight = 20 ++++ + +In addition to the YAML configuration file, Conformance EE supports command-line flags for controlling test execution behavior. + +## Flags Reference + +| Flag | Default | Description | +|------|---------|-------------| +| `--datacenters` | `""` | Comma-separated list of datacenters to test | +| `--kube-versions` | `""` | Comma-separated Kubernetes versions to test | +| `--skip-cluster-creation` | `false` | Skip cluster creation; use existing clusters | +| `--skip-cluster-deletion` | `false` | Keep clusters after tests (useful for debugging) | +| `--update-clusters` | `false` | Upgrade existing clusters before running tests | + +## Usage Examples + +### Test Specific Datacenters + +```bash +./e2e.test --datacenters=dc-1,dc-2 +``` + +### Test Specific Kubernetes Versions + +```bash +./e2e.test --kube-versions=1.31,1.32 +``` + +### Debug Mode (Keep Clusters) + +When debugging test failures, use `--skip-cluster-deletion` to keep clusters alive for investigation: + +```bash +./e2e.test --skip-cluster-deletion +``` + +### Reuse Existing Clusters + +If clusters from a previous run are still available, skip creation: + +```bash +./e2e.test --skip-cluster-creation +``` + +### Upgrade and Test + +Upgrade existing clusters to the next Kubernetes version before running tests: + +```bash +./e2e.test --update-clusters --skip-cluster-creation +``` + +## Ginkgo Flags + +Since Conformance EE uses Ginkgo v2, all standard Ginkgo flags are also available: + +| Flag | Description | +|------|-------------| +| `--ginkgo.v` | Verbose output | +| `--ginkgo.nodes=N` | Number of parallel Ginkgo nodes | +| `--ginkgo.focus="pattern"` | Run only specs matching the regex pattern | +| `--ginkgo.skip="pattern"` | Skip specs matching the regex pattern | +| `--ginkgo.label-filter="expression"` | Filter specs by Ginkgo labels | +| `--ginkgo.dry-run` | List all specs without executing them | + +### Example: Run Only Canal CNI Tests + +```bash +./e2e.test --ginkgo.focus="canal" +``` + +### Example: Run 8 Parallel Nodes + +```bash +./e2e.test --ginkgo.nodes=8 +``` diff --git a/content/conformance-ee/main/installation/_index.en.md b/content/conformance-ee/main/installation/_index.en.md new file mode 100644 index 000000000..4ecf211a2 --- /dev/null +++ b/content/conformance-ee/main/installation/_index.en.md @@ -0,0 +1,16 @@ ++++ +title = "Installation" +date = 2026-03-17T10:07:15+02:00 +weight = 15 ++++ + +This chapter covers how to install and run Conformance EE in your environment. + +{{% notice tip %}} +It is recommended to first familiarize yourself with our [architecture documentation]({{< ref "../architecture/" >}}). +{{% /notice %}} + +## Table of Content + +{{% children depth=5 %}} +{{% /children %}} diff --git a/content/conformance-ee/main/installation/container-image/_index.en.md b/content/conformance-ee/main/installation/container-image/_index.en.md new file mode 100644 index 000000000..8162c7838 --- /dev/null +++ b/content/conformance-ee/main/installation/container-image/_index.en.md @@ -0,0 +1,56 @@ ++++ +title = "Container Image" +date = 2026-03-17T10:07:15+02:00 +weight = 20 ++++ + +Conformance EE is distributed as a container image and can be run either as a Kubernetes Job or locally via the interactive TUI. + +## Container Image + +The official container image is available at: + +``` +quay.io/kubermatic/conformance-ee +``` + +### Tags + +| Tag | Description | +|-----|-------------| +| `latest` | Latest release | +| `v*` | Specific version (e.g., `v1.0.0`) | + +### Pulling the Image + +```bash +docker pull quay.io/kubermatic/conformance-ee:latest +``` + +## Building from Source + +### Build the Container Image + +```bash +git clone https://github.com/kubermatic/conformance-ee.git +cd conformance-ee + +docker build -f Dockerfile.ginkgo \ + --build-arg IMAGE_TAG="quay.io/kubermatic/conformance-ee:dev" \ + -t quay.io/kubermatic/conformance-ee:dev . +``` + +The multi-stage Dockerfile: + +1. **Build stage** (`golang:1.25`): Downloads Go modules, installs Ginkgo v2, compiles the test binary and conformance tester +2. **Runtime stage** (`distroless/base-debian12`): Minimal image containing only the compiled binaries + +### Build the Binary Locally + +```bash +go build -o conformance-tester ./cmd/ +``` + +{{% notice note %}} +The locally built binary provides the interactive TUI for configuring and launching tests, but the actual test execution still runs as a Kubernetes Job using the container image. +{{% /notice %}} diff --git a/content/conformance-ee/main/installation/deployment/_index.en.md b/content/conformance-ee/main/installation/deployment/_index.en.md new file mode 100644 index 000000000..0d1bbe40b --- /dev/null +++ b/content/conformance-ee/main/installation/deployment/_index.en.md @@ -0,0 +1,129 @@ ++++ +title = "In-Cluster Deployment" +date = 2026-03-17T10:07:15+02:00 +weight = 30 ++++ + +Conformance EE is designed to run as a Kubernetes Job inside a cluster with access to the KKP API. The interactive TUI handles resource creation, but you can also deploy manually. + +## Automated Deployment via TUI + +The simplest way to deploy is using the interactive terminal UI: + +```bash +./conformance-tester +``` + +The TUI will guide you through: + +1. Selecting the environment (local or existing cluster) +2. Choosing cloud providers to test +3. Configuring kubeconfig and credentials +4. Selecting Kubernetes versions, OS distributions, and datacenters +5. Deploying the test Job + +## Manual Deployment + +### 1. Create the Namespace + +```bash +kubectl create namespace conformance-tests +``` + +### 2. Create RBAC + +The conformance tester needs `cluster-admin` privileges: + +```bash +kubectl create clusterrolebinding conformance-tests-admin \ + --clusterrole=cluster-admin \ + --serviceaccount=conformance-tests:default +``` + +### 3. Create the Configuration ConfigMap + +Create a `config.yaml` with your test configuration (see [Configuration]({{< ref "../../configuration/" >}})) and store it in a ConfigMap: + +```bash +kubectl create configmap conformance-config \ + --from-file=config.yaml=./config.yaml \ + -n conformance-tests +``` + +### 4. Create the Kubeconfig Secret + +```bash +kubectl create secret generic conformance-kubeconfig \ + --from-file=kubeconfig=./kubeconfig \ + -n conformance-tests +``` + +### 5. Deploy the Job + +```yaml +apiVersion: batch/v1 +kind: Job +metadata: + name: conformance-tests + namespace: conformance-tests +spec: + backoffLimit: 0 + template: + spec: + restartPolicy: Never + containers: + - name: conformance-tester + image: quay.io/kubermatic/conformance-ee:latest + args: + - /e2e.test + - --ginkgo.v + volumeMounts: + - name: config + mountPath: /opt/config.yaml + subPath: config.yaml + - name: kubeconfig + mountPath: /opt/kubeconfig + subPath: kubeconfig + env: + - name: CONFORMANCE_TESTER_CONFIG_FILE + value: /opt/config.yaml + - name: KUBECONFIG + value: /opt/kubeconfig + volumes: + - name: config + configMap: + name: conformance-config + - name: kubeconfig + secret: + secretName: conformance-kubeconfig +``` + +```bash +kubectl apply -f conformance-job.yaml +``` + +### 6. Monitor Progress + +View live test progress via the ConfigMap reporter: + +```bash +kubectl get configmap -n conformance-tests -l app=conformance-reports -o yaml +``` + +View Job logs: + +```bash +kubectl logs -n conformance-tests job/conformance-tests -f +``` + +## Resource Requirements + +| Resource | Minimum | Recommended | +|----------|---------|-------------| +| CPU | 500m | 1000m | +| Memory | 512Mi | 1Gi | +| Disk | - | - | + +{{% notice warning %}} +Ensure the cluster has enough capacity for the conformance tester pod and that network policies allow access to the KKP API and cloud provider endpoints. +{{% /notice %}} diff --git a/content/conformance-ee/main/installation/prerequisites/_index.en.md b/content/conformance-ee/main/installation/prerequisites/_index.en.md new file mode 100644 index 000000000..cb83dab1a --- /dev/null +++ b/content/conformance-ee/main/installation/prerequisites/_index.en.md @@ -0,0 +1,57 @@ ++++ +title = "Prerequisites" +date = 2026-03-17T10:07:15+02:00 +weight = 10 ++++ + +Before installing Conformance EE, ensure the following prerequisites are met. + +## Requirements + +### Kubermatic Kubernetes Platform + +Conformance EE is designed to test clusters managed by **Kubermatic Kubernetes Platform (KKP) Enterprise Edition**. You need: + +- A running KKP installation (v2.25+) +- Access to the KKP API (admin or project-level credentials) +- A configured seed cluster with at least one cloud provider datacenter + +### Kubernetes Cluster + +A Kubernetes cluster is required to run Conformance EE as a Job: + +- Kubernetes v1.27+ +- `cluster-admin` privileges (or ability to create ClusterRoleBindings) +- Sufficient resources to run the conformance tester pod + +### Cloud Provider + +Currently, the following cloud providers are supported: + +| Provider | Status | +|----------|--------| +| KubeVirt | Supported | + +{{% notice note %}} +Additional providers can be added by implementing the `Provider` interface. See the [architecture documentation]({{< ref "../../architecture/" >}}) for details. +{{% /notice %}} + +### Tools + +For local development and building: + +| Tool | Version | Purpose | +|------|---------|---------| +| Go | 1.25+ | Building the conformance tester binary | +| Docker | 20.10+ | Building the container image | +| kubectl | v1.27+ | Interacting with Kubernetes clusters | +| Ginkgo | v2.28+ | Running tests locally (optional) | + +## Network Access + +The conformance tester requires network access to: + +- **KKP API**: To create and manage clusters +- **Kubernetes API**: Of the seed cluster for job deployment and ConfigMap reporting +- **Container Registry**: `quay.io/kubermatic/conformance-ee` for pulling the test image +- **Infrastructure Provider APIs**: For cluster provisioning (e.g., KubeVirt API) diff --git a/content/conformance-ee/main/references/_index.en.md b/content/conformance-ee/main/references/_index.en.md new file mode 100644 index 000000000..bd76fc2b3 --- /dev/null +++ b/content/conformance-ee/main/references/_index.en.md @@ -0,0 +1,12 @@ ++++ +title = "References" +date = 2026-03-17T10:07:15+02:00 +weight = 50 ++++ + +This section contains reference documentation for Conformance EE settings, modifiers, and providers. + +## Table of Content + +{{% children depth=5 %}} +{{% /children %}} diff --git a/content/conformance-ee/main/references/cluster-modifiers/_index.en.md b/content/conformance-ee/main/references/cluster-modifiers/_index.en.md new file mode 100644 index 000000000..67cbdf824 --- /dev/null +++ b/content/conformance-ee/main/references/cluster-modifiers/_index.en.md @@ -0,0 +1,53 @@ ++++ +title = "Cluster Modifiers" +date = 2026-03-17T10:07:15+02:00 +weight = 10 ++++ + +Cluster modifiers define the configuration axes for Kubernetes clusters created during conformance testing. Each modifier belongs to a **group**, and within each group only one modifier is active per cluster. This creates an exclusive selection axis for generating the cluster matrix. + +## Modifier Groups + +| Group | Options | Description | +|--------------------|------------------------------------------------|-------------| +| `cni` | canal, cilium | Container Network Interface plugin | +| `expose-strategy` | NodePort, LoadBalancer, Tunneling | How the cluster API is exposed | +| `proxy-mode` | ipvs, iptables, ebpf | Kube-proxy mode | +| `audit` | enabled, disabled | Kubernetes audit logging | +| `ssh` | enabled, disabled | SSH key agent | +| `opa` | enabled, disabled | Open Policy Agent integration | +| `mla-monitoring` | enabled, disabled | MLA monitoring stack | +| `mla-logging` | enabled, disabled | MLA logging stack | +| `node-local-dns` | enabled, disabled | Node-local DNS cache | +| `k8s-dashboard` | enabled, disabled | Kubernetes Dashboard | +| `pod-node-selector`| enabled, disabled | PodNodeSelector admission controller | +| `event-rate-limit` | enabled, disabled | EventRateLimit admission controller | +| `csi-driver` | enabled, disabled | CSI driver integration | +| `update-window` | configured, none | Maintenance window (excluded from dedup) | +| `oidc` | disabled | OIDC configuration (excluded from dedup) | +| `external-ccm` | enabled, disabled | External Cloud Controller Manager | +| `ipvs-strict-arp` | enabled, disabled | IPVS strict ARP mode | + +## Deduplication + +Cluster modifiers are deduplicated using SHA-256 hashing. Two modifier combinations that produce the same effective `ClusterSpec` will share the same cluster, avoiding redundant cluster creation. + +Some groups are **excluded from the dedup hash** because they don't change the cluster's functional behavior: + +- `update-window` — Maintenance windows don't affect cluster behavior +- `oidc` — OIDC is orthogonal to the provider/version matrix + +## Modifier Type + +```go +type CloudSpecModifier struct { + Name string + Group string + Modify func(spec *kubermaticv1.CloudSpec) +} +``` + +Each modifier has: +- **Name**: Human-readable description (e.g., "with cni plugin set to canal") +- **Group**: Exclusive selection axis +- **Modify**: Function that mutates the cluster spec diff --git a/content/conformance-ee/main/references/kubevirt-provider/_index.en.md b/content/conformance-ee/main/references/kubevirt-provider/_index.en.md new file mode 100644 index 000000000..ef46c122b --- /dev/null +++ b/content/conformance-ee/main/references/kubevirt-provider/_index.en.md @@ -0,0 +1,59 @@ ++++ +title = "KubeVirt Provider" +date = 2026-03-17T10:07:15+02:00 +weight = 30 ++++ + +The KubeVirt provider is the reference implementation for Conformance EE's provider interface. It handles infrastructure discovery and spec generation for KubeVirt-based clusters. + +## Infrastructure Discovery + +The provider's `DiscoverDefaultDatacenterSettings()` function queries the KubeVirt infrastructure cluster to enumerate available resources: + +| Resource | API Group | Description | +|----------|-----------|-------------| +| VPCs | `kubeovn.io/v1` | Virtual private cloud networks | +| Subnets | `kubeovn.io/v1` | Network subnets (paired with parent VPC) | +| StorageClasses | `storage.k8s.io/v1` | Available storage backends | +| Instance Types | `instancetype.kubevirt.io/v1beta1` | `VirtualMachineInstancetype` resources | +| Preferences | `instancetype.kubevirt.io/v1beta1` | `VirtualMachinePreference` resources | + +These discovered resources are converted into dynamic modifiers that expand the test matrix based on what's actually available in the target environment. + +## Provider Interface + +To add a new provider, implement the following interface: + +```go +type Provider interface { + // DiscoverDefaultDatacenterSettings discovers infrastructure resources + // from the live provider cluster and returns datacenter settings. + DiscoverDefaultDatacenterSettings( + ctx context.Context, + providerConfig *providerconfig.Config, + secrets legacytypes.Secrets, + ) (*settings.DefaultDatacenterSettings, error) + + // GetClusterModifiers returns provider-specific cluster modifiers. + GetClusterModifiers() []settings.CloudSpecModifier + + // GetMachineModifiers returns provider-specific machine modifiers. + GetMachineModifiers(opts options.Options) []settings.MachineSpecModifier + + // GetDatacenterModifiers returns provider-specific datacenter modifiers. + GetDatacenterModifiers() []settings.DatacenterSetting +} +``` + +## Supported Distributions + +The KubeVirt provider supports the following OS distributions for virtual machine disks: + +| Distribution | Supported Versions | +|--------------|--------------------| +| Ubuntu | 20.04, 22.04 | +| RHEL | 8, 9 | +| Flatcar | 3374.2.2 | +| Rocky Linux | 8, 9 | + +Custom image sources can be configured via the `imageSources` field in the configuration file. diff --git a/content/conformance-ee/main/references/machine-modifiers/_index.en.md b/content/conformance-ee/main/references/machine-modifiers/_index.en.md new file mode 100644 index 000000000..a83958a70 --- /dev/null +++ b/content/conformance-ee/main/references/machine-modifiers/_index.en.md @@ -0,0 +1,60 @@ ++++ +title = "Machine Modifiers" +date = 2026-03-17T10:07:15+02:00 +weight = 20 ++++ + +Machine modifiers define configuration axes for machine deployments within test clusters. They are split into **static** modifiers (hardcoded) and **dynamic** modifiers (discovered at runtime from the infrastructure cluster). + +## Static Machine Modifiers + +| Group | Options | Description | +|-------------------------------|------------------------------------------------|-------------| +| `instance-type` | none, discovered instance types | KubeVirt VM instance type | +| `preference` | none, discovered preferences | KubeVirt VM preference | +| `dns-policy` | ClusterFirstWithHostNet, ClusterFirst, Default, None | Pod DNS resolution policy | +| `eviction-strategy` | LiveMigrate, External | VM eviction behavior | +| `network-multiqueue` | enabled, disabled | Network multi-queue support | +| `topology-spread-constraints` | enabled, disabled | Pod topology spread | + +## Dynamic Machine Modifiers + +These modifiers are discovered at runtime from the YAML configuration and infrastructure cluster: + +| Group | Source | Description | +|--------------------|------------------------------------------------|-------------| +| `cpu` | `resources.cpu` config values | VM CPU count | +| `memory` | `resources.memory` config values | VM memory size | +| `disk-size` | `resources.diskSize` config values | VM disk size | +| `storage-class` | StorageClasses from infra cluster | Storage backend | +| `node-affinity` | Node names from infra cluster | Node placement | +| `image-source` | `imageSources` config per distribution | OS disk image | +| `nameservers` | `nameservers` config | DNS nameservers | + +## Datacenter Modifiers + +Discovered from the KubeVirt infrastructure cluster: + +| Group | Source | Description | +|--------------------------|------------------------------------------------|-------------| +| `vpc` | VPC CRDs (`kubeovn.io/v1`) | Virtual private cloud | +| `kubevirt-vpc-subnet` | Subnet CRDs per VPC | Network subnet | +| `kubevirt-storage-class` | StorageClasses from infra cluster | Storage class | + +## Modifier Type + +```go +type MachineSpecModifier[T any] struct { + Name string + Group string + Modify func(spec T) +} + +type DatacenterSetting struct { + Name string + Group string + Modifier func(dc *kubermaticv1.Datacenter) +} +``` + +Each modifier follows the same pattern as cluster modifiers: a **Name** for human identification, a **Group** for exclusive selection, and a **Modify**/**Modifier** function for spec mutation. diff --git a/content/conformance-ee/main/references/reporting/_index.en.md b/content/conformance-ee/main/references/reporting/_index.en.md new file mode 100644 index 000000000..2a5023ba4 --- /dev/null +++ b/content/conformance-ee/main/references/reporting/_index.en.md @@ -0,0 +1,85 @@ ++++ +title = "Reporting" +date = 2026-03-17T10:07:15+02:00 +weight = 40 ++++ + +Conformance EE provides two reporting mechanisms: **JUnit XML reports** for CI integration and **ConfigMap live reporting** for real-time test visibility. + +## JUnit XML Reports + +After each Ginkgo spec, a JUnit XML file is written to the configured `reportsRoot` directory. The XML structure reflects the Ginkgo `By()` steps within each spec as individual test cases. + +### Report Structure + +``` +reports/ +├── junit.with_kubernetes_1.31.1_and_ubuntu_22.04_and_canal.xml +├── junit.with_kubernetes_1.31.1_and_flatcar_3374.2.2_and_cilium.xml +└── ... +``` + +### CI Compatibility + +JUnit XML files are compatible with: + +| CI System | Configuration | +|-----------|---------------| +| GitHub Actions | `EnricoMi/publish-unit-test-result-action` | +| Jenkins | JUnit post-build action | +| GitLab CI | `artifacts.reports.junit` | +| Azure DevOps | PublishTestResults task | + +## ConfigMap Live Reporting + +While JUnit XML is only available after the suite completes, the ConfigMap reporter provides **live visibility** into test progress during execution. + +### How It Works + +After each spec on any Ginkgo node, `UpsertSpecReport()` patches a shared Kubernetes ConfigMap using JSON merge patches. This is safe for concurrent writes from multiple Ginkgo nodes. + +### ConfigMap Naming + +``` +conformance-reports- +``` + +The name is sanitized to meet Kubernetes naming rules (lowercase, alphanumeric, hyphens only, max 253 characters). + +### Spec Key Format + +Each spec gets a unique key derived from SHA-256 of the spec's full text: + +``` +spec- +``` + +### Spec Value Format + +```json +{ + "state": "passed", + "duration": "2m34s", + "text": "KubeVirt with kubernetes version 1.31.1 operating system set to ubuntu 22.04 ...", + "failure": "" +} +``` + +### Viewing Live Results + +```bash +# List all report ConfigMaps +kubectl get configmap -n conformance-tests -l app=conformance-reports + +# View detailed results +kubectl get configmap conformance-reports- \ + -n conformance-tests -o jsonpath='{.data}' | jq . +``` + +### Concurrency Safety + +The reporter uses Kubernetes JSON merge patches (`MergePatchType`) which are safely applied by the API server without read-modify-write races. A 5-attempt exponential backoff (100ms, 200ms, 300ms, ...) handles transient API errors. + +### Size Limits + +ConfigMaps in Kubernetes have a 1 MiB size limit. At suite completion, `PostProcessingSuite` reads all JUnit XML files and pushes them into the same ConfigMap, respecting this limit. diff --git a/content/conformance-ee/main/release-notes/_index.en.md b/content/conformance-ee/main/release-notes/_index.en.md new file mode 100644 index 000000000..312638c10 --- /dev/null +++ b/content/conformance-ee/main/release-notes/_index.en.md @@ -0,0 +1,30 @@ ++++ +title = "Release Notes" +date = 2026-03-17T10:07:15+02:00 +weight = 60 ++++ + +{{% notice warning %}} +This document is work in progress and might not be in a correct or up-to-date state. +{{% /notice %}} + +## Conformance EE Releases + +Release artifacts are published to: + +- **Container Image**: `quay.io/kubermatic/conformance-ee` +- **Source Code**: [GitHub Releases](https://github.com/kubermatic/conformance-ee/releases) + +### Latest (main) + +The `main` branch tracks the latest development. Tagged releases follow semantic versioning. + +#### Highlights + +- Ginkgo v2-based test framework with combinatorial scenario generation +- SHA-256 deduplication for efficient cluster reuse +- Interactive Bubble Tea TUI for test configuration +- KubeVirt provider with full infrastructure discovery +- In-cluster execution via Kubernetes Jobs +- JUnit XML and ConfigMap live reporting +- Support for multiple Kubernetes versions and OS distributions diff --git a/content/conformance-ee/main/support-policy/_index.en.md b/content/conformance-ee/main/support-policy/_index.en.md new file mode 100644 index 000000000..c3c0e763f --- /dev/null +++ b/content/conformance-ee/main/support-policy/_index.en.md @@ -0,0 +1,34 @@ ++++ +title = "Support Policy" +date = 2026-03-17T10:07:15+02:00 +weight = 40 ++++ + +Conformance EE is an Enterprise Edition tool available through Kubermatic. It is designed for use with **Kubermatic Kubernetes Platform (KKP) Enterprise Edition**. + +## Support Scope + +As a default, support covers the following: + +- Debugging issues related to Conformance EE test execution +- Enhancing documentation +- Fixing bugs that block test execution or produce incorrect results + +What is not covered: + +- Issues related to the underlying Kubernetes cluster infrastructure +- Cloud provider-specific issues outside of the Conformance EE test framework +- Custom modifications to the scenario generation engine +- Issues related to third-party CI/CD integrations + +## Supported Versions + +Conformance EE follows the KKP release cadence. The latest version of Conformance EE is tested against the latest supported KKP versions. + +| Conformance EE | Supported KKP Versions | +|----------------|----------------------| +| main | v2.30 | + +{{% notice info %}} +**Discover our enterprise-grade support offerings and customized solutions for your organization's needs. [Contact our solutions team](mailto:sales@kubermatic.com) to explore how we can help ensure your success.** +{{% /notice %}} diff --git a/content/conformance-ee/main/tutorials/_index.en.md b/content/conformance-ee/main/tutorials/_index.en.md new file mode 100644 index 000000000..63ffe2a12 --- /dev/null +++ b/content/conformance-ee/main/tutorials/_index.en.md @@ -0,0 +1,17 @@ ++++ +title = "Guides" +linkTitle = "Tutorials" +date = 2026-03-17T10:07:15+02:00 +description = "Step-by-step guides for common Conformance EE workflows and scenarios" +weight = 25 +chapter = true ++++ + +# Guides + +Step-by-step instructions for common Conformance EE tasks and scenarios. + +## Table of Content + +{{% children depth=5 %}} +{{% /children %}} diff --git a/content/conformance-ee/main/tutorials/ci-cd/_index.en.md b/content/conformance-ee/main/tutorials/ci-cd/_index.en.md new file mode 100644 index 000000000..90532dabc --- /dev/null +++ b/content/conformance-ee/main/tutorials/ci-cd/_index.en.md @@ -0,0 +1,111 @@ ++++ +title = "CI/CD Integration" +date = 2026-03-17T10:07:15+02:00 +weight = 30 ++++ + +This guide covers integrating Conformance EE into your CI/CD pipeline. + +## GitHub Actions + +Conformance EE provides a built-in GitHub Actions workflow for building and pushing the container image. You can extend this to also run conformance tests. + +### Image Build Workflow + +The project includes a workflow at `.github/workflows/build-and-push.yaml` that: + +1. Triggers on git tag push (`v*`) +2. Builds the multi-stage Docker image +3. Pushes to `quay.io/kubermatic/conformance-ee` with version and `latest` tags + +### Running Tests in CI + +Create a workflow that runs conformance tests against your KKP installation: + +```yaml +name: Conformance Tests +on: + schedule: + - cron: '0 2 * * 1' # Weekly on Monday at 2 AM + workflow_dispatch: + +jobs: + conformance: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Set up kubeconfig + run: | + mkdir -p ~/.kube + echo "${{ secrets.KUBECONFIG }}" | base64 -d > ~/.kube/config + + - name: Deploy Conformance Tests + run: | + kubectl create namespace conformance-tests --dry-run=client -o yaml | kubectl apply -f - + kubectl create configmap conformance-config \ + --from-file=config.yaml=./config.yaml \ + -n conformance-tests --dry-run=client -o yaml | kubectl apply -f - + kubectl apply -f conformance-job.yaml + + - name: Wait for Completion + run: | + kubectl wait --for=condition=complete \ + job/conformance-tests \ + -n conformance-tests \ + --timeout=3600s + + - name: Collect Reports + if: always() + run: | + kubectl cp conformance-tests/conformance-tests:/reports ./reports + + - name: Upload Test Results + if: always() + uses: actions/upload-artifact@v4 + with: + name: conformance-reports + path: reports/ +``` + +## JUnit Report Integration + +Conformance EE produces JUnit XML reports that are compatible with most CI systems: + +### GitHub Actions + +```yaml +- name: Publish Test Results + uses: EnricoMi/publish-unit-test-result-action@v2 + if: always() + with: + files: reports/junit.*.xml +``` + +### Jenkins + +Configure the **JUnit** post-build action to collect `reports/junit.*.xml`. + +### GitLab CI + +```yaml +conformance: + artifacts: + reports: + junit: reports/junit.*.xml +``` + +## Live Monitoring via ConfigMap + +During long-running test suites, monitor progress via the ConfigMap reporter: + +```bash +# Watch for updates +kubectl get configmap -n conformance-tests -l app=conformance-reports -w + +# Get detailed results +kubectl get configmap conformance-reports- \ + -n conformance-tests -o jsonpath='{.data}' | jq . +``` + +Each entry in the ConfigMap represents a completed spec with its state, duration, and any failure message. diff --git a/content/conformance-ee/main/tutorials/customizing-matrix/_index.en.md b/content/conformance-ee/main/tutorials/customizing-matrix/_index.en.md new file mode 100644 index 000000000..c36cfd646 --- /dev/null +++ b/content/conformance-ee/main/tutorials/customizing-matrix/_index.en.md @@ -0,0 +1,107 @@ ++++ +title = "Customizing the Test Matrix" +date = 2026-03-17T10:07:15+02:00 +weight = 20 ++++ + +This guide explains how to customize the test matrix to focus on specific configurations or expand coverage. + +## Understanding the Test Matrix + +Conformance EE generates test scenarios by combining: + +- **Cloud Providers** (e.g., KubeVirt) +- **Kubernetes Versions** (e.g., 1.31, 1.32) +- **OS Distributions** (e.g., Ubuntu, Flatcar) +- **Cluster Modifiers** (CNI, proxy mode, expose strategy, etc.) +- **Machine Modifiers** (CPU, memory, disk, DNS policy, etc.) +- **Datacenter Modifiers** (VPC, subnet, storage class) + +The total number of scenarios is the product of all these dimensions, after deduplication. + +## Filtering by Modifier Description + +Use the `included` and `excluded` fields to filter the test matrix: + +### Include Only Specific Configurations + +```yaml +included: + clusterDescriptions: + - "with cni plugin set to canal" + - "with proxy mode set to ipvs" + machineDescriptions: + - "with ubuntu" +``` + +When `included` is non-empty, only scenarios matching at least one description are kept. + +### Exclude Problematic Combinations + +```yaml +excluded: + clusterDescriptions: + - "with mla-logging enabled" + - "with opa integration enabled" +``` + +Excluded descriptions are applied after includes and remove any matching scenarios. + +## Controlling Resource Sizes + +Reduce the matrix by limiting resource combinations: + +```yaml +# Minimal: single resource size +resources: + cpu: [2] + memory: ["4Gi"] + diskSize: ["20Gi"] + +# Full: multiple resource sizes (4x more scenarios) +resources: + cpu: [2, 4] + memory: ["4Gi", "8Gi"] + diskSize: ["20Gi", "50Gi"] +``` + +## Testing Specific Kubernetes Versions + +```yaml +releases: + - "1.31" + # Add more versions to expand testing + - "1.32" +``` + +## Testing Specific Distributions + +```yaml +enableDistributions: + - ubuntu + - flatcar + - rhel + - rockylinux + +# Or exclude specific ones +excludeDistributions: + - rhel +``` + +## Using Ginkgo Label Filters + +For runtime filtering without changing the config file, use Ginkgo label filters: + +```bash +./e2e.test --ginkgo.label-filter="kubevirt && canal" +``` + +## Estimating Matrix Size + +Use `--ginkgo.dry-run` to see all generated scenarios without executing them: + +```bash +./e2e.test --ginkgo.dry-run --ginkgo.v +``` + +This lists every spec that would run, helping you estimate the size of your test matrix and verify your filters are working correctly. diff --git a/content/conformance-ee/main/tutorials/first-test/_index.en.md b/content/conformance-ee/main/tutorials/first-test/_index.en.md new file mode 100644 index 000000000..f60cec9a2 --- /dev/null +++ b/content/conformance-ee/main/tutorials/first-test/_index.en.md @@ -0,0 +1,102 @@ ++++ +title = "Running Your First Test" +date = 2026-03-17T10:07:15+02:00 +weight = 10 ++++ + +This guide walks you through running your first conformance test using Conformance EE. + +## Prerequisites + +Before starting, ensure you have: + +- A running KKP installation with at least one KubeVirt datacenter configured +- A kubeconfig with access to the KKP seed cluster +- The Conformance EE container image pulled or built locally + +## Step 1: Create a Minimal Configuration + +Create a `config.yaml` with a minimal test matrix: + +```yaml +providers: + - kubevirt + +releases: + - "1.31" + +enableDistributions: + - ubuntu + +resources: + cpu: [2] + memory: ["4Gi"] + diskSize: ["20Gi"] + +imageSources: + ubuntu: + "22.04": "docker://quay.io/kubermatic-virt-disks/ubuntu:22.04" + +controlPlaneReadyWaitTimeout: 10m +nodeReadyTimeout: 20m +nodeCount: 1 +reportsRoot: /reports +``` + +{{% notice tip %}} +Start with a minimal configuration to verify your setup works before expanding the test matrix. +{{% /notice %}} + +## Step 2: Launch the Interactive TUI + +Run the conformance tester binary: + +```bash +./conformance-tester +``` + +The TUI will walk you through selecting your environment, providing credentials, and choosing which tests to run. + +## Step 3: Monitor Test Progress + +### Via ConfigMap + +```bash +kubectl get configmap -n conformance-tests -w +``` + +### Via Job Logs + +```bash +kubectl logs -n conformance-tests job/conformance-tests -f +``` + +## Step 4: Review Results + +After tests complete, JUnit XML reports are available in the reports directory: + +``` +reports/ +├── junit.with_kubernetes_1.31.1_and_ubuntu_22.04_and_canal.xml +└── ... +``` + +These files can be imported into any CI system that supports JUnit XML format. + +## Troubleshooting + +### Cluster Creation Timeout + +If cluster creation exceeds the 10 minute timeout: + +- Verify the KKP API is accessible from the test pod +- Check KKP controller logs for provisioning errors +- Ensure the cloud provider has sufficient quota + +### Node Not Ready + +If worker nodes fail to reach Ready state: + +- Check machine controller logs in the user cluster +- Verify the OS image is accessible from the KubeVirt infrastructure +- Ensure the storage class exists and has available capacity diff --git a/data/mainMenu.yml b/data/mainMenu.yml index edc0ed131..55833dbfc 100644 --- a/data/mainMenu.yml +++ b/data/mainMenu.yml @@ -136,6 +136,14 @@ items: image: /img/menu/kubermatic-developer-platform.jpg columns: - column: + - title: Conformance EE + url: /conformance-ee/main/ + text: Automated end-to-end conformance testing for KKP + noArrowIcon: true + panel: + summary: Conformance EE is an automated end-to-end conformance test framework for Kubermatic Kubernetes Platform. It combinatorially generates, runs, and reports thousands of test scenarios across cloud providers, Kubernetes versions, and OS distributions. + columns: + - column: - title: Kubermatic Kubernetes Subscription url: /products/kubermatic-kubernetes-subscription/ text: Upstream Kubernetes support diff --git a/data/mobileMenu.yml b/data/mobileMenu.yml index f8f89c5c1..3eda9e32b 100644 --- a/data/mobileMenu.yml +++ b/data/mobileMenu.yml @@ -19,6 +19,8 @@ items: url: /products/kubermatic-virtualization/ - title: Kubermatic Developer Platform url: /products/kubermatic-developer-platform/ + - title: Conformance EE + url: /conformance-ee/main/ - title: Kubermatic Kubernetes Subscription url: /products/kubermatic-kubernetes-subscription/ - title: Managed Kubermatic Kubernetes Platform diff --git a/data/products.yaml b/data/products.yaml index 4e70ab393..a625bbe7c 100644 --- a/data/products.yaml +++ b/data/products.yaml @@ -140,6 +140,17 @@ kubermatic-virtualization: name: main - release: v1.0.0 name: v1.0.0 +conformance-ee: + name: conformance-ee + logo: "img/logo-kubermatic.svg" + title: Conformance EE + textName: Conformance EE + description: Automated end-to-end conformance test framework for Kubermatic Kubernetes Platform, generating and running thousands of test scenarios across cloud providers. + weight: 7 + shareImage: img/share-kubermatic.png + versions: + - release: main + name: main # do not yet reveal KDP to the public # developer-platform: # name: developer-platform