Skip to content

Add a solution #90

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
adminwcs wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add a solution #90

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
290 changes: 290 additions & 0 deletions ...form-Exclusive_Scenarios_and_Log_Monitoring_Plugin_in_Upgrade_to_Version_4.2.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,290 @@
---
products:
- Alauda Container Platform
kind:
- Solution
---


# Resolution for Conflict Between Platform-Exclusive Scenarios and Log Monitoring Plugin in Upgrade to Version 4.2

## Background

When upgrading the ACP version to 4.2.x, conflicts between the platform-exclusive scenario and the self-selected nodes for the log monitoring plugin cause upgrade failures. This solution is required to resolve the issue.

## Applicable Scenarios

Upgrading to version 4.2.x.

## Detection

### Step 1: Check if it is a Platform-Exclusive Scenario

```shell
1. Log in to the master node of the global cluster.

2. Execute the following command. If the return value is Base, it indicates a platform-exclusive scenario. Proceed to the next step of detection.

➜ kubectl get productbase.v1alpha1.product.alauda.io base -o jsonpath='{.spec.deployType}'
Copy link
Contributor

@coderabbitai coderabbitai bot Dec 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

cd docs/en/solutions && pwd && wc -l Resolution_for_Conflict_Between_Platform-Exclusive_Scenarios_and_Log_Monitoring_Plugin_in_Upgrade_to_Version_4.2.md

Repository: alauda/knowledge

Length of output: 216

🏁 Script executed:

cd docs/en/solutions && sed -n '20,60p' Resolution_for_Conflict_Between_Platform-Exclusive_Scenarios_and_Log_Monitoring_Plugin_in_Upgrade_to_Version_4.2.md

Repository: alauda/knowledge

Length of output: 2459

🏁 Script executed:

# Search for other uses of 'productbase' or similar Alauda resources in the codebase
rg -i 'productbase|product\.alauda\.io' --type md

Repository: alauda/knowledge

Length of output: 284

🏁 Script executed:

# Search for other kubectl command examples in documentation to see if this pattern is consistent
rg 'kubectl get.*alauda\.io' --type md -A 1 -B 1

Repository: alauda/knowledge

Length of output: 2244

🏁 Script executed:

# Search for productbases (plural) in codebase
rg -i 'productbases' --type md

Repository: alauda/knowledge

Length of output: 42

🏁 Script executed:

# Search for any CRD definitions or API references that might show the correct resource name
rg 'kind.*ProductBase|ProductBase.*kind' --type md -i

Repository: alauda/knowledge

Length of output: 42

🏁 Script executed:

# Search for any references to "Product" resources in general
rg 'Product' --type md | head -20

Repository: alauda/knowledge

Length of output: 1761

🏁 Script executed:

# Look for the full file content to understand context better
cat -n docs/en/solutions/Resolution_for_Conflict_Between_Platform-Exclusive_Scenarios_and_Log_Monitoring_Plugin_in_Upgrade_to_Version_4.2.md | head -100

Repository: alauda/knowledge

Length of output: 4714

Clarify detection conditions and verify API resource format.

Line 28 uses productbase.v1alpha1.product.alauda.io, which deviates from the standard kubectl resource naming pattern (<plural>.<api-group>) seen in other Alauda resources within the codebase. Confirm this format is correct for Alauda CRDs.

Additionally, lines 46, 49, and 52 state to execute patches "if yes" but do not explicitly address the "no" scenarios: What should users do if the moduleinfo query (line 42) returns no results, or if plugins don't use LocalVolume, or if they only use platform-exclusive nodes? This clarification ensures users know whether to skip those patches or proceed without changes.

🤖 Prompt for AI Agents 
In
docs/en/solutions/Resolution_for_Conflict_Between_Platform-Exclusive_Scenarios_and_Log_Monitoring_Plugin_in_Upgrade_to_Version_4.2.md
around line 28, confirm and correct the kubectl resource name and jsonpath (the
current `productbase.v1alpha1.product.alauda.io` does not follow the usual
`<plural>.<api-group>` pattern): verify the actual CRD name with `kubectl
api-resources` (or `kubectl get crd`) and update the example to use the correct
plural form and jsonpath selector; also update lines 46, 49 and 52 to explicitly
document the "no" branches — state that if the moduleinfo query returns no
results, or plugins do not use LocalVolume, or only use platform-exclusive
nodes, the specified patches should be skipped (or provide the exact alternative
steps to take), so readers know whether to apply or skip each patch.

```

### Step 2: Check if the Global Log Monitoring Plugin Uses Local Storage and Non-Platform-Exclusive Nodes

```shell
1. Log in to the master node of the global cluster.

2. Retrieve the platform-exclusive nodes.

➜ kubectl get nodes -l node-role.kubernetes.io/cpaas-system -o wide

3. Execute the following command to confirm if the log monitoring plugin is deployed.

➜ kubectl get moduleinfo -l 'cpaas.io/cluster-name=global,cpaas.io/module-name in (prometheus,victoriametrics,logcenter,logclickhouse)' -o custom-columns=NAME:'.metadata.name',MODULE:'.metadata.labels.cpaas\.io/module-name'

4. Confirm whether the deployed log monitoring plugin uses local storage and non-platform-exclusive nodes.

Prometheus/Victoriametrics Monitoring Plugin: Check if the return indicates LocalVolume and uses non-platform-exclusive nodes. If yes, execute the corresponding operations for this plugin in the Problem Resolution steps.
➜ kubectl get moduleinfo <moduleinfo-name> -o jsonpath='Type:{.spec.config.storage.type}{"\n"}Nodes:{.spec.config.storage.nodes}{"\n"}'

Elasticsearch Log Storage Plugin: Check if the return indicates LocalVolume and uses non-platform-exclusive nodes. If yes, execute the corresponding operations for this plugin in the Problem Resolution steps.
➜ kubectl get moduleinfo <moduleinfo-name> -o jsonpath='Type:{.spec.config.components.storageClassConfig.type}{"\n"}ESNodes:{.spec.config.components.elasticsearch.k8sNodes}{"\n"}ESMasterNodes:{.spec.config.components.elasticsearch.masterK8sNodes}{"\n"}KafkaNodes:{.spec.config.components.kafka.k8sNodes}{"\n"}'

ClickHouse Log Storage Plugin: Check if the return indicates LocalVolume and uses non-platform-exclusive nodes. If yes, execute the corresponding operations for this plugin in the Problem Resolution steps.
➜ kubectl get moduleinfo <moduleinfo-name> -o jsonpath='Type:{.spec.config.components.storageClassConfig.type}{"\n"}CKNodes:{.spec.config.components.clickhouse.k8sNodes}{"\n"}'
```
Comment on lines +46 to +54
Copy link
Contributor

@coderabbitai coderabbitai bot Dec 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Provide example outputs and clearer decision logic for detection steps.

Lines 47, 50, 53 use complex jsonpath queries but don't show example outputs. Provide:

  • Sample output when Type:LocalVolume is detected (patch needed)
  • Sample output when storage is not LocalVolume (no patch needed)
  • Explicit decision statements (e.g., "If output shows Type:LocalVolume and nodes are not platform-exclusive, apply the [Plugin Name] patch below")

This reduces ambiguity and helps users confidently determine which patches to apply.

🤖 Prompt for AI Agents 
In
docs/en/solutions/Resolution_for_Conflict_Between_Platform-Exclusive_Scenarios_and_Log_Monitoring_Plugin_in_Upgrade_to_Version_4.2.md
around lines 46–54, the jsonpath checks for Prometheus/Victoriametrics,
Elasticsearch and ClickHouse lack example outputs and explicit decision rules;
add for each plugin a) a sample output when Type:LocalVolume is returned
(showing nodes list that is non-platform-exclusive), b) a sample output when
storage is not LocalVolume, and c) a clear decision statement such as "If output
shows Type:LocalVolume AND nodes are not platform-exclusive, apply the [Plugin
Name] patch below; otherwise no patch needed." Place these examples immediately
after each kubectl command and ensure the decision line is concise and identical
in structure for all three plugins.


## Problem Resolution

### Prometheus Monitoring Plugin

```shell
cat <<EOF | kubectl create -f -
apiVersion: operator.alauda.io/v1alpha1
kind: ResourcePatch
metadata:
labels:
target: fb6047e14c28909f84766d7902c9b546
name: change-prometheus-prometheus-0-nodeselector
spec:
jsonPatch:
- op: remove
path: /spec/nodeSelector/node-role.kubernetes.io~1cpaas-system
release: cpaas-system/kube-prometheus
target:
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
name: kube-prometheus-0
namespace: cpaas-system
---
apiVersion: operator.alauda.io/v1alpha1
kind: ResourcePatch
metadata:
labels:
target: 3e1291e8a1158e0ae1ca0febb722a6eb
name: change-prometheus-prometheus-1-nodeselector
spec:
jsonPatch:
- op: remove
path: /spec/nodeSelector/node-role.kubernetes.io~1cpaas-system
release: cpaas-system/kube-prometheus
target:
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
name: kube-prometheus-1
namespace: cpaas-system
---
apiVersion: operator.alauda.io/v1alpha1
kind: ResourcePatch
metadata:
labels:
target: 71ee303f02f933b9cabfbf667dab683e
name: change-prometheus-prometheus-2-nodeselector
spec:
jsonPatch:
- op: remove
path: /spec/nodeSelector/node-role.kubernetes.io~1cpaas-system
release: cpaas-system/kube-prometheus
target:
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
name: kube-prometheus-2
namespace: cpaas-system
---
apiVersion: operator.alauda.io/v1alpha1
kind: ResourcePatch
metadata:
labels:
target: 8b98cf81313f70917e55c320ac8528ee
name: change-prometheus-alertmanager-nodeselector
spec:
jsonPatch:
- op: remove
path: /spec/nodeSelector/node-role.kubernetes.io~1cpaas-system
release: cpaas-system/kube-prometheus
target:
apiVersion: monitoring.coreos.com/v1
kind: Alertmanager
name: kube-prometheus
namespace: cpaas-system
EOF
```
Comment on lines +61 to +130
Copy link
Contributor

@coderabbitai coderabbitai bot Dec 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

# First, let's read the file to understand the full context and see if there are prerequisite sections
cat docs/en/solutions/Resolution_for_Conflict_Between_Platform-Exclusive_Scenarios_and_Log_Monitoring_Plugin_in_Upgrade_to_Version_4.2.md

Repository: alauda/knowledge

Length of output: 9000

🏁 Script executed:

# Also check if there are related documentation files about ResourcePatch
find . -type f -name "*.md" | xargs grep -l "ResourcePatch" | head -10

Repository: alauda/knowledge

Length of output: 543

🏁 Script executed:

# Search for any explanation of the target labels or verification procedures in the doc
grep -n "target\|verify\|verification\|rollback\|prerequisite\|prerequisites" docs/en/solutions/Resolution_for_Conflict_Between_Platform-Exclusive_Scenarios_and_Log_Monitoring_Plugin_in_Upgrade_to_Version_4.2.md

Repository: alauda/knowledge

Length of output: 811

Add prerequisite documentation, verification steps, and rollback procedures.

The Problem Resolution section provides ResourcePatch manifests without explaining the prerequisite ResourcePatch operator, how to verify patches were applied, or how to recover if they fail. Additionally, the hardcoded target labels (e.g., fb6047e14c28909f84766d7902c9b546) are never explained—users won't know if these are universal identifiers or require customization for different environments.

Add:

  • What the ResourcePatch operator is and whether it must be pre-installed (if so, link to installation docs)
  • Verification commands after applying patches (e.g., kubectl get resourcepatch, checking that nodeSelector was removed from the target resources)
  • Rollback instructions if patches fail or cause issues
  • Clarification on whether hardcoded target labels are environment-specific or universal across all deployments
  • How to adapt resource names (kube-prometheus-0, cpaas-system, etc.) if your environment uses different naming

Also applies to: 135-174, 179-247, 253-289

🤖 Prompt for AI Agents 
In
docs/en/solutions/Resolution_for_Conflict_Between_Platform-Exclusive_Scenarios_and_Log_Monitoring_Plugin_in_Upgrade_to_Version_4.2.md
around lines 61-130 (and similarly for 135-174, 179-247, 253-289), the
ResourcePatch manifests are presented without prerequisites, verification,
rollback guidance, or explanation of the hardcoded target labels and resource
names; update the doc to (1) state that the ResourcePatch operator must be
installed (add a link to its install docs or CR/helm instructions), (2) show
post-apply verification commands (e.g., kubectl get resourcepatch -n
<namespace>, kubectl get prometheus <name> -n <namespace> -o yaml | grep
nodeSelector to confirm removal), (3) provide rollback steps (kubectl delete
resourcepatch <name> or patch to restore nodeSelector with example commands and
a note on operator reconciliation delays), (4) explain that the metadata.labels
'target' values are environment-specific identifiers and instruct users how to
obtain or generate the correct target label for their release/cluster, and (5)
convert hardcoded values (kube-prometheus-0, cpaas-system, etc.) into clearly
marked variables or examples and add a short note on how to adapt
names/namespaces for different environments.


### Victoriametrics Monitoring Plugin

```shell
cat <<EOF | kubectl create -f -
apiVersion: operator.alauda.io/v1alpha1
kind: ResourcePatch
metadata:
labels:
target: c3849be9de68b3745c2448e4a91e03ca
name: change-victoriametrics-vmcluster-nodeselector
spec:
jsonPatch:
- op: remove
path: /spec/vminsert/nodeSelector/node-role.kubernetes.io~1cpaas-system
- op: remove
path: /spec/vmselect/nodeSelector/node-role.kubernetes.io~1cpaas-system
- op: remove
path: /spec/vmstorage/nodeSelector/node-role.kubernetes.io~1cpaas-system
release: cpaas-system/victoriametrics
target:
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMCluster
name: cluster
namespace: cpaas-system
---
apiVersion: operator.alauda.io/v1alpha1
kind: ResourcePatch
metadata:
labels:
target: 3592effe9303af75b2419d6ae3627a6e
name: change-victoriametrics-alertmanager-nodeselector
spec:
jsonPatch:
- op: remove
path: /spec/nodeSelector/node-role.kubernetes.io~1cpaas-system
release: cpaas-system/victoriametrics
target:
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlertmanager
name: alertmanager
namespace: cpaas-system
EOF
```

### Elasticsearch Log Plugin

```shell
cat <<EOF | kubectl create -f -
apiVersion: operator.alauda.io/v1alpha1
kind: ResourcePatch
metadata:
labels:
target: b8a8e4e1de90097a4271feb975390b5e
name: change-elasticsearch-elasticsearch-nodeselector
spec:
jsonPatch:
- op: remove
path: /spec/template/spec/nodeSelector/node-role.kubernetes.io~1cpaas-system
release: cpaas-system/logcenter
target:
apiVersion: apps/v1
kind: StatefulSet
name: cpaas-elasticsearch
namespace: cpaas-system
---
apiVersion: operator.alauda.io/v1alpha1
kind: ResourcePatch
metadata:
labels:
target: cb421402b28b1500cdeb51abacf3103f
name: change-elasticsearch-elasticsearch-master-nodeselector
spec:
jsonPatch:
- op: remove
path: /spec/template/spec/nodeSelector/node-role.kubernetes.io~1cpaas-system
release: cpaas-system/logcenter
target:
apiVersion: apps/v1
kind: StatefulSet
name: cpaas-elasticsearch-master
namespace: cpaas-system
---
apiVersion: operator.alauda.io/v1alpha1
kind: ResourcePatch
metadata:
labels:
target: ac007aec25b3c1248bc7078c93b96a22
name: change-elasticsearch-kafka-nodeselector
spec:
jsonPatch:
- op: remove
path: /spec/template/spec/nodeSelector/node-role.kubernetes.io~1cpaas-system
release: cpaas-system/logcenter
target:
apiVersion: apps/v1
kind: StatefulSet
name: cpaas-kafka
namespace: cpaas-system
---
apiVersion: operator.alauda.io/v1alpha1
kind: ResourcePatch
metadata:
labels:
target: decb9711da9687416144fb09c9a90b92
name: change-elasticsearch-zookeeper-nodeselector
spec:
jsonPatch:
- op: remove
path: /spec/template/spec/nodeSelector/node-role.kubernetes.io~1cpaas-system
release: cpaas-system/logcenter
target:
apiVersion: apps/v1
kind: StatefulSet
name: cpaas-zookeeper
namespace: cpaas-system
EOF
```

### ClickHouse Log Plugin

```shell
cat <<EOF | kubectl create -f -
apiVersion: operator.alauda.io/v1alpha1
kind: ResourcePatch
metadata:
labels:
target: cb60c71fba588ce93207398793d5df0a
name: change-clickhouse-clickhouse-nodeselector
spec:
jsonPatch:
- op: remove
path: /spec/templates/podTemplates/0/spec/nodeSelector/node-role.kubernetes.io~1cpaas-system
- op: remove
path: /spec/templates/podTemplates/1/spec/nodeSelector/node-role.kubernetes.io~1cpaas-system
release: cpaas-system/logclickhouse
target:
apiVersion: clickhouse.altinity.com/v1
kind: ClickHouseInstallation
name: cpaas-clickhouse
namespace: cpaas-system
---
apiVersion: operator.alauda.io/v1alpha1
kind: ResourcePatch
metadata:
labels:
target: 8431e11f34de6f62013c123e5ca373bf
name: change-clickhouse-razor-nodeselector
spec:
jsonPatch:
- op: remove
path: /spec/template/spec/nodeSelector/node-role.kubernetes.io~1cpaas-system
release: cpaas-system/logclickhouse
target:
apiVersion: apps/v1
kind: StatefulSet
name: razor
namespace: cpaas-system
EOF
Comment on lines +56 to +289
Copy link
Contributor

@coderabbitai coderabbitai bot Dec 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Add operational context: timing, verification, and rollback procedures.

The document lacks guidance on:

  • When to apply patches in the upgrade lifecycle (before upgrade starts, during, after initial failure?)
  • Verification that patches were successfully applied
  • Rollback procedure if patches fail or cause unintended side effects
  • Order of operations if multiple plugins need patches (do they need to be sequential or can they be parallel?)

Adding these sections would significantly improve usability and reduce risk of misconfiguration.

```