Skip to content

DOC-1401 RPCN Secrets navigation change in UI #386

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Aug 11, 2025
Merged

DOC-1401 RPCN Secrets navigation change in UI #386

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
232f1cc
DOC-1401 RPCN Secrets navigation change in UI
micheleRP Aug 11, 2025
16e4e2a
minor edit
micheleRP Aug 11, 2025
81bf2c3
Incorporate coderabbit suggestions
micheleRP Aug 11, 2025
121adfb
remove quick-add
micheleRP Aug 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 11 additions & 10 deletions modules/develop/pages/connect/configuration/resource-management.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

== Prerequisites

- A running xref:get-started:cluster-types/byoc/index.adoc[BYOC] (not BYOVPC/BYOVNet) or xref:get-started:cluster-types/dedicated/create-dedicated-cloud-cluster.adoc[Dedicated] cluster.
- A running xref:get-started:cluster-types/byoc/index.adoc[BYOC] or xref:get-started:cluster-types/dedicated/create-dedicated-cloud-cluster.adoc[Dedicated] cluster.
- An estimate of the throughput of your data pipeline. You can get some basic statistics by running your data pipeline locally using the xref:redpanda-connect:components:processors/benchmark.adoc[`benchmark` processor].

=== Understanding compute units
Expand Down Expand Up @@ -143,7 +143,7 @@ Data Plane API::
. Make a request to xref:api:ROOT:cloud-dataplane-api.adoc#get-/v1/redpanda-connect/pipelines[`GET /v1/redpanda-connect/pipelines`], which lists details of all pipelines on your cluster by ID.
+
* Memory (`memory_shares`) is displayed in megabytes. For example, `1` compute unit is `400M` or 400 MB.
* CPU resources (`cpu_shares`) are displayed milliCPU. For example, `1` compute unit is `100m` or 0.1 CPU.
* CPU resources (`cpu_shares`) are displayed in milliCPU. For example, `1` compute unit is `100m` or 0.1 CPU.

--
=====
Expand Down Expand Up @@ -182,14 +182,15 @@ curl -X PUT "https://<data-plane-api-url>/v1/redpanda-connect/pipelines/xxx..."
-H 'accept: application/json'\
-H 'authorization: Bearer xxx...' \
-H "content-type: application/json" \
-d '{"config_yaml":"input:\n generate:\n interval: 1s\n mapping: |\n root.id = uuid_v4()\n root. user.name = fake(\"name\")\n root.user.email = fake(\"email\")\n root.content = fake(\"paragraph\")\n\npipeline:\n processors:\n - mutation: |\n root.title = \"PRIVATE AND CONFIDENTIAL\"\n\noutput:\n kafka_franz:\n seed_brokers:\n - seed-j888.byoc.prd.cloud.redpanda.com:9092\n sasl:\n - mechanism: SCRAM-SHA-256\n password: password\n username: connect\n topic: processed-emails\n tls:\n enabled: true\n", \
"description":"Email processor", \
"display_name":"emailprocessor-pipeline", \
"resources":{ \
"memory_shares":"800M" \
"cpu_shares":"200m", \
} \
}'
-d '{
"config_yaml": "input:\n generate:\n interval: 1s\n mapping: |\n root.id = uuid_v4()\n root.user.name = fake(\"name\")\n root.user.email = fake(\"email\")\n root.content = fake(\"paragraph\")\n\npipeline:\n processors:\n - mutation: |\n root.title = \"PRIVATE AND CONFIDENTIAL\"\n\noutput:\n kafka_franz:\n seed_brokers:\n - seed-j888.byoc.prd.cloud.redpanda.com:9092\n sasl:\n mechanism: SCRAM-SHA-256\n password: password\n username: connect\n topic: processed-emails\n tls:\n enabled: true\n",
"description": "Email processor",
"display_name": "emailprocessor-pipeline",
"resources": {
"memory_shares": "800M",
"cpu_shares": "200m"
}
}'
----
+
A successful response shows the updated resource allocations with the `cpu_shares` value returned in milliCPU.
Expand Down
118 changes: 41 additions & 77 deletions modules/develop/pages/connect/configuration/secret-management.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,56 +3,31 @@

Learn how to manage secrets in Redpanda Connect, and how to add them to your data pipelines without exposing them.

Secrets are stored in the secret management solution of your Cloud provider and are retrieved when you run a pipeline configuration that references them.

== Prerequisites

* A running Serverless, Dedicated, or BYOC (not BYOVPC/BYOVNet) cluster
* On BYOC clusters only, check that secrets management is enabled:

.. Log in to https://cloud.redpanda.com[Redpanda Cloud^].
.. Go to the **Connect** page.
.. Select the **Redpanda Connect** tab and make sure you can see a **Secrets** tab.

+
If you cannot see a **Secrets** tab, contact https://support.redpanda.com/hc/en-us/requests/new[Redpanda Support^]
Secrets are stored in the secret management solution of your cloud provider and are retrieved when you run a pipeline configuration that references them.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we try to tell them where? Or we're being purposefuly vague here because this is documenting a 3rd party product (style guide + high risk of going out-of-date)?


== Manage secrets

You can manage secrets from the Cloud UI or Data Plane API.
You can manage secrets from the Cloud UI or the Data Plane API.

=== Create a secret

You can create a secret and reference it in multiple data pipelines on the same cluster.

[tabs]
=====
Cloud UI (BYOC and Dedicated)::
Cloud UI::
+
--
. Log in to https://cloud.redpanda.com[Redpanda Cloud^].
. Go to the **Connect** page.
. Select the **Redpanda Connect** tab and then the **Secrets** tab.
. Click **Create secret**.
. In **Secret name**, enter a name for the secret. You cannot rename the secret once it is created.
. In **Secret value**, enter the secret you need to add.
. Go to the **Secrets Store** page.
. Click **Create secret**.
+
The secret details are listed in the **Secrets** tab ready to <<add-a-secret-to-a-data-pipeline,add to your data pipelines>>.

--
. For **ID**, enter a name for the secret. You cannot rename the secret once it is created.
. For **Value**, enter the secret you need to add.
. For **Scopes**, select Redpanda Connect.
. Optionally, add labels to help organize your secrets.
. Click **Create**.

Cloud UI (Serverless)::
+
--
. Log in to https://cloud.redpanda.com[Redpanda Cloud^].
. Go to the **Connect** page.
. Select the **Secrets** tab and click **Create secret**.
. In **Secret name**, enter a name for the secret. You cannot rename the secret once it is created.
. In **Secret value**, enter the secret you need to add.
. Click **Create secret**.
+
The secret details are listed in the **Secrets** tab ready to <<add-a-secret-to-a-data-pipeline,add to your data pipelines>>.
You can now <<add-a-secret-to-a-data-pipeline,add the secret to your data pipeline>>.

--

Expand All @@ -63,7 +38,7 @@ You must use a Base64-encoded secret.

. xref:manage:api/cloud-api-quickstart.adoc#try-the-cloud-api[Authenticate and get the base URL] for the Data Plane API.
. Make a request to xref:api:ROOT:cloud-dataplane-api.adoc#post-/v1/secrets[`POST /v1/secrets`].
+

[,bash]
----
curl -X POST "https://<dataplane-api-url>/v1/secrets" \
Expand All @@ -72,15 +47,14 @@ curl -X POST "https://<dataplane-api-url>/v1/secrets" \
-H 'content-type: application/json' \
-d '{"id":"<secret-name>","scopes":["SCOPE_REDPANDA_CONNECT"],"secret_data":"<secret-value>"}'
----
+

You must include the following values:

- `<dataplane-api-url>`: The base URL for the Data Plane API.
- `<token>`: The API key you generated during authentication.
- `<secret-name>`: The ID or name of the secret you want to add. Use only the following characters: `^[A-Z][A-Z0-9_]*$`.
- `<secret-value>`: The Base64-encoded secret.
- This scope: `"SCOPE_REDPANDA_CONNECT"`.

+
The response returns the name of the secret and the scope `"SCOPE_REDPANDA_CONNECT"`.

Expand All @@ -97,26 +71,13 @@ NOTE: Changes to secret values do not take effect until a pipeline is restarted.

[tabs]
=====
Cloud UI (BYOC and Dedicated)::
Cloud UI::
+
--
. Log in to https://cloud.redpanda.com[Redpanda Cloud^].
. Go to the **Connect** page.
. Select the **Redpanda Connect** tab and then the **Secrets** tab.
. Find the secret you want to update and click the edit icon.
. Enter the new secret value and click **Update Secret**.
. Start and stop any pipelines that reference the secret.

--

Cloud UI (Serverless)::
+
--
. Log in to https://cloud.redpanda.com[Redpanda Cloud^].
. Go to the **Connect** page.
. Select the **Secrets** tab.
. Find the secret you want to update and click the edit icon.
. Enter the new secret value and click **Update Secret**.
. Go to the **Secrets Store** page.
. Find the secret you want to update, and click the edit icon.
. Enter the new secret value or labels, and click **Update**.
. Start and stop any pipelines that reference the secret.

--
Expand Down Expand Up @@ -161,24 +122,12 @@ NOTE: Changes do not affect pipelines that are already running.

[tabs]
=====
Cloud UI (BYOC and Dedicated)::
+
--
. Log in to https://cloud.redpanda.com[Redpanda Cloud^].
. Go to the **Connect** page.
. Select the **Redpanda Connect** tab and then the **Secrets** tab.
. Find the secret you want to remove and click the delete icon.
. Confirm your deletion.

--

Cloud UI (Serverless)::
Cloud UI::
+
--
. Log in to https://cloud.redpanda.com[Redpanda Cloud^].
. Go to the **Connect** page.
. Select the **Secrets** tab.
. Find the secret you want to remove and click the delete icon.
. Go to the **Secrets Store** page.
. Find the secret you want to remove, and click the delete icon.
. Confirm your deletion.

--
Expand Down Expand Up @@ -207,13 +156,28 @@ You must include the following values:

== Add a secret to a data pipeline

You can add a secret to any pipeline in your cluster using the notation `${secrets.SECRET_NAME}`. In the Cloud UI, you can copy the notation from the **Secrets** tab.
[tabs]
=====
Cloud UI::
+
--
. Go to the **Connect** page, and create a pipeline (or open an existing pipeline to edit).
. Click the quick-add **Secret** button to add a new or existing secret to the pipeline.
--

Data Plane API::
+
--
You can add a secret to any pipeline in your cluster using the notation `${secrets.SECRET_NAME}`.
For example:

```yml
sasl:
- mechanism: SCRAM-SHA-256
username: "user"
password: "${secrets.PASSWORD}"
```
[,yml]
----
sasl:
- mechanism: SCRAM-SHA-256
username: "user"
password: "${secrets.PASSWORD}"
----
--
=====