Add GetComponentListRequest endpoint. #2011

tomasz-sadura · 2025-11-03T11:10:42Z

No description provided.

github-actions · 2025-11-03T11:12:21Z

The latest Buf updates on your PR. Results from workflow Buf CI / validate (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
`✅ passed`	`✅ passed`	`✅ passed`	`✅ passed`	Nov 6, 2025, 3:08 PM

malinskibeniamin · 2025-11-03T16:22:02Z

proto/redpanda/api/dataplane/v1/pipeline.proto

+  // JSON string containing the lists of available components.
+  string components_json = 1;


How much more information can we infer here from the proto, is it just a stringifed JSON? Can it be a more specific map of processors, inputs, outputs and other component types?

We are going to be using protovalidate in the future and this would be a great addition

I can return a more structured response if that's helpful

@eblairmckee let us know what else you need, to make your life on the UI easier, I think having better types coming back from frontend will mean you have less heavy lifting/transformations to do.

updated the return schema, does it look ok?

birdayz · 2025-11-05T10:03:19Z

proto/redpanda/api/dataplane/v1/pipeline.proto

+message GetComponentListRequest {
+  // Regex pattern to filter component types.
+  // Examples: "inputs|outputs", "processors", ".*" (for all)
+  string component_type_regex = 1;


i suggest to follow the existing pattern for filters: putting into a filter struct: https://github.com/redpanda-data/console/blob/master/proto/redpanda/api/dataplane/v1/topic.proto#L120

I added read_mask, it covers per-component filtering as well

birdayz · 2025-11-05T10:04:48Z

proto/redpanda/api/dataplane/v1/pipeline.proto

+  string component_type_regex = 1;
+
+  // List of component fields to skip in the response.
+  repeated string skip_fields = 2;


this sounds like a read_mask, https://github.com/redpanda-data/cloudv2/blob/main/proto/descriptors/redpanda/iam/v1/user_service.proto#L125

https://google.aip.dev/157

you can then use our fieldmask library in pkg to strip fields not covered by read mask

the idea here was to allow to skip examples which weight a lot. The config can be nested, I think we are not able to build fieldmask to filter it on all levels, e.g. (don't have examples here but we can look at description):

{ "components": { "inputs": [ { "name": "amqp_0_9", "type": "input", "status": "stable", "summary": "Connects to an AMQP (0.91) queue. AMQP is a messaging protocol used by various message brokers, including RabbitMQ.", "description": "xxx", "categories": [ "Services" ], "config": { "type": "object", "kind": "scalar", "children": [ { "name": "urls", "type": "string", "kind": "array", "description": "A list of URLs to connect to. The first URL to successfully establish a connection will be used until the connection is closed. If an item of the list contains commas it will be expanded into multiple URLs.", "version": "3.58.0" }, { "name": "queue", "type": "string", "kind": "scalar", "description": "An AMQP queue to consume from." }, { "name": "queue_declare", "type": "object", "kind": "scalar", "description": "Allows you to passively declare the target queue. If the queue already exists then the declaration passively verifies that they match the target fields.", "children": [ { "name": "enabled", "type": "bool", "kind": "scalar", "description": "Whether to enable queue declaration." }, ...

it doesn't look right:

["inputs.name", "inputs.type", "inputs.kind", "inputs.config.name", "inputs.config.type", "inputs.config.kind", "inputs.config.children.name", "inputs.config.children.type", "inputs.config.children.kind", "inputs.config.children.children.name", "inputs.config.children.children.type", "inputs.config.children.children.kind", ...]

I checked the examples actually does not weight so much, so I don't think we need to complicate here. Still users can use read_mask

birdayz · 2025-11-05T10:09:14Z

i like the direction, to have a rich grpc api instead of the YAML. it will make this much easier to consume.
@Jeffail i would like to get your buy-in to this approach, does the approach make sense to you?

malinskibeniamin · 2025-11-11T11:00:58Z

proto/redpanda/api/dataplane/v1/pipeline.proto

+message BloblangMethodSpec {
+  string name = 1;
+  string description = 2;
+  string status = 3;


Shouldn't this be an enum instead of a string?

malinskibeniamin · 2025-11-11T11:01:35Z

proto/redpanda/api/dataplane/v1/pipeline.proto

+// ComponentList contains the structured component data.
+message ComponentList {
+  string version = 1;
+  string date = 2;


shouldn't this be a Timestamp? or is it just stringified date

malinskibeniamin · 2025-11-11T11:02:14Z

proto/redpanda/api/dataplane/v1/pipeline.proto

  }

+  // GetComponentList returns a JSON containing lists of allowed components based on component type filter.
+  rpc GetComponentList(GetComponentListRequest) returns (GetComponentListResponse) {


[nit] I think ListComponents would be better? Usually we reserve Get prefix for obtaining 1 item

malinskibeniamin · 2025-11-11T11:03:15Z

proto/redpanda/api/dataplane/v1/pipeline.proto

+
+// ComponentList contains the structured component data.
+message ComponentList {
+  string version = 1;


Is there any extra validation that could be set at the proto level? For example I imagine this would be a semver regex

malinskibeniamin · 2025-11-11T11:03:40Z

proto/redpanda/api/dataplane/v1/pipeline.proto

+  string default_value = 8;
+  repeated string examples = 9;
+  repeated FieldSpec children = 10;
+  map<string, string> annotated_options = 11;


[question] Is this just a metadata with any arbitrary key value items?

eblairmckee · 2025-11-11T18:47:19Z

proto/redpanda/api/dataplane/v1/pipeline.proto

+}
+
+// FieldSpec represents a field specification.
+message FieldSpec {


can confirm this includes all fields requested, thank you!

weeco · 2025-11-12T11:20:40Z

I am a bit out of loop and have a couple questions:
@tomasz-sadura Is this just an API proposal or where does the translation happen between the RPCN schema and proto? Is this going to happen with protojson somehow semi automatically? Also wondering how we can ensure that new RPCN schema changes will be supported so that we don't necessarily have to update the proto specs and remain somewhat decoupled from the RPCN versioning.

How is the frontend going to work with that information @eblairmckee & @malinskibeniamin ? Eventually we need to translate this back into RPCN pipelines / yaml config. Where does this translation happen?

Add GetComponentListRequest endpoint.

fe09e05

tomasz-sadura force-pushed the ts/add-get-component-list-endpoint branch from 2d51d39 to fe09e05 Compare November 3, 2025 11:45

tomasz-sadura marked this pull request as ready for review November 3, 2025 16:04

tomasz-sadura requested review from birdayz, eblairmckee and malinskibeniamin November 3, 2025 16:05

malinskibeniamin reviewed Nov 3, 2025

View reviewed changes

structure GetComponentList response

b046842

tomasz-sadura requested a review from malinskibeniamin November 4, 2025 10:09

birdayz reviewed Nov 5, 2025

View reviewed changes

birdayz requested review from Jeffail and weeco November 5, 2025 10:07

use read_mask instead

f03edc9

tomasz-sadura requested a review from birdayz November 6, 2025 16:56

malinskibeniamin reviewed Nov 11, 2025

View reviewed changes

eblairmckee reviewed Nov 11, 2025

View reviewed changes

		// JSON string containing the lists of available components.
		string components_json = 1;

Add GetComponentListRequest endpoint. #2011

Are you sure you want to change the base?

Add GetComponentListRequest endpoint. #2011

Uh oh!

Conversation

tomasz-sadura commented Nov 3, 2025

Uh oh!

github-actions bot commented Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

birdayz Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tomasz-sadura Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

birdayz commented Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

malinskibeniamin Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

malinskibeniamin Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

weeco commented Nov 12, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

github-actions bot commented Nov 3, 2025 •

edited

Loading

birdayz Nov 5, 2025 •

edited

Loading

tomasz-sadura Nov 5, 2025 •

edited

Loading

birdayz commented Nov 5, 2025 •

edited

Loading

malinskibeniamin Nov 11, 2025 •

edited

Loading

malinskibeniamin Nov 11, 2025 •

edited

Loading