Skip to content

[AURON #1848] Introduce native protobuf deserializer#2069

Merged
richox merged 4 commits intoapache:masterfrom
Tartarus0zm:pb-parser
Mar 4, 2026
Merged

[AURON #1848] Introduce native protobuf deserializer#2069
richox merged 4 commits intoapache:masterfrom
Tartarus0zm:pb-parser

Conversation

@Tartarus0zm
Copy link
Contributor

@Tartarus0zm Tartarus0zm commented Mar 4, 2026

Which issue does this PR close?

Closes #1848

Rationale for this change

  • Supports deserialize Protobuf data

What changes are included in this PR?

  • add flink_deserializer.rs
  • pb_deserializer.rs
  • shared_array_builder.rs
  • shared_list_array_builder.rs
  • shared_map_array_builder.rs
  • shared_struct_array_builder.rs
  • add prost-types and prost-reflect

Are there any user-facing changes?

  • No

How was this patch tested?

  • pb_deserializer#test_parse_messages_with_kafka_meta_basic
  • pb_deserializer#test_parse_messages_with_kafka_meta_nested
  • pb_deserializer#test_parse_messages_with_kafka_meta_empty
  • pb_deserializer#test_parse_messages_with_kafka_meta_different_partitions

@Tartarus0zm Tartarus0zm requested a review from richox March 4, 2026 03:50
richox
richox reviewed Mar 4, 2026
View reviewed changes
native-engine/datafusion-ext-plans/src/flink/serde/pb_deserializer.rs Outdated
mapped_field_indices.push(idx);
cur_fields = fields;
} else {
panic!("nested field must be struct");
Copy link
Contributor

@richox richox Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i suggest using df_execution_err!(..) to return Results

Copy link
Contributor Author

@Tartarus0zm Tartarus0zm Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good suggestion

@Tartarus0zm
Copy link
Contributor Author

Tartarus0zm commented Mar 4, 2026

@richox CI is Green, PTAL
I have introduced prost-types and prost-reflect. Please verify that the fixes to the Cargo.lock file are correct.

@richox richox merged commit 72232d5 into apache:master Mar 4, 2026
112 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@richox richox richox approved these changes

Assignees

No one assigned

Labels

build native

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Introduce native protobuf deserializer

2 participants

@Tartarus0zm @richox