Skip to content

Support access to nested field of struct after fields command #4719

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Nov 4, 2025
Merged

Support access to nested field of struct after fields command #4719

merged 1 commit into from
Nov 4, 2025

Conversation

@qianheng-aws
Copy link
Collaborator

@qianheng-aws qianheng-aws commented Nov 3, 2025

Description

Support access to nested field of struct after fields command

Related Issues

Resolves #3459

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • New functionality has javadoc added.
  • New functionality has a user manual doc added.
  • New PPL command checklist all confirmed.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff or -s.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@qianheng-aws qianheng-aws added the bug Something isn't working label Nov 3, 2025
LantaoJin
LantaoJin approved these changes Nov 3, 2025
View reviewed changes
core/src/main/java/org/opensearch/sql/calcite/utils/OpenSearchTypeFactory.java
// TODO: should use RelRecordType instead of MapSqlType here
// https://github.com/opensearch-project/sql/issues/3459
final RelDataType relKey = TYPE_FACTORY.createSqlType(SqlTypeName.VARCHAR);
// TODO: should we provide more precise type here?
Copy link
Member

@LantaoJin LantaoJin Nov 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: Do you have any idea how to specify precise type here? Calcite 1.41.0 has more strict type checking in RexToLixTranslator. If the Map value is another Array type, it cannot cast a ANY to Array.

Copy link
Collaborator Author

@qianheng-aws qianheng-aws Nov 4, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can do type-conversion recursively for the nested fields and store them in a customized type? It will has benefit to resolve proper functions for these fields.

What's more I'm thinking we may be able to deprecate the flatten fields if these nested fields' type are precise enough. It's not worthy to do that now since we have flatten fields and it only takes effect after FIELD command. It should be done as another issue for enhancement.

Copy link
Collaborator Author

@qianheng-aws qianheng-aws Nov 4, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the beginning, I'm thinking using RelRecordType instead of MapSqlType to address this issue, which will get better performance and naturally support variant types.

But looks like using map to support dynamic fields and type tolerance will be our trend. #4349 and #4433

@yuancu yuancu merged commit e06ebc5 into opensearch-project:main Nov 4, 2025
38 of 39 checks passed
opensearch-trigger-bot bot pushed a commit that referenced this pull request Nov 4, 2025
@github-actions
Support access to nested field of struct after fields command (#4719)
091902e 
Signed-off-by: Heng Qian <[email protected]>
(cherry picked from commit e06ebc5)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@opensearch-trigger-bot opensearch-trigger-bot bot mentioned this pull request Nov 4, 2025
Merged
LantaoJin pushed a commit that referenced this pull request Nov 4, 2025
@opensearch-trigger-bot @github-actions @qianheng-aws
Support access to nested field of struct after fields command (#4719) (…
ae9fa90 
…#4727)

(cherry picked from commit e06ebc5)

Signed-off-by: Heng Qian <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Heng Qian <[email protected]>
expani pushed a commit to vinaykpud/sql that referenced this pull request Nov 4, 2025
@qianheng-aws @expani
Support access to nested field of struct after fields command (opense…
647e927 
…arch-project#4719)

Signed-off-by: Heng Qian <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@LantaoJin LantaoJin LantaoJin approved these changes

@yuancu yuancu yuancu approved these changes

@ps48 ps48 Awaiting requested review from ps48 ps48 is a code owner

@kavithacm kavithacm Awaiting requested review from kavithacm kavithacm is a code owner

@derek-ho derek-ho Awaiting requested review from derek-ho derek-ho is a code owner

@joshuali925 joshuali925 Awaiting requested review from joshuali925 joshuali925 is a code owner

@dai-chen dai-chen Awaiting requested review from dai-chen dai-chen is a code owner

@YANG-DB YANG-DB Awaiting requested review from YANG-DB YANG-DB is a code owner

@mengweieric mengweieric Awaiting requested review from mengweieric mengweieric is a code owner

@vamsimanohar vamsimanohar Awaiting requested review from vamsimanohar vamsimanohar is a code owner

@Swiddis Swiddis Awaiting requested review from Swiddis Swiddis is a code owner

@penghuo penghuo Awaiting requested review from penghuo penghuo is a code owner

@seankao-az seankao-az Awaiting requested review from seankao-az seankao-az is a code owner

@MaxKsyunz MaxKsyunz Awaiting requested review from MaxKsyunz MaxKsyunz is a code owner

@Yury-Fridlyand Yury-Fridlyand Awaiting requested review from Yury-Fridlyand Yury-Fridlyand is a code owner

@anirudha anirudha Awaiting requested review from anirudha anirudha is a code owner

@forestmvey forestmvey Awaiting requested review from forestmvey forestmvey is a code owner

@acarbonetto acarbonetto Awaiting requested review from acarbonetto acarbonetto is a code owner

@GumpacG GumpacG Awaiting requested review from GumpacG GumpacG is a code owner

@ykmr1224 ykmr1224 Awaiting requested review from ykmr1224 ykmr1224 is a code owner

@noCharger noCharger Awaiting requested review from noCharger noCharger is a code owner

@RyanL1997 RyanL1997 Awaiting requested review from RyanL1997 RyanL1997 is a code owner

Assignees

No one assigned

Labels

backport 2.19-dev bug Something isn't working

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[FEATURE][Calcite Engine] Support access to nested fields after command FIELDS

3 participants

@qianheng-aws @LantaoJin @yuancu