Skip to content

Support access to nested field of struct after fields command #4719

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Nov 4, 2025
Merged

Support access to nested field of struct after fields command #4719

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions core/src/main/java/org/opensearch/sql/calcite/utils/OpenSearchTypeFactory.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -177,11 +177,10 @@ public static RelDataType convertExprTypeToRelDataType(ExprType fieldType, boole
return TYPE_FACTORY.createArrayType(
TYPE_FACTORY.createSqlType(SqlTypeName.ANY, nullable), -1);
case STRUCT:
// TODO: should use RelRecordType instead of MapSqlType here
// https://github.com/opensearch-project/sql/issues/3459
final RelDataType relKey = TYPE_FACTORY.createSqlType(SqlTypeName.VARCHAR);
// TODO: should we provide more precise type here?
Copy link
Member

@LantaoJin LantaoJin Nov 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: Do you have any idea how to specify precise type here? Calcite 1.41.0 has more strict type checking in RexToLixTranslator. If the Map value is another Array type, it cannot cast a ANY to Array.

Copy link
Collaborator Author

@qianheng-aws qianheng-aws Nov 4, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can do type-conversion recursively for the nested fields and store them in a customized type? It will has benefit to resolve proper functions for these fields.

What's more I'm thinking we may be able to deprecate the flatten fields if these nested fields' type are precise enough. It's not worthy to do that now since we have flatten fields and it only takes effect after FIELD command. It should be done as another issue for enhancement.

Copy link
Collaborator Author

@qianheng-aws qianheng-aws Nov 4, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the beginning, I'm thinking using RelRecordType instead of MapSqlType to address this issue, which will get better performance and naturally support variant types.

But looks like using map to support dynamic fields and type tolerance will be our trend. #4349 and #4433

return TYPE_FACTORY.createMapType(
relKey, TYPE_FACTORY.createSqlType(SqlTypeName.BINARY), nullable);
relKey, TYPE_FACTORY.createSqlType(SqlTypeName.ANY), nullable);
case UNKNOWN:
default:
throw new IllegalArgumentException(
Expand Down
59 changes: 59 additions & 0 deletions integ-test/src/yamlRestTest/resources/rest-api-spec/test/issues/3459.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
setup:
- do:
query.settings:
body:
transient:
plugins.calcite.enabled : true

- do:
indices.create:
index: test
body:
mappings:
properties:
parent_field:
properties:
child_field:
type: integer

- do:
bulk:
index: test
refresh: true
body:
- '{"index": {}}'
- '{ "parent_field": { "child_field": 4 } }'
- '{"index": {}}'
- '{ "parent_field": { "child_field": 3 } }'
- '{"index": {}}'
- '{ "parent_field": { "child_field": 2 } }'
- '{"index": {}}'
- '{ "parent_field": { "child_field": 1 } }'
- '{"index": {}}'
- '{ "parent_field": { "child_field": 5 } }'

---
teardown:
- do:
query.settings:
body:
transient:
plugins.calcite.enabled : false

---
"Access to nested field after field command":
- skip:
features:
- headers
- allowed_warnings

- do:
headers:
Content-Type: 'application/json'
ppl:
body:
query: source=test | fields parent_field | sort parent_field.child_field | head 3

- match: { total: 3 }
- match: { schema: [ { "name": "parent_field", "type": "struct" } ] }
- match: { datarows: [ [ {"child_field": 1} ], [ {"child_field": 2} ], [ {"child_field": 3} ] ] }
Loading