Merged
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #103 +/- ##
==========================================
+ Coverage 84.12% 85.28% +1.16%
==========================================
Files 16 17 +1
Lines 1216 1346 +130
Branches 1216 1346 +130
==========================================
+ Hits 1023 1148 +125
- Misses 131 134 +3
- Partials 62 64 +2 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
adriangb
added a commit
that referenced
this pull request
Dec 17, 2025
friendlymatthew
approved these changes
Dec 17, 2025
Member
friendlymatthew
left a comment
friendlymatthew
left a comment
There was a problem hiding this comment.
Makes sense to me. I think we should address 2 comments below before merge
src/json_from_scalar.rs
Outdated
| impl Default for JsonFromScalar { | ||
| fn default() -> Self { | ||
| Self { | ||
| signature: Signature::variadic_any(Volatility::Immutable), |
Member
There was a problem hiding this comment.
It's a bit odd that we model this as Signature::variadic_any when the function requires exactly 1 argument
Signature::any(1, Volatility::Immutable) seems like a better fit
src/json_from_scalar.rs
Outdated
| } | ||
| } | ||
|
|
||
| fn scalar_to_json_union_field(scalar: ScalarValue) -> DataFusionResult<Option<JsonUnionField>> { |
Member
There was a problem hiding this comment.
It seems the return type's Option wrap is superfluous. I don't see any of the arms below return an Ok(None)
src/json_from_scalar.rs
Outdated
Comment on lines
+157
to
+311
| match array.data_type() { | ||
| DataType::Null => { | ||
| for _ in 0..array.len() { | ||
| union.push(JsonUnionField::JsonNull); | ||
| } | ||
| } | ||
|
|
||
| DataType::Boolean => { | ||
| let arr = array.as_boolean(); | ||
| for i in 0..arr.len() { | ||
| if arr.is_null(i) { | ||
| union.push_none(); | ||
| } else { | ||
| union.push(JsonUnionField::Bool(arr.value(i))); | ||
| } | ||
| } | ||
| } | ||
|
|
||
| // Integer types - coerce to i64 | ||
| DataType::Int8 => { | ||
| let arr = array.as_primitive::<Int8Type>(); | ||
| for i in 0..arr.len() { | ||
| if arr.is_null(i) { | ||
| union.push_none(); | ||
| } else { | ||
| union.push(JsonUnionField::Int(i64::from(arr.value(i)))); | ||
| } | ||
| } | ||
| } | ||
| DataType::Int16 => { | ||
| let arr = array.as_primitive::<Int16Type>(); | ||
| for i in 0..arr.len() { | ||
| if arr.is_null(i) { | ||
| union.push_none(); | ||
| } else { | ||
| union.push(JsonUnionField::Int(i64::from(arr.value(i)))); | ||
| } | ||
| } | ||
| } | ||
| DataType::Int32 => { | ||
| let arr = array.as_primitive::<Int32Type>(); | ||
| for i in 0..arr.len() { | ||
| if arr.is_null(i) { | ||
| union.push_none(); | ||
| } else { | ||
| union.push(JsonUnionField::Int(i64::from(arr.value(i)))); | ||
| } | ||
| } | ||
| } | ||
| DataType::Int64 => { | ||
| let arr = array.as_primitive::<Int64Type>(); | ||
| for i in 0..arr.len() { | ||
| if arr.is_null(i) { | ||
| union.push_none(); | ||
| } else { | ||
| union.push(JsonUnionField::Int(arr.value(i))); | ||
| } | ||
| } | ||
| } | ||
| DataType::UInt8 => { | ||
| let arr = array.as_primitive::<UInt8Type>(); | ||
| for i in 0..arr.len() { | ||
| if arr.is_null(i) { | ||
| union.push_none(); | ||
| } else { | ||
| union.push(JsonUnionField::Int(i64::from(arr.value(i)))); | ||
| } | ||
| } | ||
| } | ||
| DataType::UInt16 => { | ||
| let arr = array.as_primitive::<UInt16Type>(); | ||
| for i in 0..arr.len() { | ||
| if arr.is_null(i) { | ||
| union.push_none(); | ||
| } else { | ||
| union.push(JsonUnionField::Int(i64::from(arr.value(i)))); | ||
| } | ||
| } | ||
| } | ||
| DataType::UInt32 => { | ||
| let arr = array.as_primitive::<UInt32Type>(); | ||
| for i in 0..arr.len() { | ||
| if arr.is_null(i) { | ||
| union.push_none(); | ||
| } else { | ||
| union.push(JsonUnionField::Int(i64::from(arr.value(i)))); | ||
| } | ||
| } | ||
| } | ||
| DataType::UInt64 => { | ||
| let arr = array.as_primitive::<UInt64Type>(); | ||
| for i in 0..arr.len() { | ||
| if arr.is_null(i) { | ||
| union.push_none(); | ||
| } else { | ||
| union.push(JsonUnionField::Int(i64::try_from(arr.value(i)).map_err(|_| { | ||
| exec_datafusion_err!("UInt64 value {} is out of range for i64", arr.value(i)) | ||
| })?)); | ||
| } | ||
| } | ||
| } | ||
|
|
||
| // Float types - coerce to f64 | ||
| DataType::Float32 => { | ||
| let arr = array.as_primitive::<Float32Type>(); | ||
| for i in 0..arr.len() { | ||
| if arr.is_null(i) { | ||
| union.push_none(); | ||
| } else { | ||
| union.push(JsonUnionField::Float(f64::from(arr.value(i)))); | ||
| } | ||
| } | ||
| } | ||
| DataType::Float64 => { | ||
| let arr = array.as_primitive::<Float64Type>(); | ||
| for i in 0..arr.len() { | ||
| if arr.is_null(i) { | ||
| union.push_none(); | ||
| } else { | ||
| union.push(JsonUnionField::Float(arr.value(i))); | ||
| } | ||
| } | ||
| } | ||
|
|
||
| // String types | ||
| DataType::Utf8 => { | ||
| let arr = array.as_string::<i32>(); | ||
| for i in 0..arr.len() { | ||
| if arr.is_null(i) { | ||
| union.push_none(); | ||
| } else { | ||
| union.push(JsonUnionField::Str(arr.value(i).to_string())); | ||
| } | ||
| } | ||
| } | ||
| DataType::LargeUtf8 => { | ||
| let arr = array.as_string::<i64>(); | ||
| for i in 0..arr.len() { | ||
| if arr.is_null(i) { | ||
| union.push_none(); | ||
| } else { | ||
| union.push(JsonUnionField::Str(arr.value(i).to_string())); | ||
| } | ||
| } | ||
| } | ||
| DataType::Utf8View => { | ||
| let arr = array.as_string_view(); | ||
| for i in 0..arr.len() { | ||
| if arr.is_null(i) { | ||
| union.push_none(); | ||
| } else { | ||
| union.push(JsonUnionField::Str(arr.value(i).to_string())); | ||
| } | ||
| } | ||
| } |
Member
There was a problem hiding this comment.
More of a nit and note about future work, we could probably remove a lot of this code if we impl JsonUnion::from_iter where the Iterator::Item is an Option<T> where None represents null 🤔
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.