Spark: Binary reader supports String for Array/Map reading #395
Conversation
|
possible to add test cases? |
|
I’m encountering Docker setup failures when running Array(String)
Array(Int)
Array(Array(String))
Map(String, String)
Map(String, Array(String))
|
| val arrayValue = value.asInstanceOf[ClickHouseArraySequence] | ||
| val convertedArray = Array.tabulate(arrayValue.length) { i => | ||
| decodeValue( | ||
| arrayValue.getValue(i, createClickHouseValue(null, _dataType)), |
There was a problem hiding this comment.
createClickHouseValue(null, _dataType) for every element looks costly. does clickhouse-jave provide constant variables for default null values?
There was a problem hiding this comment.
getValue needs a corresponding ClickHouseValue container based on data type (e.g. ClickHouseIntegerValue, ClickHouseLongValue). Unfortunately, there's no factory method to create it based on input type. createClickHouseValue creates empty containers based on input data type.
| case IntegerType => ClickHouseIntegerValue.of(if (isNull) 0 else rawValue.asInstanceOf[Int]) | ||
| case LongType => ClickHouseLongValue.of(if (isNull) 0L else rawValue.asInstanceOf[Long]) | ||
| case DoubleType => ClickHouseDoubleValue.of(if (isNull) 0.0 else rawValue.asInstanceOf[Double]) | ||
| case FloatType => ClickHouseFloatValue.of(if (isNull) 0.0f else rawValue.asInstanceOf[Float]) |
There was a problem hiding this comment.
also, can we use the clickhouse-java defined default value variable instead of hardcoding it for each data type?
There was a problem hiding this comment.
Good one. will fix
There was a problem hiding this comment.
Call ofNull/ofEmpty for each ClickHouse value type provided.
Not sure which failures you encountered, it works well on my local env, anyway, manually testing is also good. |
|
@pan3793 would appreciate your review again. |
pan3793
changed the title
Summary
Fix binary reader String handling in Arrays and Map (e.g.,
Array(String),Map(String, String)).