Support Hive data mask E2E test and data encrypt DML statement test. #36492

ClaireLytt · 2025-09-06T10:07:49Z

Changes proposed in this pull request:

Before committing this PR, I'm sure that I have checked the following options:

My code follows the code of conduct of this project.
I have self-reviewed the commit code.
I have (or in comment I request) added corresponding labels for the pull request.
I have passed maven check locally : ./mvnw clean install -B -T1C -Dmaven.javadoc.skip -Dmaven.jacoco.skip -e.
I have made corresponding changes to the documentation.
I have added corresponding unit tests for my changes.
I have updated the Release Notes of the current development version. For more details, see Update Release Note

ClaireLytt · 2025-09-06T10:13:40Z

strongduanmu · 2025-09-06T12:31:17Z

...java/org/apache/shardingsphere/test/e2e/env/container/atomic/storage/impl/HiveContainer.java

-                    "beeline -u \"jdbc:hive2://localhost:10000/default\" -e \"CREATE DATABASE IF NOT EXISTS encrypt; CREATE DATABASE IF NOT EXISTS expected_dataset;\"");
-            System.out.println("Databases created successfully in postStart()");
+                    "beeline -u \"jdbc:hive2://localhost:10000/default\" -e \"CREATE DATABASE IF NOT EXISTS encrypt; "
+                            + "CREATE DATABASE IF NOT EXISTS expected_dataset; CREATE DATABASE IF NOT EXISTS mask;\"");


Why execute CREATE DATABASE IF NOT EXISTS expected_dataset; CREATE DATABASE IF NOT EXISTS mask; sql here?

I have changed this hardcoding approach in the new commit.

strongduanmu · 2025-09-06T12:34:00Z

.../sql/src/test/java/org/apache/shardingsphere/test/e2e/sql/env/DataSetEnvironmentManager.java

@@ -94,10 +94,19 @@ public void fillData() {
            }
            String insertSQL;
            try (Connection connection = dataSourceMap.get(dataNode.getDataSourceName()).getConnection()) {
-                DatabaseType databaseType = DatabaseTypeFactory.get(connection.getMetaData().getURL());
-                insertSQL = generateInsertSQL(dataNode.getTableName(), dataSetMetaData.getColumns(), databaseType);
+                DatabaseType currentDatabaseType;


If we can get databaseType from class field, I think we can remove DatabaseType databaseType = DatabaseTypeFactory.get(connection.getMetaData().getURL());

.../sql/src/test/java/org/apache/shardingsphere/test/e2e/sql/env/DataSetEnvironmentManager.java

…into issueNo

ClaireLytt · 2025-09-11T13:18:06Z

...java/org/apache/shardingsphere/test/e2e/env/container/atomic/storage/impl/HiveContainer.java

strongduanmu · 2025-09-08T01:23:54Z

...java/org/apache/shardingsphere/test/e2e/env/container/atomic/storage/impl/HiveContainer.java

+            return;
+        }
+        StringBuilder createDatabaseSQL = new StringBuilder();
+        for (String databaseName : allDatabaseNames) {


Please rename databaseName to each.

strongduanmu · 2025-09-14T02:47:12Z

.../sql/src/test/java/org/apache/shardingsphere/test/e2e/sql/env/DataSetEnvironmentManager.java

-                insertSQL = generateInsertSQL(dataNode.getTableName(), dataSetMetaData.getColumns(), databaseType);
+                String insertTableName = dataNode.getTableName();
+                if ("Hive".equalsIgnoreCase(databaseType.getType())) {
+                    insertTableName = dataNode.getDataSourceName() + "." + dataNode.getTableName();


Why add dataNode.getDataSourceName() in insertTableName?

I have removed it.

strongduanmu · 2025-09-14T02:52:29Z

.../sql/src/test/java/org/apache/shardingsphere/test/e2e/sql/env/DataSetEnvironmentManager.java

+                Object value = each.getValue();
+                int index = each.getIndex();
+                if ("Hive".equalsIgnoreCase(databaseType.getType())) {
+                    if (value instanceof Date) {


Why call setString for Date type?

Since the Date type does not support the setObject, as you can see in the source code. I have already used the setDate which is more appropriate.

strongduanmu · 2025-09-14T02:54:25Z

.../sql/src/test/java/org/apache/shardingsphere/test/e2e/sql/env/DataSetEnvironmentManager.java

-                    setParameters(preparedStatement, each);
-                    preparedStatement.addBatch();
+                boolean isHive = "Hive".equalsIgnoreCase(databaseType.getType());
+                if (isHive) {


Can you use DatabaseMetaData.supportsBatchUpdates() to judge this logic?

strongduanmu · 2025-09-14T02:55:19Z

.../sql/src/test/java/org/apache/shardingsphere/test/e2e/sql/env/DataSetEnvironmentManager.java

+                return DatabaseTypeFactory.get(url);
+            } catch (final SQLFeatureNotSupportedException ex) {
+                String driverName = connection.getMetaData().getDriverName();
+                if (driverName != null && driverName.toLowerCase().contains("hive")) {


Please put null on the left hand.

strongduanmu · 2025-09-14T02:55:51Z

.../sql/src/test/java/org/apache/shardingsphere/test/e2e/sql/env/DataSetEnvironmentManager.java

+            } catch (final SQLFeatureNotSupportedException ex) {
+                String driverName = connection.getMetaData().getDriverName();
+                if (driverName != null && driverName.toLowerCase().contains("hive")) {
+                    return DatabaseTypeFactory.get("jdbc:hive2://localhost:10000/default");


Why not just return HiveDatabaseType?

If return HiveDatabaseType ,it may lead to the occurrence of ServiceProviderNotFoundException: No implementation class load from SPI 'DialectDatabaseMetaData' with type 'null' errors, so I choose to use TypedSPILoader

test/e2e/sql/src/test/java/org/apache/shardingsphere/test/e2e/sql/it/sql/dml/BaseDMLE2EIT.java

ClaireLytt added 2 commits September 6, 2025 18:00

support mask test

b89c476

Merge branch 'apache:master' into issueNo

57756e3

strongduanmu added in: test feature: mask db: Hive labels Sep 6, 2025

strongduanmu added this to the 5.5.3 milestone Sep 6, 2025

strongduanmu requested changes Sep 6, 2025

View reviewed changes

ClaireLytt added 8 commits September 7, 2025 16:38

update code

96da46c

Merge branch 'issueNo' of https://github.com/ClaireLytt/shardingsphere …

d03e4fd

…into issueNo

add dml test

b72712c

Merge master into issueNo and resolve conflicts

8b806a7

add encrypt dml test

ef6ec4f

update

a0bcd72

Merge remote-tracking branch 'apache/master' into issueNo

40c0df2

update

1f7adf9

ClaireLytt changed the title ~~Support Hive data mask E2E Test~~ Support Hive data mask E2E test and data encrypt DML statement test. Sep 11, 2025

update sql scripts

82a0c0b

strongduanmu requested changes Sep 14, 2025

View reviewed changes

ClaireLytt added 10 commits September 14, 2025 15:52

update DataSetEnvironmentManager.java

8abe871

merge

782a702

replace to each

1c1d7c0

use typedspi

5e2d7fd

Update HiveContainer.java

f964f1e

remove repeated logic

fbdce0e

remove repeated logic

caf2d74

it-env

e90e400

skip batch test

d90f580

Merge remote-tracking branch 'apache/master' into issueNo

309de31

ClaireLytt added 2 commits September 18, 2025 11:48

merge master and update

cb96217

update

7fe41cb

Support Hive data mask E2E test and data encrypt DML statement test. #36492

Are you sure you want to change the base?

Support Hive data mask E2E test and data encrypt DML statement test. #36492

Uh oh!

Conversation

ClaireLytt commented Sep 6, 2025

Changes proposed in this pull request:

Uh oh!

ClaireLytt commented Sep 6, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ClaireLytt commented Sep 11, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!