postgresql binary support #36372

ShenFeng312 · 2025-08-20T12:10:23Z

Changes proposed in this pull request:

Before committing this PR, I'm sure that I have checked the following options:

My code follows the code of conduct of this project.
I have self-reviewed the commit code.
I have (or in comment I request) added corresponding labels for the pull request.
I have passed maven check locally : ./mvnw clean install -B -T1C -Dmaven.javadoc.skip -Dmaven.jacoco.skip -e.
I have made corresponding changes to the documentation.
I have added corresponding unit tests for my changes.
I have updated the Release Notes of the current development version. For more details, see Update Release Note

ShenFeng312 · 2025-08-21T07:03:01Z

@terrymanu @RaigorJiang
I ran the integration tests and unit tests locally, and everything seems fine. In this PR, I have removed some unnecessary test cases and incorrect implementations. However, I am not sure how to add my integration tests since they seem to be incompatible with the previous ones. Additionally, I couldn't find any testcontainers-related dependencies in both the shardingsphere-protocol-postgresql module and its parent module. Where should I add them? Please help me complete the PR. Below is my test code. There's also a part in PostgreSQLArrayBinaryProtocolValueTest, but due to dependency issues, I have temporarily commented it out.

import org.apache.shardingsphere.db.protocol.postgresql.packet.command.query.extended.bind.protocol.util.codec.decoder.PgBinaryObj;
import org.apache.shardingsphere.db.protocol.postgresql.packet.command.query.extended.bind.protocol.util.codec.encoder.*;
import org.postgresql.core.Oid;
import org.postgresql.core.QueryExecutor;
import org.postgresql.jdbc.PgConnection;
import org.postgresql.jdbc.PgStatement;

import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.math.BigDecimal;
import java.nio.charset.StandardCharsets;
import java.sql.*;
import java.util.Arrays;
import java.util.TimeZone;

public class ClientTest {

    public static void main(String[] args) throws IOException {


        String url = "jdbc:postgresql://localhost:3307/test";

        String username = "root";
        String password = "root";
        Short[][] param1 = new Short[2][4];
        param1[0][0] = 1;
        param1[0][1] = 2;
        param1[1][0] = 3;
        Integer[][] param2 = new Integer[2][4];
        param2[0][0] = 1;
        param2[0][1] = 2;
        param2[1][0] = 3;
        Long[][] param3 = new Long[2][4];
        param3[0][0] = 1L;
        param3[0][1] = 2L;
        param3[1][0] = 3L;
        Float[][] param4 = new Float[2][4];
        param4[0][0] = 1.1F;
        param4[0][1] = 2.2F;
        param4[1][0] = Float.NaN;
        param4[1][1] = Float.POSITIVE_INFINITY;
        param4[1][2] = Float.NEGATIVE_INFINITY;
        Double[][] param5 = new Double[2][4];
        param5[0][0] = 1.1;
        param5[0][1] = 2.2;
        param5[1][0] = Double.NaN;
        param5[1][1] = Double.POSITIVE_INFINITY;
        param5[1][2] = Double.NEGATIVE_INFINITY;
        Number[][] param6 = new Number[2][4];
        param6[0][0] = new BigDecimal("1.1");
        param6[0][1] = new BigDecimal("2.2");
        param6[1][0] = Double.NaN;
        param6[1][1] = Double.POSITIVE_INFINITY;
        param6[1][2] = Double.NEGATIVE_INFINITY;
        Boolean[][] param7 = new Boolean[2][4];
        param7[0][0] = true;
        param7[0][1] = false;
        String[][] param8 = new String[2][4];
        param8[0][0] = "aaa";
        param8[0][1] = "bbbb";
        String[][] param9 = new String[2][4];
        param9[0][0] = "aaa";
        param9[0][1] = "bbbb";
        byte[][][] param10 = new byte[2][4][];
        param10[0][0] = new byte[10];
        param10[0][1] = "aaa".getBytes(StandardCharsets.UTF_8);


       String query = "SELECT\n" +
                "    ?::int2[]       AS int2_array,       -- smallint\n" +
                "   ?::int4[] AS int4_array,   -- integer\n" +
                "    ?::int8[] AS int8_array, -- bigint\n" +
                "    ?::float4[] AS float4_array, -- real\n" +
                "    ?::float8[] AS float8_array, -- double precision\n" +
                "    ?::numeric[] AS numeric_array, -- numeric / decimal\n" +
                "    ?::bool[] AS bool_array,       -- boolean\n" +
                "    ?::text[] AS text_array,      -- text\n" +
                "   ?::varchar[] AS varchar_array, -- varchar\n" +
                "    ?::bytea[] AS bytea_array, -- bytea\n" +
                "    ?::date[] AS date_array,      -- date\n" +
                "   ?::time[] AS time_array,          -- time without time zone\n" +
                "    ?::timestamp[] AS timestamp_array, -- timestamp\n" +
                "    ?::bool AS boolv, -- bool\n" +
                "    ?::bytea AS byteav, -- bytea\n" +
                "    ?::date AS datev, -- date\n" +
                "    ?::float8  AS float8v, -- float8\n" +
                "    ?::float4 AS float4v, -- float4\n" +
                "    ?::int2 AS int2v, -- int2\n" +
                "    ?::int4 AS int4v, -- int4\n" +
                "    ?::int8 AS int8v, -- int8\n" +
                "    ?::numeric AS numericv, -- numeric\n" +
                "    ?::text AS textv, -- text\n" +
                "    ?::varchar AS varcharv, -- varchar\n" +
                "    ?::time AS timev, -- time\n" +
                "    ?::timestamp AS timestampv -- timestamp\n";

        int[] arrayOids = new int[]{Oid.INT2_ARRAY, Oid.INT4_ARRAY, Oid.INT8_ARRAY, Oid.FLOAT4_ARRAY, Oid.FLOAT8_ARRAY, Oid.NUMERIC_ARRAY, Oid.BOOL_ARRAY, Oid.TEXT_ARRAY, Oid.VARCHAR_ARRAY, Oid.BYTEA_ARRAY};
        String[] arrayOidNames = new String[]{"int2[]", "int4[]", "int8[]", "float4[]", "float8[]", "numeric[]", "bool[]", "text[]", "varchar[]", "bytea[]"};
        byte[][] sendResult = new byte[26][];
        Object[] params = new Object[]{param1, param2, param3, param4, param5, param6, param7, param8, param9, param10};
        try (Connection connection = DriverManager.getConnection(url, username, password)) {
            PgConnection pgConnection = (PgConnection) connection;
            QueryExecutor queryExecutor = pgConnection.getQueryExecutor();
            queryExecutor.addBinarySendOid(Oid.DATE);
            queryExecutor.addBinarySendOid(Oid.DATE_ARRAY);
            queryExecutor.addBinarySendOid(Oid.TIME);
            queryExecutor.addBinarySendOid(Oid.TIME_ARRAY);
            queryExecutor.addBinarySendOid(Oid.BOOL_ARRAY);
            queryExecutor.addBinarySendOid(Oid.TIMESTAMP);
            queryExecutor.addBinarySendOid(Oid.TIMESTAMP_ARRAY);
            queryExecutor.addBinarySendOid(Oid.NUMERIC_ARRAY);
            queryExecutor.addBinarySendOid(Oid.VARCHAR);
            queryExecutor.addBinarySendOid(Oid.TEXT);
            queryExecutor.addBinarySendOid(Oid.BOOL);


            queryExecutor.addBinaryReceiveOid(Oid.DATE);
            queryExecutor.addBinaryReceiveOid(Oid.DATE_ARRAY);
            queryExecutor.addBinaryReceiveOid(Oid.TIME);
            queryExecutor.addBinaryReceiveOid(Oid.TIME_ARRAY);
            queryExecutor.addBinaryReceiveOid(Oid.BOOL_ARRAY);
            queryExecutor.addBinaryReceiveOid(Oid.TIMESTAMP);
            queryExecutor.addBinaryReceiveOid(Oid.TIMESTAMP_ARRAY);
            queryExecutor.addBinaryReceiveOid(Oid.NUMERIC_ARRAY);
            queryExecutor.addBinaryReceiveOid(Oid.VARCHAR);
            queryExecutor.addBinaryReceiveOid(Oid.TEXT);
            queryExecutor.addBinaryReceiveOid(Oid.BOOL);


            PreparedStatement preparedStatement = connection.prepareStatement(query);
            PgStatement pgStatement = (PgStatement) preparedStatement;
            pgStatement.setPrepareThreshold(-1);


            for (int i = 0; i < arrayOids.length; i++) {
                ByteArrayOutputStream baos = new ByteArrayOutputStream();
                ArrayEncoding.getArrayEncoder(params[i]).toBinaryRepresentation((Object[]) params[i], arrayOids[i], baos);
                byte[] byteArray = baos.toByteArray();
                PgBinaryObj pgBinaryObj = new PgBinaryObj(byteArray);
                pgBinaryObj.setType(arrayOidNames[i]);
                sendResult[i] = byteArray;
                preparedStatement.setObject(i + 1, pgBinaryObj);
            }

            Date[][] dates = new Date[2][4];
            //1970-01-01
            dates[0][0] = new Date(-TimeZone.getDefault().getRawOffset());
            ByteArrayOutputStream baos = new ByteArrayOutputStream();
            ArrayEncoding.getArrayEncoder(dates).toBinaryRepresentation(dates, Oid.DATE_ARRAY, baos);
            byte[] byteArray = baos.toByteArray();
            PgBinaryObj dateBinary = new PgBinaryObj(byteArray);
            dateBinary.setType("date[]");
            sendResult[10] = byteArray;

            preparedStatement.setObject(11, dateBinary);

            baos.reset();
            Time[][] times = new Time[2][4];
            //time0
            times[0][0] = new Time(-TimeZone.getDefault().getRawOffset());
            ArrayEncoding.getArrayEncoder(times).toBinaryRepresentation(times, Oid.TIME_ARRAY, baos);
            byteArray = baos.toByteArray();
            dateBinary = new PgBinaryObj(byteArray);
            dateBinary.setType("time[]");
            sendResult[11] = byteArray;

            preparedStatement.setObject(12, dateBinary);

            baos.reset();
            Timestamp[][] timestamps = new Timestamp[2][4];
            //timestamps0 1970-01-01
            timestamps[0][0] = new Timestamp(-TimeZone.getDefault().getRawOffset());
            ArrayEncoding.getArrayEncoder(timestamps).toBinaryRepresentation(timestamps, Oid.TIMESTAMP_ARRAY, baos);
            byteArray = baos.toByteArray();
            dateBinary = new PgBinaryObj(byteArray);
            dateBinary.setType("timestamp[]");
            sendResult[12] = byteArray;

            preparedStatement.setObject(13, dateBinary);

            baos.reset();
            BooleanArrayEncoder.INSTANCE.write(true,baos);
            byteArray = baos.toByteArray();
            byteArray = Arrays.copyOfRange(byteArray, 4, byteArray.length);
            dateBinary = new PgBinaryObj(byteArray);
            dateBinary.setType("bool");
            sendResult[13] = byteArray;

            preparedStatement.setObject(14,dateBinary);


            byteArray = new byte[10];
            dateBinary = new PgBinaryObj(byteArray);
            dateBinary.setType("bytea");
            sendResult[14] = byteArray;

            preparedStatement.setObject(15, dateBinary);

            baos.reset();
            DateArrayEncoder.INSTANCE.write(new Date(-TimeZone.getDefault().getRawOffset()),baos);
            byteArray = baos.toByteArray();
            byteArray = Arrays.copyOfRange(byteArray, 4, byteArray.length);
            dateBinary = new PgBinaryObj(byteArray);
            dateBinary.setType("date");
            sendResult[15] = byteArray;

            preparedStatement.setObject(16, dateBinary);

            baos.reset();
            Float8ArrayEncoder.INSTANCE.write(1.2,baos);
            byteArray = baos.toByteArray();
            byteArray = Arrays.copyOfRange(byteArray, 4, byteArray.length);
            dateBinary = new PgBinaryObj(byteArray);
            dateBinary.setType("float8");
            sendResult[16] = byteArray;

            preparedStatement.setObject(17, dateBinary);

            baos.reset();
            Float4ArrayEncoder.INSTANCE.write(1.2f,baos);
            byteArray = baos.toByteArray();
            byteArray = Arrays.copyOfRange(byteArray, 4, byteArray.length);
            dateBinary = new PgBinaryObj(byteArray);
            dateBinary.setType("float4");
            sendResult[17] = byteArray;

            preparedStatement.setObject(18, dateBinary);

            baos.reset();
            Int2ArrayEncoder.INSTANCE.write((short) 1,baos);
            byteArray = baos.toByteArray();
            byteArray = Arrays.copyOfRange(byteArray, 4, byteArray.length);
            dateBinary = new PgBinaryObj(byteArray);
            dateBinary.setType("int2");
            sendResult[18] = byteArray;

            preparedStatement.setObject(19, dateBinary);

            baos.reset();
            Int4ArrayEncoder.INSTANCE.write( 1,baos);
            byteArray = baos.toByteArray();
            byteArray = Arrays.copyOfRange(byteArray, 4, byteArray.length);
            dateBinary = new PgBinaryObj(byteArray);
            dateBinary.setType("int4");
            sendResult[19] = byteArray;

            preparedStatement.setObject(20, dateBinary);

            baos.reset();
            Int8ArrayEncoder.INSTANCE.write(1L,baos);
            byteArray = baos.toByteArray();
            byteArray = Arrays.copyOfRange(byteArray, 4, byteArray.length);
            dateBinary = new PgBinaryObj(byteArray);
            dateBinary.setType("int8");
            sendResult[20] = byteArray;

            preparedStatement.setObject(21, dateBinary);

            baos.reset();
            NumericArrayEncoder.INSTANCE.write(new BigDecimal("1.2"),baos);
            byteArray = baos.toByteArray();
            byteArray = Arrays.copyOfRange(byteArray, 4, byteArray.length );
            dateBinary = new PgBinaryObj(byteArray);
            dateBinary.setType("numeric");
            sendResult[21] = byteArray;

            preparedStatement.setObject(22, dateBinary);

            baos.reset();
            StringArrayEncoder.INSTANCE.write("testString",baos);
            byteArray = baos.toByteArray();
            byteArray = Arrays.copyOfRange(byteArray, 4, byteArray.length );
            dateBinary = new PgBinaryObj(byteArray);
            dateBinary.setType("text");
            sendResult[22] = byteArray;

            preparedStatement.setObject(23, dateBinary);

            baos.reset();
            StringArrayEncoder.INSTANCE.write("testVarchar",baos);
            byteArray = baos.toByteArray();
            byteArray = Arrays.copyOfRange(byteArray, 4, byteArray.length );
            dateBinary = new PgBinaryObj(byteArray);
            dateBinary.setType("varchar");
            sendResult[23] = byteArray;

            preparedStatement.setObject(24, dateBinary);

            baos.reset();
            TimeArrayEncoder.INSTANCE.write(new Time(-TimeZone.getDefault().getRawOffset()),baos);
            byteArray = baos.toByteArray();
            byteArray = Arrays.copyOfRange(byteArray, 4, byteArray.length );
            dateBinary = new PgBinaryObj(byteArray);
            dateBinary.setType("time");
            sendResult[24] = byteArray;

            preparedStatement.setObject(25, dateBinary);

            baos.reset();
            TimestampArrayEncoder.INSTANCE.write(new Timestamp(-TimeZone.getDefault().getRawOffset()),baos);
            byteArray = baos.toByteArray();
            byteArray = Arrays.copyOfRange(byteArray, 4, byteArray.length );
            dateBinary = new PgBinaryObj(byteArray);
            dateBinary.setType("timestamp");
            sendResult[25] = byteArray;

            preparedStatement.setObject(26, dateBinary);

//
//            baos.reset();
//            Int2ArrayEncoder.INSTANCE.write((short) 1,baos);
//            byteArray = baos.toByteArray();
//            byteArray = Arrays.copyOfRange(byteArray, 4, byteArray.length );
//            dateBinary = new PgBinaryObj(byteArray);
//            dateBinary.setType("int2");
//            sendResult[18] = byteArray;



            preparedStatement.execute();


            ResultSet resultSet = preparedStatement.getResultSet();
            resultSet.next();
            Object value = resultSet.getObject(6);
//            value.toString();
            value = resultSet.getObject(11);
            value = resultSet.getObject(12);
            value = resultSet.getObject(13);
            value = resultSet.getObject(14);
            for (int col = 1; col <= resultSet.getMetaData().getColumnCount(); col++) {
                value = resultSet.getBytes(col);
                boolean equals = new String((byte[]) value, StandardCharsets.US_ASCII).equals(new String((byte[]) sendResult[col - 1], StandardCharsets.US_ASCII));
                System.out.println("col:"+col+ equals);
            }


        } catch (SQLException e) {
            e.printStackTrace();
        }
    }
}

ShenFeng312 · 2025-08-21T07:06:05Z

Additionally, this PR contains some unused Decoder classes. I will keep them if we need them, but if not, I will remove them.

ShenFeng312 · 2025-08-21T10:31:00Z

[ERROR] /home/shenfeng/github/shardingsphere/database/protocol/type/postgresql/src/main/java/org/apache/shardingsphere/database/protocol/postgresql/packet/command/query/extended/bind/protocol/PostgreSQLArrayParameterDecoder.java:168:18: Control variable 'i' is modified. [ModifiedControlVariable] Are you serious?

ShenFeng312 · 2025-08-22T01:18:37Z

@terrymanu @RaigorJiang Hello, all test cases passed except for one Spotless exception. Can we proceed with the code review? By the way, I'd also like to discuss how to add my integration tests.

ShenFeng312 · 2025-09-01T06:26:04Z

Hi @terrymanu @RaigorJiang, any updates on this? Thanks!

...rotocol/dialect/postgresql/src/main/java/org/postgresql/jdbc/ShardingSpherePgArrayUtils.java

...stgresql/packet/command/query/extended/bind/protocol/PostgreSQLArrayBinaryProtocolValue.java

.../postgresql/packet/command/query/extended/bind/protocol/PostgreSQLArrayParameterDecoder.java

...esql/packet/command/query/extended/bind/protocol/PostgreSQLArrayBinaryProtocolValueTest.java

RaigorJiang · 2025-09-02T02:03:52Z

Hi @ShenFeng312 , thank you for your great work. Your thoroughness is evident in this pull request.
However, the code contains many irregularities, and some of the logic is complex and difficult to read, so it cannot be merged at this time.

Thank you!

ShenFeng312 · 2025-09-10T09:07:38Z

PTAL @RaigorJiang @terrymanu

RaigorJiang

Hi @ShenFeng312 , every time I read this pull request, I'm amazed at the effort you put into it. I salute you.
This pull request involves handling multiple data types, affecting many files and complex logic, especially since you also deleted some files and performed some refactoring.

I plan to spend more time researching these type implementations and again request that you adjust the code style. There are still many areas that don't conform to ShardingSphere code standards, such as comments and blank lines.
Please refer to: https://shardingsphere.apache.org/community/en/involved/conduct/code/

ShenFeng312 · 2025-09-17T08:04:20Z

Hi! @RaigorJiang
Thank you very much for your kind words and encouragement, I really appreciate it. 🙏
I have carefully followed the ShardingSphere code style guide and tried my best to fix all the issues I could identify. Since the document contains many detailed rules and I might have misunderstood some points, it’s possible that I still missed a few places.

If you don’t mind, I’d really appreciate your help in pointing out any remaining style problems so that I can fully align with the project standards.

Thanks again for your support!

ShenFeng312 marked this pull request as draft August 20, 2025 12:14

ShenFeng312 force-pushed the master branch from da66bf8 to c5ec839 Compare August 21, 2025 06:16

ShenFeng312 changed the title ~~binary support~~ postgresql binary support Aug 21, 2025

ShenFeng312 marked this pull request as ready for review August 21, 2025 07:03

fix binary protocol transmission issue in postgresql

2cbdc92

ShenFeng312 force-pushed the master branch from 5cd6d05 to 2cbdc92 Compare August 21, 2025 07:42

ShenFeng312 added 6 commits August 21, 2025 20:55

fix code style and test

1ac1dd3

fix code style and test

26ddf48

fix code style

c224c99

fix decoder

602dce6

fix code style

adbb089

fix spotless check

c95419b

This was referenced Aug 22, 2025

Updated write method for PostgreSQLInt8ArrayBinaryProtocolValue #36125

Open

Feature: Implement PostgreSQL Binary Protocol Encoders and Decoders for timetz ,timestamptz, and uuid and Their Multi-dimensional Arrays #36381

Open

ShenFeng312 added 2 commits August 25, 2025 09:20

merge

f5dc74f

fix conflicts

bbcc58b

terrymanu requested a review from RaigorJiang September 1, 2025 09:36

RaigorJiang requested changes Sep 2, 2025

View reviewed changes

fix cr

f67e284

ShenFeng312 requested a review from RaigorJiang September 3, 2025 01:19

RaigorJiang requested changes Sep 12, 2025

View reviewed changes

ShenFeng312 added 2 commits September 16, 2025 15:55

fix cr

57ff0ea

fix cr

bb639fa

ShenFeng312 added 2 commits September 16, 2025 16:52

fix cr

bbbcae6

fix cr

62ffb68

ShenFeng312 requested a review from RaigorJiang September 17, 2025 01:10

ShenFeng312 added 2 commits September 17, 2025 09:13

fix cr

40e5165

fix cr

aba7635

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

postgresql binary support #36372

postgresql binary support #36372

Uh oh!

ShenFeng312 commented Aug 20, 2025 •

edited

Loading

Uh oh!

ShenFeng312 commented Aug 21, 2025

Uh oh!

ShenFeng312 commented Aug 21, 2025

Uh oh!

ShenFeng312 commented Aug 21, 2025

Uh oh!

ShenFeng312 commented Aug 22, 2025 •

edited

Loading

Uh oh!

ShenFeng312 commented Sep 1, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

RaigorJiang commented Sep 2, 2025

Uh oh!

ShenFeng312 commented Sep 10, 2025

Uh oh!

RaigorJiang left a comment

Uh oh!

ShenFeng312 commented Sep 17, 2025

Uh oh!

Uh oh!

postgresql binary support #36372

Are you sure you want to change the base?

postgresql binary support #36372

Uh oh!

Conversation

ShenFeng312 commented Aug 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes proposed in this pull request:

Uh oh!

ShenFeng312 commented Aug 21, 2025

Uh oh!

ShenFeng312 commented Aug 21, 2025

Uh oh!

ShenFeng312 commented Aug 21, 2025

Uh oh!

ShenFeng312 commented Aug 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ShenFeng312 commented Sep 1, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

RaigorJiang commented Sep 2, 2025

Uh oh!

ShenFeng312 commented Sep 10, 2025

Uh oh!

RaigorJiang left a comment

Choose a reason for hiding this comment

Uh oh!

ShenFeng312 commented Sep 17, 2025

Uh oh!

Uh oh!

ShenFeng312 commented Aug 20, 2025 •

edited

Loading

ShenFeng312 commented Aug 22, 2025 •

edited

Loading