Multi modules: separate Java library from native libs by gotson · Pull Request #1387 · xerial/sqlite-jdbc

gotson · 2026-03-05T05:36:41Z

Another approach is proposed here: #1388

Motivation

This addresses #1273

Moving the single project build into multi-modules, the goal being to separate the native libraries into separate jars, which can then be selectively added to a project depending on the needs.

Implementation

The native libraries have been moved each to a dedicated project/artifact, with the following file naming: sqlite-jdbc-jni-libs-${OS}-${ARCH}

In addition, some aggregation projects have been added:

per OS: sqlite-jdbc-jni-libs-${OS}
all OSes (except Android): sqlite-jdbc-jni-libs-all

Here is the full list:

sqlite-jdbc-jni-libs-all

sqlite-jdbc-jni-libs-android
sqlite-jdbc-jni-libs-android-aarch64
sqlite-jdbc-jni-libs-android-arm
sqlite-jdbc-jni-libs-android-x86
sqlite-jdbc-jni-libs-android-x86_64

sqlite-jdbc-jni-libs-freebsd
sqlite-jdbc-jni-libs-freebsd-aarch64
sqlite-jdbc-jni-libs-freebsd-x86
sqlite-jdbc-jni-libs-freebsd-x86_64

sqlite-jdbc-jni-libs-linux
sqlite-jdbc-jni-libs-linux-aarch64
sqlite-jdbc-jni-libs-linux-arm
sqlite-jdbc-jni-libs-linux-armv6
sqlite-jdbc-jni-libs-linux-armv7
sqlite-jdbc-jni-libs-linux-ppc64
sqlite-jdbc-jni-libs-linux-riscv64
sqlite-jdbc-jni-libs-linux-x86
sqlite-jdbc-jni-libs-linux-x86_64

sqlite-jdbc-jni-libs-linux_musl
sqlite-jdbc-jni-libs-linux_musl-aarch64
sqlite-jdbc-jni-libs-linux_musl-x86
sqlite-jdbc-jni-libs-linux_musl-x86_64

sqlite-jdbc-jni-libs-mac
sqlite-jdbc-jni-libs-mac-x86_64
sqlite-jdbc-jni-libs-macos-aarch64

sqlite-jdbc-jni-libs-windows
sqlite-jdbc-jni-libs-windows-aarch64
sqlite-jdbc-jni-libs-windows-armv7
sqlite-jdbc-jni-libs-windows-x86
sqlite-jdbc-jni-libs-windows-x86_64

The Java code has been moved into a sqlite-jdbc-jni project/artifact. This does not include the native libs anymore.

The existing sqlite-jdbc project/artifact has been replaced by an aggregation POM that will pull:

sqlite-jdbc-jni
sqlite-jdbc-jni-libs-all

It will not pull the Android libs, since anyway those have to be handled specifically and cannot be extracted from the JAR resources by the loader.

commit and push commented for now

This reverts commit 8d6a7ca.

This reverts commit 1880772.

gotson · 2026-03-05T05:45:30Z

@kkriske tagging you here for information. All the tests are green for GraalVM, but since the packaging has changed quite a bit i don't know if there's anything we would need to pay attention to in particular.

gotson · 2026-03-05T05:45:52Z

@michael-o if you could have a look that would be great, this would address your request to get a JAR free of native libs.

michael-o · 2026-03-05T08:25:26Z

@michael-o if you could have a look that would be great, this would address your request to get a JAR free of native libs.

Checking...

michael-o

First review:

Have you also considered using classifiers for JARs which contains the native libraries instead of separate modules?
Aggregator modules must always be of packaging POM, not JAR.

I'll go over to native compilation.

.github/workflows/ci.yml

michael-o · 2026-03-05T08:43:00Z

sqlite-jdbc-jni/pom.xml

+    <packaging>jar</packaging>
+
+    <name>SQLite JDBC</name>
+    <description>SQLite JDBC library without native libraries</description>


The artifact id is a contradition to the description, no? It says JNI, yet it does not contain any native libraries?

JNI is about the code, not necessarily about the bundling of the native libraries compiled with the JNI wrapper. Open to better naming.

What about: SQLite JDBC library with a JNI bridge?

You need to keep in mind that in the future one want to create an FFM version of this library for Java 21+ and the name wouldn't fit anymore. See what we are doing in Tomat to interact with OpenSSL: We have a JNI-based library and now an FFM-based implementation: https://tomcat.apache.org/presentations/2024-06-05-coceu-OpenSSL-and-FFM.pdf

In case there is a FFM implementation, we would split the sqlite-jdbc-jni:

we would add a new artifact that would contain the JDBC code and an interface for native access, maybe something like sqlite-jdbc-api (inspired by sqlf4j-api

the JNI code would remain in sqlite-jdbc-jni

we would add another implementation sqlite-jdbc-ffm

If you have better artifact names to propose please do, naming is always hard !

This sounds resonable and why not do it now, I mean the split? No need to create the FFM module now, can be done later.

Makefile.common

gotson · 2026-03-05T09:04:15Z

Have you also considered using classifiers for JARs which contains the native libraries instead of separate modules?

I am not very familiar with them, i read about it from the Gradle blog posts, but it seems like it's confusing to use. I don't see the upside.

Would you have more to share on that and how it would help ?

Aggregator modules must always be of packaging POM, not JAR.

That is for Maven aggregators, not for jar aggregators. The library aggregator for windows for example is just a jar with no code, just deps.

michael-o · 2026-03-05T12:07:23Z

Have you also considered using classifiers for JARs which contains the native libraries instead of separate modules?

I am not very familiar with them, i read about it from the Gradle blog posts, but it seems like it's confusing to use. I don't see the upside.

No, consider that you have one main artifact and multiple side artifacts. javadoc and sources are two of them. One could create the JAR with Maven assembly plugin and attach them.

Example: org.sonarsource.scanner.cli:sonar-scanner-cli:5.0.2.4997

          <plugin>
            <artifactId>maven-assembly-plugin</artifactId>
            <executions>
              <execution>
                <id>assemble-linux</id>
                <phase>package</phase>
                <goals>
                  <goal>single</goal>
                </goals>
                <configuration>
                  <finalName>sonar-scanner-${project.version}</finalName>
                  <escapeString>\</escapeString>
                  <descriptors>
                    <descriptor>src/main/assembly/dist-linux.xml</descriptor>
                  </descriptors>
                  <filters>
                    <filter>src/main/assembly/filter-dist.properties</filter>
                  </filters>
                </configuration>
              </execution>
            </executions>
          </plugin>

Would you have more to share on that and how it would help ?

Aggregator modules must always be of packaging POM, not JAR.

That is for Maven aggregators, not for jar aggregators. The library aggregator for windows for example is just a jar with no code, just deps.

No, that is wrong. These are classical POM dependencies which the sole purpose to aggregate other dependencies, see:

osipovmi@deblndw011x:~/var/Projekte/sqlite-jdbc/sqlite-jdbc-jni-libs/sqlite-jdbc-jni-libs-freebsd (multi-modules =)
$ tree
.
├── pom.xml
└── target
    ├── maven-archiver
    │   └── pom.properties
    └── sqlite-jdbc-jni-libs-freebsd-3.51.2.1-SNAPSHOT.jar

3 directories, 3 files
osipovmi@deblndw011x:~/var/Projekte/sqlite-jdbc/sqlite-jdbc-jni-libs/sqlite-jdbc-jni-libs-freebsd (multi-modules =)
$ tar tzf target/sqlite-jdbc-jni-libs-freebsd-3.51.2.1-SNAPSHOT.jar
META-INF/
META-INF/MANIFEST.MF
META-INF/maven/
META-INF/maven/org.xerial/
META-INF/maven/org.xerial/sqlite-jdbc-jni-libs-freebsd/
META-INF/maven/org.xerial/sqlite-jdbc-jni-libs-freebsd/pom.xml
META-INF/maven/org.xerial/sqlite-jdbc-jni-libs-freebsd/pom.properties

This module must be a pom as it does not deliver any resources/classes itself, but solely collects all native JARs.

gotson · 2026-03-06T02:39:19Z

@michael-o thanks for the feedback, this is really appreciated. Maven is not my area of expertise.

This module must be a pom as it does not deliver any resources/classes itself, but solely collects all native JARs.

You are right, i got confused with the Maven aggregators.

No, consider that you have one main artifact and multiple side artifacts. javadoc and sources are two of them. One could create the JAR with Maven assembly plugin and attach them.

Example: org.sonarsource.scanner.cli:sonar-scanner-cli:5.0.2.4997

I read up on that yesterday, indeed we could keep a single project, and use the Maven Jar Plugin to build different classifier jars.

However here are the shortcomings i see with this approach:

we would effectively publish the same native libraries multiple times to Central, ie the existing jar with everything, and differnt native classifiers containing various combinations of native libraries.
if we want to publish each OS+ARCH combination, and also OS with all arch, and also 1 with all the native libs, this becomes tricky. It seems we can't handle transitive dependencies with classifiers ?
the version of all the artifacts become highly coupled. In the future we may want to stop publishing again the native libs when only the code has changed (for example for patch releases).

My understanding may be wrong though, as i am not super familiar with Maven classifiers.

Would be happy to hear about the pros/cons of both approaches.

gotson · 2026-03-06T03:08:48Z

This module must be a pom as it does not deliver any resources/classes itself, but solely collects all native JARs.

~~so i tried changing it, but then the test fails because the deps cannot be resolved.~~

~~https://github.com/xerial/sqlite-jdbc/actions/runs/22747190311/job/65973467296~~

~~Any tips on that ?~~

Never mind, i figured out i needed to add <type>pom</type> to those deps.

gotson · 2026-03-06T03:25:27Z

but now GraalVM tests fails because of the POM type dependency used for tests T_T

gotson · 2026-03-06T03:48:57Z

for the sake of testing i created another PR that uses classifier jars: #1388

michael-o · 2026-03-06T15:53:42Z

@michael-o thanks for the feedback, this is really appreciated. Maven is not my area of expertise.

This module must be a pom as it does not deliver any resources/classes itself, but solely collects all native JARs.

You are right, i got confused with the Maven aggregators.

No, consider that you have one main artifact and multiple side artifacts. javadoc and sources are two of them. One could create the JAR with Maven assembly plugin and attach them.
Example: org.sonarsource.scanner.cli:sonar-scanner-cli:5.0.2.4997

I read up on that yesterday, indeed we could keep a single project, and use the Maven Jar Plugin to build different classifier jars.

However here are the shortcomings i see with this approach:
* we would effectively publish the same native libraries multiple times to Central, ie the existing jar with everything, and differnt native classifiers containing various combinations of native libraries.

* if we want to publish each OS+ARCH combination, and also OS with all arch, and also 1 with all the native libs, this becomes tricky. It seems we can't handle transitive dependencies with classifiers ?

* the version of all the artifacts become highly coupled. In the future we may want to stop publishing again the native libs when only the code has changed (for example for patch releases).
My understanding may be wrong though, as i am not super familiar with Maven classifiers.

Would be happy to hear about the pros/cons of both approaches.

I'll get back to this next week with more guidance.

kkriske · 2026-03-06T21:31:10Z

@kkriske tagging you here for information. All the tests are green for GraalVM, but since the packaging has changed quite a bit i don't know if there's anything we would need to pay attention to in particular.

As long as the resource locations of the actual libs stays the same, it doesn't matter they are in separate artifacts. I don't see any conceptual issues with this towards native-image.

gotson and others added 24 commits March 4, 2026 18:09

multi modules

fd57d27

makefile adjustments

710c3d5

jreleaser

f5dc368

rename bare project to jni, add pom descriptions

7602bbb

plugins

945160b

add workflow to bump sqlite version

4381ff0

commit and push commented for now

update makefile following module renaming

e1c89e4

adjust build native workflow

a85e2bb

adjust ci

18f4b18

move spotless to parent pom and update version

5dbfed3

fix java version

8eeb945

bump minimum java from 11 to 17

3b7de99

move maven enforcer to parent and bump minimum java version to 17

3dce792

adjust makefile

c5d37ba

adjust makefile

4f44548

fix multiple class loader test edge cases

e7677da

try debug CI failing

8d6a7ca

try debug CI failing

1880772

fix makefile

13d038f

fix makefile for codesign

65ccf1d

fix makefile for codesign

0e34aeb

chore: update native libraries

5bdcb8e

Revert "try debug CI failing"

d75e242

This reverts commit 8d6a7ca.

Revert "try debug CI failing"

d9a513b

This reverts commit 1880772.

gotson added the review wanted This needs more eyes label Mar 5, 2026

gotson mentioned this pull request Mar 5, 2026

Modular or pure Java builds for smaller binaries #1273

Open

update docs

fa6916d

michael-o suggested changes Mar 5, 2026

View reviewed changes

gotson added 3 commits March 6, 2026 10:41

fix pom packaging

c5e176f

use shell and tr for lowercase

0d965c2

move spotless plugin to dedicated profile

9d49cec

fix pom deps

b59bd25

gotson mentioned this pull request Mar 6, 2026

Classifier jars: separate Java library from native libs #1388

Draft

Conversation

gotson commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Implementation

Uh oh!

gotson commented Mar 5, 2026

Uh oh!

gotson commented Mar 5, 2026

Uh oh!

michael-o commented Mar 5, 2026

Uh oh!

michael-o left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

michael-o Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

gotson Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

michael-o Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

gotson Mar 6, 2026

Choose a reason for hiding this comment

Uh oh!

michael-o Mar 6, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gotson commented Mar 5, 2026

Uh oh!

michael-o commented Mar 5, 2026

Uh oh!

gotson commented Mar 6, 2026

Uh oh!

gotson commented Mar 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gotson commented Mar 6, 2026

Uh oh!

gotson commented Mar 6, 2026

Uh oh!

michael-o commented Mar 6, 2026

Uh oh!

kkriske commented Mar 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gotson commented Mar 5, 2026 •

edited

Loading

michael-o left a comment •

edited

Loading

gotson commented Mar 6, 2026 •

edited

Loading