NIFI-9770 Added nifi-cql-bundle to bring support for Cassandra 4.X an… #10006

MikeThomsen · 2025-06-10T20:28:43Z

…d ScyllaDB.

Summary

NIFI-00000

Tracking

Please complete the following tracking steps prior to pull request creation.

Issue Tracking

Apache NiFi Jira issue created

Pull Request Tracking

Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

Pull Request based on current revision of the main branch
Pull Request refers to a feature branch with one commit containing changes

Verification

Please indicate the verification steps performed prior to pull request creation.

Build

Build completed using mvn clean install -P contrib-check
- JDK 21

Licensing

New dependencies are compatible with the Apache License 2.0 according to the License Policy
New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

Documentation formatting appears as expected in rendered files

MikeThomsen · 2025-06-10T20:32:22Z

@exceptionfactory @mattyb149 ScyllaDB aims for 100% compatibility at the interface level on Java drivers, and the basic testcontainers-powered testing I did seems to support it being a drop-in replacement for the Java DataStax driver.

The two "codec" Java files that are in the Cassandra provider package were generated with ChatGPT.

exceptionfactory

Thanks for putting this together @MikeThomsen.

This will take some time to review, but on a quick scan, I observed a number of issues related to Property Descriptor naming, logging, and use of legacy classes where more modern options are available.

The convention issues can be addressed, but one more fundamental concern is the use of the com.datadata.oss driver, which is over a year old. As mentioned on the DataStax Java Driver page, the driver is now maintained under the Apache Cassandra project.

joewitt · 2025-06-10T20:46:51Z

nifi-extension-bundles/nifi-cql-bundle/nifi-cql-nar/pom.xml

@@ -0,0 +1,51 @@
+<?xml version="1.0" encoding="UTF-8"?>


given 'nifi-cql-bundle' is the top level concept it really should say the word 'cassandra' fully written out.

so 'nifi-cassandra-bundle' at the top level at least.

The bundle includes a driver NAR for ScyllaDB support, so I think calling it nifi-cql-bundle makes more sense here. That said, it's been a while AFAICT since Cassandra and ScyllaDB support were discussed, so I'm open to splitting this into two separate bundles because we can't predict where Cassandra and ScyllaDB will be in the future.

joewitt · 2025-06-10T20:47:59Z

nifi-extension-bundles/nifi-cql-bundle/nifi-cql-nar/src/main/resources/META-INF/NOTICE

+        Expert Group and released to the public domain, as explained at
+        http://creativecommons.org/publicdomain/zero/1.0/
+
+  (ASLv2) The Netty Project


please look for other exmaples of how we reference netty. all this text should not be needed

joewitt · 2025-06-10T20:49:10Z

nifi-assembly/pom.xml

        </dependency>
+        <dependency>
+            <groupId>org.apache.nifi</groupId>
+            <artifactId>nifi-cql-nar</artifactId>


I am not in favor of including cassandra by default in our convenenience binary. It makes great sense to build a great supportable cassandra interface again but it needs to be really well written and should not be bundled by default until we find out more about usage.

For all external system integrations I'm in favor of a (perhaps set of) interface(s) that separate the external system's API/SDK from the NiFi components in order to BYO library so we can try to be future-proof for upcoming versions and such. This is often a lot more difficult and time-consuming than when we used to just accept all contributions that connected to these systems.

Having said that, I don't want to set a negative tone for PR contributions for external system integrations or else everyone will end up with their own private repos and NiFi's bread and butter of universal data distribution will suffer.

MikeThomsen · 2025-06-13T11:33:32Z

@exceptionfactory We should be good to go now on the dependency update.

If you, @joewitt and @mattyb149 could weight in on keeping this as a single bundle vs two (one Cassandra, one ScyllaDB) I'd like to know your thoughts. I see more advantages in having a single, unified CQL bundle, but the disadvantage is that API divergence between the two databases could necessitate breaking compatibility later.

joewitt · 2025-06-13T17:38:49Z

I don't know what CQL is but i'm guessing Cassandra Query Language.

Im not familiar with Cassandra vs ScyllaDB

But I am very familiar with complex dependencies, maintenance, and vulnerability considerations. This one just makes me more attentive than others because it tends to tick those boxes of concern.

To be clear...I do recognize Cassandra is very popular and an excellent tool. And the nifi community is better off when we give them a great integration. I get it. I just dont have a good enough sense of this contrib in doing that.

I also noted the AI generated looking code and thought that was a potential flag. I'm not sure we can use that as is but maybe...

exceptionfactory

Thanks for the updating the driver references @MikeThomsen. I plan to take a closer at the dependency tree, as it appears to include a number of libraries.

As far as the code itself, there are several areas that need to be addressed to align with current conventions including the following:

Remove use of displayName() from Property Descriptors
Use human-readable Title Case names for the name() in Property Descriptors
Define supported Property Descriptors statically using List.of()
Defined Relationships statically using Set.of()
Rewrite log messages to avoid use of due to {} with an Exception, since the Exception is not an argument for the log message itself
Use an enum that implements DescribedValue for defining allowable values and read the enum from the Context
Standardize naming using either CQL or Cql in some class names. It looks like the driver uses CqlSession, although NiFi components use SQL, so using CQL seems better in places
Align Java package names more closely with module names, such as processors.cql and services.scylladb in applicable modules

If you can work through these items across the board, that would address a number of things and enable focus on the implementation itself.

MikeThomsen · 2025-06-14T12:48:02Z

@joewitt

I don't know what CQL is but i'm guessing Cassandra Query Language.

Im not familiar with Cassandra vs ScyllaDB

TL;DR, yes CQL = Cassandra Query Language. ScyllaDB is an increasingly popular, C++ rewrite of Apache Cassandra that aims for (or seems to aim for) 100% driver API compatibility so it is a drop-in replacement as far as Cassandra users are concerned.

I also noted the AI generated looking code and thought that was a potential flag. I'm not sure we can use that as is but maybe...

I understand the concern, but when I reviewed the output from ChatGPT the TypeCodec implementations looked like what any reasonable person would write by hand to implement those mappings so I'm not sure there is enough novelty here to raise red flags. That said, I can try to do some googling on Cassandra to see if OpenAI was taking liberties with someone else's code.

MikeThomsen · 2025-06-14T12:51:54Z

@exceptionfactory

If you can work through these items across the board, that would address a number of things and enable focus on the implementation itself.

Yeah. Understood. Thanks for this. FWIW, the implementation is directly inspired by how the graph bundle generally handles the interaction between its controller services and processors (since Cypher and Gremin are so radically different as protocols)

…d ScyllaDB.

exceptionfactory · 2025-07-25T18:30:00Z

...nifi-cql-services-api/src/main/java/org/apache/nifi/service/cql/api/CQLExecutionService.java

+            .identifiesControllerService(SSLContextService.class)
+            .build();
+
+    PropertyDescriptor CLIENT_AUTH = new PropertyDescriptor.Builder()


This property should not be used for client operations, it is only relevant to services that implement listening sockets.

exceptionfactory · 2025-07-25T18:31:08Z

...nifi-cql-services-api/src/main/java/org/apache/nifi/service/cql/api/CQLExecutionService.java

+import java.util.stream.Stream;
+
+public interface CQLExecutionService extends ControllerService {
+    PropertyDescriptor CONTACT_POINTS = new PropertyDescriptor.Builder()


These Property Descriptors should not be included on the interface. The Controller Service interface defines contract behavior, but the Property Descriptors are specific to a given implementation, and should never be included on a Controller Service interface definition.

MikeThomsen requested review from exceptionfactory and mattyb149 June 10, 2025 20:28

exceptionfactory requested changes Jun 10, 2025

View reviewed changes

joewitt reviewed Jun 10, 2025

View reviewed changes

exceptionfactory requested changes Jun 13, 2025

View reviewed changes

MikeThomsen added 2 commits July 1, 2025 11:59

NIFI-9770 Added nifi-cql-bundle to bring support for Cassandra 4.X an…

822575e

…d ScyllaDB.

NIFI-9770 Updated dependencies and made changes requested in a PR

3dd373a

MikeThomsen force-pushed the NIFI-9770 branch from d48999e to 3dd373a Compare July 1, 2025 17:22

exceptionfactory reviewed Jul 25, 2025

View reviewed changes

NIFI-9770 Added nifi-cql-bundle to bring support for Cassandra 4.X an… #10006

Are you sure you want to change the base?

NIFI-9770 Added nifi-cql-bundle to bring support for Cassandra 4.X an… #10006

Uh oh!

Conversation

MikeThomsen commented Jun 10, 2025

Summary

Tracking

Issue Tracking

Pull Request Tracking

Pull Request Formatting

Verification

Build

Licensing

Documentation

Uh oh!

MikeThomsen commented Jun 10, 2025

Uh oh!

exceptionfactory left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joewitt Jun 10, 2025

Choose a reason for hiding this comment

Uh oh!

MikeThomsen Jun 10, 2025

Choose a reason for hiding this comment

Uh oh!

joewitt Jun 10, 2025

Choose a reason for hiding this comment

Uh oh!

joewitt Jun 10, 2025

Choose a reason for hiding this comment

Uh oh!

mattyb149 Jun 10, 2025

Choose a reason for hiding this comment

Uh oh!

MikeThomsen commented Jun 13, 2025

Uh oh!

joewitt commented Jun 13, 2025

Uh oh!

exceptionfactory left a comment

Choose a reason for hiding this comment

Uh oh!

MikeThomsen commented Jun 14, 2025

Uh oh!

MikeThomsen commented Jun 14, 2025

Uh oh!

exceptionfactory Jul 25, 2025

Choose a reason for hiding this comment

Uh oh!

exceptionfactory Jul 25, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

exceptionfactory left a comment •

edited

Loading