Is JMX auth and SSL supported?

[cin]

I'm not seeing where you can inject credentials for JMX or truststore/keystore information in the sidecar. Is this supported?

[smiklosovic]

Hi @cin , yes, it is supported, it is available under these flags:

$ java -jar cassandra-sidecar-2.0.0-alpha4.jar cassandra-sidecar help

...
      --jmx-service=[ADDRESS][:PORT]|[JMX SERVICE URL]
                            Address (and optional port) of a Cassandra instance to connect to via JMX. ADDRESS may be a
                              hostname, IPv4 dotted or decimal address, or IPv6 address. When ADDRESS is omitted, the loopback
                              address is used. PORT, when specified, must be a valid port number. The default port 7199 will be
                              substituted if omitted.Defaults to ':7199'
      --jmx-user=[STRING]   User for JMX for Cassandra
      --jmx-password=[STRING]
                            Password for JMX for Cassandra
      --jmx-truststore=[PATH]
                            Path to truststore file for Cassandra
      --jmx-truststore-password=[PATH]

[cin]

Sorry, I should have been more specific. I meant through the service.

{
    "type": "backup",
    "storageLocation": "s3://xxx",
    "snapshotTag": "hello",
    "entities": "system_auth",
    "k8sNamespace": "xxx",
    "k8sSecretName": "xxx",
    "dataDirectory": "/var/lib/cassandra",
    "globalRequest": true
}

[smiklosovic]

@cin no, it works like you start a sidecar with these credentials and after that requests are just sent to it, you can not specify
credentials for jmx per rest call.

The reasoning behind that is that you would basically send e.g. a password for JMX in a plain text, plus I am not sure how would one go about sending a truststore via rest too ... or I am missing something here :) Would you mind to explain me your idea and workflow?

[cin]

That actually makes sense. The only downside of that is that you have to restart the service to update credentials if/when they change. I think we can live with that though. For our use case, we're running in kubernetes and can probably just restart the container while leaving C* running (as opposed to restarting the pod). The only way I could think to work around this would be to watch a secret in kubernetes for the credentials, but you guys are designing this to work outside of kubernetes too. So that solution's probably too specific to kubernetes to work.

Thanks for clearing this up! I'll try out passing the credentials at service startup. I think this can be closed as a non-issue. Thanks again!

[smiklosovic]

Hi @cin , this is a very interesting point you are making here.

We do have a solution for Cassandra truststores / keystores when you are starting a Cassandra node in Kubernetes. Check out this section of our Kubernetes operator wiki (1). In a nutshell, you create a Kubernetes secret and upon Cassandra container startup, it is automatically set up via operators' configuration mechanism.

We are also covered when credentials are necessary for CQL connection to such cluster from Sidecar. Check these docs (2).

So, your request to secure JMX connections would be modelled similarly as (2), there would be a secret which would be resolved dynamically upon each JMX call. We already do this for SSL for CQL - if you read (2) until the very end, there is detailed explanation as how this is done on Java side which integrates with Sidecar seamlessly.

I think your request makes sense and we could resolve creds for cases you run in Kubernetes from secrets as it was done for CQL - by doing so, you might change your creds and it would all continue to work as they would be resolved every single time.

Do you want this to see implemented?

(1) https://github.com/instaclustr/cassandra-operator/wiki/Accounts,-login-and-SSL#ssl
(2) https://github.com/instaclustr/cassandra-operator/wiki/Accounts,-login-and-SSL#ssl-with-sidecar

[cin]

I'd be in favor of this feature for sure! It will also keep the credentials out of ps and /proc//cmdline. Although one can argue if someone has access to exec into your pods, you're already dead in the water. They could even decode your secrets at that point. I've just been trying to keep creds out of commands/history.

[smiklosovic]

Thats fair enough, but as you said, not having secrets hardcoded anywhere in container is a good idea, these secrets were invented for some reason right ... Exec into a container, on a security level, equals to somebody gaining the access to your Kubernetes cluster as such (as one has to have kubectl active etc) and having this under control is just a well defined problem easier to handle, I suppose.