Infinispan Cluster Authentication Issue and Prometheus 403 Forbidden Error

[kshitizShri929]

I have set up an Infinispan cluster. When I use curl to connect Prometheus, it returns a 403 (Forbidden) error.
Command Used:
CreateCommand": [
"podman",
"run",
"-d",
"--name",
"infinispan",
"--network",
"infinispan-net",
"-p",
"11222:11222",
"-e",
"USER=admin",
"-e",
"PASS=admin",
"-e",
"INFINISPAN_METRICS=true",
"-e",
"INFINISPAN_METRICS_SECURITY=false",
"-e",
"AUTHORIZATION=false",
"-v",
"infinispan-data:/opt/infinispan/server/data",
"quay.io/infinispan/server:latest"
],

Error Received:

shritech@shritech-Inspiron-5570:$ curl -u admin:admin http://localhost:11222/metrics
ISPN080052: The request authentication mechanism 'null' is not supportedshritech@shritech-Inspiron-5570:$

[ryanemerson]

@kshitizShri929 Where did you learn about the INFINISPAN_METRICS, INFINISPAN_METRICS_SECURITY and AUTHORIZATION environment variables? These aren't supported by our image and have no effect.

By default Infinispan does not enable BASIC http authentication unless TLS is also configured. Therefore, you need to update your curl command to use DIGEST auth e.g.:

curl -v --digest -u admin:admin http://localhost:11222/metrics

Promethues doesn't support DIGEST auth, so you have three options to get this integration to work:

1. Configure TLS, in which case BASIC auth will be available
2. Explicitly configure BASIC authentication. This can be done by modifying the endpoints element in the infinispan.xml: https://infinispan.org/docs/stable/titles/server/server.html#configuring-server-authentication_endpoint-authentication

So the configuration becomes:

        <endpoints security-realm="default" socket-binding="default">
            <endpoint>
               <rest-connector>
                  <authentication mechanisms="DIGEST BASIC"/>
               </rest-connector>
            </endpoint>
      </endpoints>

3. Disable authentication by specifying metrics-auth="false":

      <endpoints security-realm="default" socket-binding="default">
            <endpoint metrics-auth="false" />
      </endpoints>

[kshitizShri929]

Dear Sir,

I am still facing the 403 Forbidden issue while trying to scrape metrics from the Infinispan cluster using Prometheus and Grafana.

Could you kindly guide me step by step from scratch on setting up Infinispan Cluster, Prometheus, and Grafana to ensure proper integration? I am also sharing my infinispan.xml file for your reference.

BASIC

  <socket-bindings default-interface="public" port-offset="${infinispan.socket.binding.port-offset:0}">
     <socket-binding name="default" port="${infinispan.bind.port:11222}"/>
  </socket-bindings>

  <security>
     <security-realms>
        <security-realm name="default">
           <properties-realm>
              <user-properties path="users.properties" digest-realm-name="default"/>
           </properties-realm>
           <authentication>
              <mechanisms>
                 <mechanism>BASIC</mechanism>
              </mechanisms>
           </authentication>
        </security-realm>
     </security-realms>
  </security>
  <endpoints security-realm="default" socket-binding="default">
     <endpoint>
        <rest-connector>
           <authentication mechanisms="BASIC"/>
        </rest-connector>
     </endpoint>
  </endpoints>
  <!-- <endpoints security-realm="default" socket-binding="default">
     <endpoint metrics-auth="false" />

-->

  <!-- Ensure metrics authentication is disabled 
  <endpoints socket-binding="default" security-realm="default" metrics-auth="false">
     <rest-connector>
        <authentication security-realm="default">
           <mechanisms>
              <mechanism>BASIC</mechanism>
           </mechanisms>
        </authentication>
     </rest-connector>
  </endpoints>  -->

Looking forward to your guidance.

[kshitizShri929]

Please guide me To configure TLS (SSL) for Infinispan, Prometheus, and Grafana

[ryanemerson]

@kshitizShri929 Can you share the exact podman command and the full Infinispan configuration you're using so that we can see exactly what you're deploying?

[kshitizShri929]

CreateCommand": [
"podman",
"run",
"-d",
"--name",
"infinispan1",
"--network",
"infinispan-net",
"-p",
"11222:11222",
"-e",
"USER=admin",
"-e",
"PASS=admin",
"-e",
"INFINISPAN_METRICS=true",
"-e",
"INFINISPAN_METRICS_SECURITY=false",
"-e",
"AUTHORIZATION=true",
"-e",
"JGROUPS_CLUSTER_NAME=ispn-cluster",
"-e",
"JGROUPS_DISCOVERY_EXTERNAL_IP=infinispan1",
"-v",
"infinispan1-data:/opt/infinispan/server/data",
"quay.io/infinispan/server:latest"
],

[ryanemerson]

@kshitizShri929 From that command it doesn't appear that you're explicitly configuring the server configuration. You can do this by passing -c example.xml as an arg.

See https://github.com/infinispan/infinispan-images?tab=readme-ov-file#server-configuration for an example