help request: Secrets gcp integration error

[klinux]

Description

I'm trying to configure GCP secrets integration, but I'm receiving this error:

apisix-7948456797-vdrtj apisix 2025/11/28 04:07:59 [error] 49#49: *17968 lua entry thread aborted: runtime error: unknown reason
apisix-7948456797-vdrtj apisix stack traceback:
apisix-7948456797-vdrtj apisix coroutine 0:
apisix-7948456797-vdrtj apisix  [C]: in function 'error'
apisix-7948456797-vdrtj apisix  /usr/local/apisix//deps/share/lua/5.1/resty/jwt.lua:572: in function 'sign'
apisix-7948456797-vdrtj apisix  /usr/local/apisix/apisix/utils/google-cloud-oauth.lua:92: in function 'generate_jwt_token'
apisix-7948456797-vdrtj apisix  /usr/local/apisix/apisix/utils/google-cloud-oauth.lua:54: in function 'refresh_access_token'
apisix-7948456797-vdrtj apisix  /usr/local/apisix/apisix/utils/google-cloud-oauth.lua:41: in function 'generate_access_token'
apisix-7948456797-vdrtj apisix  /usr/local/apisix/apisix/secret/gcp.lua:109: in function 'get_secret'
apisix-7948456797-vdrtj apisix  /usr/local/apisix/apisix/secret/gcp.lua:161: in function 'make_request_to_gcp'
apisix-7948456797-vdrtj apisix  /usr/local/apisix/apisix/secret/gcp.lua:184: in function 'get'
apisix-7948456797-vdrtj apisix  /usr/local/apisix/apisix/secret.lua:156: in function 'fetch_by_uri'
apisix-7948456797-vdrtj apisix  /usr/local/apisix/apisix/secret.lua:178: in function 'fetch'
apisix-7948456797-vdrtj apisix  /usr/local/apisix/apisix/secret.lua:214: in function 'fetch_secrets'
apisix-7948456797-vdrtj apisix  /usr/local/apisix/apisix/consumer.lua:241: in function 'create_obj_fun'
apisix-7948456797-vdrtj apisix  /usr/local/apisix/apisix/core/lrucache.lua:111: in function 'consumer_lrucache'
apisix-7948456797-vdrtj apisix  /usr/local/apisix/apisix/consumer.lua:250: in function 'create_obj_fun'
apisix-7948456797-vdrtj apisix  /usr/local/apisix/apisix/core/lrucache.lua:111: in function 'lrucache'
apisix-7948456797-vdrtj apisix  /usr/local/apisix/apisix/consumer.lua:262: in function 'consumers_kv'
apisix-7948456797-vdrtj apisix  /usr/local/apisix/apisix/consumer.lua:276: in function 'find_consumer'
apisix-7948456797-vdrtj apisix  /usr/local/apisix/apisix/plugins/key-auth.lua:83: in function 'find_consumer'
apisix-7948456797-vdrtj apisix  /usr/local/apisix/apisix/plugins/key-auth.lua:104: in function 'phase_func'
apisix-7948456797-vdrtj apisix  /usr/local/apisix/apisix/plugin.lua:1194: in function 'run_plugin'
apisix-7948456797-vdrtj apisix  /usr/local/apisix/apisix/init.lua:788: in function 'http_access_phase'

Environment

• APISIX version (run apisix version): 3.14
• Operating system (run uname -a): oficial image 3.14
• OpenResty / Nginx version (run openresty -V or nginx -V):
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info):
• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):

[Baoyuantop]

Hi @klinux, I followed the instructions in this document to verify the settings and did not see the problem you mentioned. Could you please share your routes and the relevant configurations for secret and consumer?

[klinux]

Hi @Baoyuantop thank you for the reply.

I followed the instructions here https://apisix.apache.org/docs/apisix/terminology/secret/, but I got the same error following this this document that you point.

I'm using the ingress controller to configure my auth-key, here how I configured.

Consumer

apiVersion: apisix.apache.org/v1alpha1
kind: Consumer
metadata:
  name: backoffice-consumer
  namespace: default
spec:
  gatewayRef:
    name: apisix
    namespace: apisix
  credentials:
    - type: key-auth
      name: backoffice-api-key
      config:
        key: $secret://gcp/1/backoffice-api-key

Plugin configure

apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
  name: api-key-validation
  namespace: default
spec:
  plugins:
    - name: key-auth
      config:
        key: backoffice-api-key

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: backoffice-route
  namespace: default
spec:
  parentRefs:
    - name: apisix
      namespace: apisix
      sectionName: https
  hostnames:
    - api.**********
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /svc/backoffice
      filters:
        - type: ExtensionRef
          extensionRef:
            group: apisix.apache.org
            kind: PluginConfig
            name: remove-context
        - type: ExtensionRef
          extensionRef:
            group: apisix.apache.org
            kind: PluginConfig
            name: api-key-validation
      backendRefs:
        - name: backoffice
          port: 80

Obs: If I set the key of consumer as a string, it works.

Here the secret config, curl "http://127.0.0.1:9180/apisix/admin/secrets/gcp/1" -H "X-API-KEY: xxx"

{
    "value": {
        "create_time": 1764299776,
        "ssl_verify": false,
        "auth_config": {
            "project_id": "my-project",
            "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n",
            "client_email": "apisix-client@my-project.iam.gserviceaccount.com"
        },
        "id": "gcp/1",
        "update_time": 1764324135
    },
    "modifiedIndex": 572,
    "createdIndex": 484,
    "key": "/apisix/secrets/gcp/1"
}

The error persists:

apisix-57b5c69889-qcz9d apisix 2025/11/28 10:27:38 [error] 49#49: *954 lua entry thread aborted: runtime error: unknown reason
apisix-57b5c69889-qcz9d apisix stack traceback:
apisix-57b5c69889-qcz9d apisix coroutine 0:
apisix-57b5c69889-qcz9d apisix  [C]: in function 'error'
apisix-57b5c69889-qcz9d apisix  /usr/local/apisix//deps/share/lua/5.1/resty/jwt.lua:572: in function 'sign'
apisix-57b5c69889-qcz9d apisix  /usr/local/apisix/apisix/utils/google-cloud-oauth.lua:92: in function 'generate_jwt_token'
apisix-57b5c69889-qcz9d apisix  /usr/local/apisix/apisix/utils/google-cloud-oauth.lua:54: in function 'refresh_access_token'
apisix-57b5c69889-qcz9d apisix  /usr/local/apisix/apisix/utils/google-cloud-oauth.lua:41: in function 'generate_access_token'
apisix-57b5c69889-qcz9d apisix  /usr/local/apisix/apisix/secret/gcp.lua:109: in function 'get_secret'
apisix-57b5c69889-qcz9d apisix  /usr/local/apisix/apisix/secret/gcp.lua:161: in function 'make_request_to_gcp'
apisix-57b5c69889-qcz9d apisix  /usr/local/apisix/apisix/secret/gcp.lua:184: in function 'get'
apisix-57b5c69889-qcz9d apisix  /usr/local/apisix/apisix/secret.lua:156: in function 'fetch_by_uri'
apisix-57b5c69889-qcz9d apisix  /usr/local/apisix/apisix/secret.lua:178: in function 'fetch'
apisix-57b5c69889-qcz9d apisix  /usr/local/apisix/apisix/secret.lua:214: in function 'fetch_secrets'
apisix-57b5c69889-qcz9d apisix  /usr/local/apisix/apisix/consumer.lua:241: in function 'create_obj_fun'
apisix-57b5c69889-qcz9d apisix  /usr/local/apisix/apisix/core/lrucache.lua:111: in function 'consumer_lrucache'
apisix-57b5c69889-qcz9d apisix  /usr/local/apisix/apisix/consumer.lua:250: in function 'create_obj_fun'
apisix-57b5c69889-qcz9d apisix  /usr/local/apisix/apisix/core/lrucache.lua:111: in function 'lrucache'
apisix-57b5c69889-qcz9d apisix  /usr/local/apisix/apisix/consumer.lua:262: in function 'consumers_kv'
apisix-57b5c69889-qcz9d apisix  /usr/local/apisix/apisix/consumer.lua:276: in function 'find_consumer'
apisix-57b5c69889-qcz9d apisix  /usr/local/apisix/apisix/plugins/key-auth.lua:83: in function 'find_consumer'
apisix-57b5c69889-qcz9d apisix  /usr/local/apisix/apisix/plugins/key-auth.lua:104: in function 'phase_func'
apisix-57b5c69889-qcz9d apisix  /usr/local/apisix/apisix/plugin.lua:1194: in function 'run_plugin'
apisix-57b5c69889-qcz9d apisix  /usr/local/apisix/apisix/init.lua:788: in function 'http_access_phase'
apisix-57b5c69889-qcz9d apisix  access_by_lua(nginx.conf:366):2: in main chunk, client: 172.69.11.135, server: _, request: "GET /svc/backoffice/?status=IN_ANALYSIS HTTP/2.0", host: "api.xxxxxx.xxxxxxxxx"

[juzhiyuan]

@Baoyuantop is it because #12749 (reply in thread)?

(cc @kayx23 @bzp2010 )