failed to load policy: upstream_connection

[contd]

Recently we have been seeing a lot of warnings in the logs about configuration.lua:74: build_policy_chain(): failed to load policy: upstream_connection version: builtin err: module 'init' not found but not sure what to do about it but it's also flooding our log collection and might also be part of a bigger issue we are having with APIcast. Here are some example full messages we are seeing:

2025/11/15 14:11:31 [warn] 97#97: 2025/11/15 14:11:32 [warn] 185#185: *2 [lua] configuration.lua:74: build_policy_chain(): failed to load policy: upstream_connection version: buil │
│ tin err: module 'init' not found:                                                                                                                                                   │
│     no file '/opt/app-root/src/src/apicast/policy/upstream/upstream_connection/builtin/init.lua'                                                                                    │
│     no file '/opt/app-root/src/src/apicast/policy/upstream_connection/init.lua', context: ngx.timer                                                                                 │
│                                                                                                                                                                                     │
│ 2025/11/15 14:11:31 [warn] 97#97: [lua] configuration.lua:74: build_policy_chain(): failed to load policy: upstream_connection version: builtin err: module 'init' not found:       │
│     no file '/opt/app-root/src/src/apicast/policy/upstream/upstream_connection/builtin/init.lua'                                                                                    │
│     no file '/opt/app-root/src/src/apicast/policy/upstream_connection/init.lua'

I did exec into the POD and find that the path of the init.lua is actually at '/opt/app-root/src/src/apicast/policy/upstream' so I set the env variable:

APICAST_POLICY_LOAD_PATH: '/opt/app-root/src/src/apicast/policy/upstream'

But it didn't seem to help. We have another namespace that runs our regression version of our microservices with APIcast and that one doesn't seem to pull from our 3scale API management site like it used to and is no longer working. The only thing we changes was moving everything over to a new EKS cluster and AWS. The weirder thing is the same image for APIcast is being used for a 3rd namespace (our UAT version of our microservices with APIcast) and it works fine with none of those warnings. The settings on the 3scale API management site are essentially the same butdifferent domains (hence differently defines ingresses for each namespace). This didn't start happening until we moved our development and regression namespaces (environments) to the UAT cluster which already had the UAT environment working fine.

Any help would be greatly appreciated and I have a feeling the issue is not with any configuration on the 3scale API management site since that hasn't changed and well over 2 years especially for these environments.

[tkan145]

Hi there, what version of 3scale are you using and how did you deploy APIcast? It sounds like your image is broken, upstream and upstream connection are two different policies. Do you have any customization?

We have another namespace that runs our regression version of our microservices with APIcast and that one doesn't seem to pull from our 3scale API management site like it used to and is no longer working.

Did you also update APIcast to point to the new domain? Additionally, there is some additional work to do if you move the setup to another cluster with a different domain.