Add headers-more module like dynamic module in "Ingress-Nginx Controller"

[jvalderrama-magnolia]

Environment:

Controller Version:1.9.0
Helm Chart Version:4.8.0
k8S Version: EKS 1.27.4

Description:

When you include the header-more-nginx-module into k8s Ingress-Nginx Controller image modifiying it using a Dockerfile:

ARG INGRESS_NGINX_CONTROLLER_VERSION
FROM registry.k8s.io/ingress-nginx/controller:${INGRESS_NGINX_CONTROLLER_VERSION} as build

ARG INGRESS_NGINX_CONTROLLER_VERSION
ENV INGRESS_NGINX_CONTROLLER_VERSION=${INGRESS_NGINX_CONTROLLER_VERSION}

USER root
RUN apk add \
        automake \
        ca-certificates \
        curl \
        gcc \
        g++ \
        make \
        pcre-dev \
        zlib-dev

RUN NGINX_VERSION=$(nginx -V 2>&1 |sed -n -e 's/nginx version: //p' |cut -d'/' -f2); \
    curl -L "http://nginx.org/download/nginx-${NGINX_VERSION}.tar.gz" | tar -C /tmp/nginx --strip-components=1 -xz

WORKDIR /src/headers-more-nginx-module
RUN curl -L https://github.com/openresty/headers-more-nginx-module/archive/refs/tags/v0.36.tar.gz | tar --strip-components=1 -xz

WORKDIR /tmp/nginx
RUN ./configure --with-compat --add-dynamic-module=/src/headers-more-nginx-module && \
    make modules

FROM registry.k8s.io/ingress-nginx/controller:${INGRESS_NGINX_CONTROLLER_VERSION}

COPY --from=build /tmp/nginx/objs/ngx_http_headers_more_filter_module.so /etc/nginx/modules/

Later build, push and deploy this new image using the helm chart and the modified image like this:

resource "helm_release" "ingress" {
  name         = local.ingress_nginx_release_name
  repository   = "https://kubernetes.github.io/ingress-nginx"
  chart        = "ingress-nginx"
  version      = "4.8.0"
  max_history  = 1
  namespace    = kubernetes_namespace.nginx-ingress.id
  force_update = true

  set {
    name = "controller.image.registry"
    value = "reponame"
  }
  set {
    name = "controller.image.image"
    value = "ingress-nginx/controller"
  }
  set {
    name = "controller.image.tag"
    value = "v1.9.0-headers-more"
  }

  set {
    name = "controller.image.digest"
    value = "sha256:b4976bf5d1ec156bd73281e9f8efcd4643b21ce991cf461808a5430128084hpo"
  }

  set {
    name = "controller.config.main-snippet"
    value = <<EOF
load_module modules/ngx_http_headers_more_filter_module.so;
EOF
  }
...

Expected
Pods running with nginx and headers-more-nginx-module

Error
Once deployed the new image it raises:

Error: exit status 1
2023/12/21 12:15:28 [emerg] 38#38: dlopen() "/usr/local/nginx/modules/ngx_http_headers_more_filter_module.so" failed (Error loading shared library /usr/local/nginx/modules/ngx_http_headers_more_filter_module.so: No such file or directory) in /tmp/nginx/nginx-cfg2885873991:15
nginx: [emerg] dlopen() "/usr/local/nginx/modules/ngx_http_headers_more_filter_module.so" failed (Error loading shared library /usr/local/nginx/modules/ngx_http_headers_more_filter_module.so: No such file or directory) in /tmp/nginx/nginx-cfg2885873991:15
nginx: configuration file /tmp/nginx/nginx-cfg2885873991 test failed