[bug]: performance regression compiling jsonnet since 0.34.0-rc.1?

[jperville]

Description of the bug

I have been happily using Kapitan for a while.
I noticed much slower compile times with the recent 0.34 stable releases vs 0.34.0-rc.1.
I use go-jsonnet but the performance is as bad as if python-jsonnet was used.

As a workaround, I am staying on 0.34-rc.1 for the moment.

Steps to reproduce

I have a work kapitan project (which I cannot share publicly, but can share privately).

1. using kapitan 0.34.0-rc1: `kapitan compile --verbose -J vendor .`
2. using kapitan 0.34.4: `kapitan compile --verbose -J vendor .`

Compare the compile time at the end of the verbose log.

What happened?

I expected to have similar compile times between 0.34.0-rc.1 and 0.34.4

Additional context

When trying to compile my work kapitan project (7 targets, compiling 6 in parallel).

The compile times are:

• 1.4 seconds with kapitan 0.34.0-rc.1
• 6.7 seconds with kapitan 0.34.4

I published logs of both compilations as gist here:
https://gist.github.com/jperville/7ea40d8dde3b11e9b81f72fc9aad8aca

Here is the content of my .kapitan file:

version: 0.34

compile:
  indent: 2
  parallelism: 6
  cache: true
  fetch: true
  use-go-jsonnet: true
  targets:
  - aks-dev
  - gke-dev
  - eks-dev
  - k3s-dev
  - kind-dev
  - kubespray-dev
  - minikube-dev

Version (kapitan)

0.34.4

Version (kapitan)

I use kapitan directly (using docker: kapicorp/kapitan:0.34.4)

Python virtual environment

• pyenv
• venv
• no environment

OS information

Ubuntu 22.04

Relevant log output

[jperville]

Even better, on the small project, using go-jsonnet seems slower than using python-jsonnet (kapitan 0.34.4).
In the logs below, the first compile uses python-jsonnet and the second go-jsonnet.

root@4b4c76a59f1a:/yyy# kapitan --version
0.34.4
root@4b4c76a59f1a:/yyy# time kapitan compile -J vendor .
Rendered inventory (0.15s): discovered 7 targets.
Compiling 7/7 targets using 6 concurrent processes: (12 CPU detected)
Dependency https://charts.jetstack.io/charts/cert-manager-v1.12.7.tgz: extracted to vendor/charts
Compiled aks-dev (0.29s)
Compiled gke-dev (0.28s)
Compiled kind-dev (0.33s)
Compiled eks-dev (0.40s)
Compiled minikube-dev (1.34s)
Compiled kubespray-dev (1.69s)
Compiled k3s-dev (1.96s)
Compiled 7 targets in 2.05s

real 0m5.098s
user 0m13.929s
sys 0m1.301s
root@4b4c76a59f1a:/yyy# time kapitan compile -J vendor . --use-go-jsonnet
Rendered inventory (0.16s): discovered 7 targets.
Compiling 7/7 targets using 6 concurrent processes: (12 CPU detected)
Dependency https://charts.jetstack.io/charts/cert-manager-v1.12.7.tgz: extracted to vendor/charts
Compiled gke-dev (0.64s)
Compiled aks-dev (0.74s)
Compiled kind-dev (0.75s)
Compiled eks-dev (1.15s)
Compiled minikube-dev (3.70s)
Compiled kubespray-dev (6.47s)
Compiled k3s-dev (6.50s)
Compiled 7 targets in 6.57s

real 0m10.119s
user 0m19.308s
sys 0m11.978s

The same compilation using kapitan 0.34.0-rc.1

root@8d5fc7b46f1d:/yyy# kapitan --version
0.34.0-rc.1
root@8d5fc7b46f1d:/yyy#
root@8d5fc7b46f1d:/yyy# time kapitan compile -J vendor .
Rendered inventory (0.18s): discovered 7 targets.
Compiling 7/7 targets using 6 concurrent processes: (12 CPU detected)
Dependency https://charts.jetstack.io/charts/cert-manager-v1.12.7.tgz: extracted to vendor/charts
Compiled kind-dev (0.24s)
Compiled gke-dev (0.26s)
Compiled aks-dev (0.32s)
Compiled eks-dev (0.46s)
Compiled minikube-dev (1.16s)
Compiled kubespray-dev (1.55s)
Compiled k3s-dev (1.75s)
Compiled 7 targets in 1.83s

real 0m4.637s
user 0m11.857s
sys 0m1.191s
root@8d5fc7b46f1d:/yyy# time kapitan compile -J vendor . --use-go-jsonnet
Rendered inventory (0.17s): discovered 7 targets.
Compiling 7/7 targets using 6 concurrent processes: (12 CPU detected)
Dependency https://charts.jetstack.io/charts/cert-manager-v1.12.7.tgz: extracted to vendor/charts
Compiled aks-dev (0.20s)
Compiled gke-dev (0.21s)
Compiled kind-dev (0.21s)
Compiled eks-dev (0.22s)
Compiled k3s-dev (0.97s)
Compiled kubespray-dev (1.04s)
Compiled minikube-dev (0.87s)
Compiled 7 targets in 1.15s

real 0m3.900s
user 0m9.299s
sys 0m1.063s