diff --git a/CHANGELOG.md b/CHANGELOG.md index 53156d1f..9a7e8e05 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,7 @@ - Add a flag to determine if database initialization steps should be executed ([#669]). - Add new roles for dag-processor and triggerer processes ([#679]). +- Added a note on webserver workers to the trouble-shooting section ([#685]). ### Fixed @@ -22,6 +23,7 @@ [#678]: https://github.com/stackabletech/airflow-operator/pull/678 [#679]: https://github.com/stackabletech/airflow-operator/pull/679 [#683]: https://github.com/stackabletech/airflow-operator/pull/683 +[#685]: https://github.com/stackabletech/airflow-operator/pull/685 ## [25.7.0] - 2025-07-23 diff --git a/docs/modules/airflow/pages/troubleshooting/index.adoc b/docs/modules/airflow/pages/troubleshooting/index.adoc index bb8fd12c..aa2ef848 100644 --- a/docs/modules/airflow/pages/troubleshooting/index.adoc +++ b/docs/modules/airflow/pages/troubleshooting/index.adoc @@ -25,3 +25,29 @@ webservers: # Also add to other roles! ---- See e.g. https://github.com/minio/minio/issues/20845[this MinIO issue] for details. + +== Setting API Workers + +In Airflow the webserver (called the API Server in Airflow 3.x+) can use multiple workers. +This is determined by the environment variable `+AIRFLOW__API__WORKERS+` and is set by default to `4` in Airflow 2.x and `1` in Airflow 3.x+. +The reason for this difference is that Airflow uses a backend library to manage child processes and in 3.x+ this library can cause child processes to be killed if a hard-coded startup timeout is exceeded. +For most cases with Airflow 3.x+ a default of `1` should be sufficient, but if you run into performance issues and would like to add more workers, you can either modulate multiple worker processes at the level of webserver, keeping the default of a single worker per webserver: + +[source,yaml] +---- +webservers: + roleGroups: + default: + replicas: 2 # add a replica (with a single worker) +---- + +or change the environment variable using `envOverrides`: + +[source,yaml] +---- +webservers: + envOverrides: + AIRFLOW__API__WORKERS: 2 # something other than the default of 1 +---- + +TIP: Our strong recommendation is to increase the webserver replicas, with each webserver running a single worker, as this removes the risk of running into timeouts or memory issues.