Should Superset recommend building your own image and if so, document how to do that?

[sfirke]

Starting in 4.1.0 release last month, Superset's main pre-built docker image (lean) does not have the necessary drivers for using either a Postgres or MySQL backend. This caused problems when upgrading for people relying on the project's pre-built images (here, here, here).

In light of this, I wonder if we should be suggesting that most people build their own Superset image to extend lean. I avoided building my own image at first, preferring to run pre-built images from Docker Hub, but now I see it's not that hard as long as we document it. My inspiration here is the Airflow docs, which show very clearly how and why to extend and customize the Airflow image. I used this yesterday and it was a breeze.

@ilsaloving shared a Dockerfile that they developed in response to this change in 4.1.0, it would make a good starting point.

This would also address a concern that @mistercrunch raised in the past about people using the -dev images: they run as root which is not appropriate for production. We can show people how to avoid that, as in ilsaloving's Dockerfile above. And it would address the perennial questions about how to properly install a headless browser and driver.

Thoughts from the community on whether this is a direction to go in and what to include?

[mistercrunch]

Would be nice to have 1 opinionated "production recommended setup" maybe. The issue is that there are so many potential variations that it's really hard to cater to everyone. If people care about having an image that's not bloated (and maintainable), you don't want the 50+ database drivers and the pain that comes to managing that. Say if you use postgres as your backend, and only BigQuery as your warehouse, you probably don't want all the other stuff blotting your docker image. Slows down builds/ci/autoscaling, ...

For us as maintainers it's hard to provide something that works for everyone. Installing all database drivers and keeping them up to date on arm/amd requires a PHD in pip to install if at all possible.

Some common variation:

• metadata database drivers (mysql/postgres mostly, but there are deploys out there on SQL server and Oracle ) :(
• analytics database drivers, some database offer multiple driver options (looking at you mysql/mariadb!, native/non-native, binaries, ...)
• want to run alerts/reports, which headless browser you want to use? chrom/firefox/msft? which version? oh that one has security issues now...
• building for arm or amd?
• statsd?
• redis/rabbitmq?
• ...

One idea would be to have a fat version to complement the lean one, with everything you might need, but personally really don't want to maintain that.

Best solution is probably to instruct people to build upon lean, pick your version of lean in YOUR Dockerfile, and uv pip install what you need on top, with some guidance around common options. Your Dockerfile can be lean + a few things you need

I'm not against lean+ or opinionated images, but not sure who's going to maintain those.

As far as the ASF is concerned, the only "ASF-stamped-and-guaranteed" are the tarballs hosted on SVN. I know it's not convenient for anyone, but it's hard to distribute anything beyond our code base as who knows what's in our hundreds of dependencies and what CVEs may exist in that sub-sub-sub-package at this moment or in 2 months... The docker images we provide today are, per the ASF/maintainers "reference implementations" and "use at your own risk".

[sfirke]

Best solution is probably to instruct people to build upon lean, pick your version of lean in YOUR Dockerfile, and uv pip install what you need on top, with some guidance around common options. Your Dockerfile can be lean + a few things you need

I think we are in agreement that such instruction is the way to go.

I want to mimic that Airflow doc I linked above and basically say "since we can't provide an image that meets everyone's specific needs" - for the reasons you gave - "here's how to build the one with the drivers you need".

[mistercrunch]

Since helm is the official recommend way for production, I'd extend what we have there to support providing your own Dockerfile, which would inherit from lastest docker image on dockerhub (which is a pointer to the latest release). The example/default Dockerfile provided would probably install pip install psycopg2-binary and an example driver or two, maybe the top 2-3 analytics database's driver (say bigquery/snowflake/redshift). Maybe redis and a headless driver too.