Reduce size of runtime-adapter image (exclude Python/tensorflow to convert keras models)

[GolanLevy]

The current image weight is very high (2.14Gb) which slows down the predictor's uptime.

Correct me if I'm wrong please, but the only reason the adapter needs to install tensorflow is to convert keras models to tensorflow models, which sounds weird to do it on runtime and not in advance, see

modelmesh-runtime-adapter/model-mesh-triton-adapter/server/utils.go

Lines 63 to 64 in f9781d2

	func convertKerasToTF(kerasFile string, targetPath string, ctx context.Context, loggr logr.Logger) error {
	cmd := exec.Command("python", "/opt/scripts/tf_pb.py", kerasFile, targetPath)

modelmesh-runtime-adapter/Dockerfile

Line 145 in f9781d2

# install python to convert keras to tf

modelmesh-runtime-adapter/Dockerfile

Line 164 in f9781d2

pip install tensorflow

modelmesh-runtime-adapter/Dockerfile

Line 172 in f9781d2

COPY --from=build /opt/app/model-mesh-triton-adapter/scripts/tf_pb.py /opt/scripts/

If we remove this option, we can remove the tensorflow installation, and since python is needed only for that, removing the entire python installation.
This reduces the image size from 2.14 GB to 256Mb.

Can we just remove it? If not, can we have two images, the original one and a new slim one?

[ckadner]

This is a bit tricky.

We don't want to drop support for Keras models. Requiring users to convert possibly hundreds/thousands of Keras models to Tensorflow prior to deploying them may not be practical.

We could possibly have two images as you suggested: a smaller one without the conversion script and a large one with it. We would need to introduce a install/deployment option in the modelmesh-serving repo.

Users who decide to use the slim image would then be required to do the Keras to TF conversion prior to deploying an ISVC.