How to convert Llama2 and Mistral models to TFlite ?

[RageshAntonyHM]

I saw that it is possible to run GPT2 in Android using KerasNLP

https://www.tensorflow.org/lite/examples/auto_complete/overview

I am eager to run Llama 2 or Mistral in the same way

I am currently using MLC-LLM and also interested other ways.

I saw this Colab to convert but it is only in GPT2

https://colab.research.google.com/github/tensorflow/codelabs/blob/main/KerasNLP/io2023_workshop.ipynb#scrollTo=2fux7O64KHQ6 (remove @googleio2023 branch in pip install)

But the KerasNLP also has Mistral and LLama as per the repo https://github.com/keras-team/keras-nlp/tree/master/keras_nlp/models/

But I unable to find how to convert these models to Tflite.

I am asking what is the equivalent for this block in Mistral and Llama.

gpt2_preprocessor = keras_nlp.models.GPT2CausalLMPreprocessor.from_preset(
    "gpt2_base_en",
    sequence_length=256,
    add_end_token=True,
)
gpt2_lm = keras_nlp.models.GPT2CausalLM.from_preset("gpt2_base_en", preprocessor=gpt2_preprocessor)

Please help me

[zhubarb]

I am interested in the same thing. Did you get any luck with Llama2 with tflite? Not sure whether it will be possible though with a 6 GB ram device. Maybe the solution is to go with llama.cpp?

[RageshAntonyHM]

I can't convert llama 2 models to Tflite.

So I am trying MAID for Android

[Rizwanali324]

I am interested in the same thing. Did you get any luck with Llama2 with tflite? Not sure whether it will be possible though with a 6 GB ram device. Maybe the solution is to go with llama.cpp?

I'm also interested to mistral .
what do you think If we use kaggle notebook (15 gb GPU and 30 gb ram )