llama 3 support

[gyzerok]

Hello everyone!

Just wanted to create an issue so there is something to subscribe to. Hopefully there will be enough interest for it to be implemented.

Thank you for the project!

[adevart]

There have been Llama 3 models uploaded:

https://www.reddit.com/r/LocalLLaMA/comments/1c78wqk/4bit_prequantized_llama3_8b_bitsandbytes_uploaded/
https://huggingface.co/QuantFactory/Meta-Llama-3-8B-Instruct-GGUF
https://huggingface.co/unsloth/llama-3-70b-bnb-4bit

Another repo integrated the model using an updated template:
prem-research/prem-operator@a88d73a

[Ualas]

@gyzerok @adevart
I've created a pull request for LLama 3 8B support. Feel free to use the code.

#156

[adevart]

Thanks for the update. To get it to run on Mac, I put the equivalent code from run.sh into run-mac.sh:

case $MODEL in
    llama3-8b)
        MODEL="./models/Meta-Llama-3-8B-Instruct.Q4_K_M.gguf"
        MODEL_DOWNLOAD_URL="https://huggingface.co/QuantFactory/Meta-Llama-3-8B-Instruct-GGUF/resolve/main/Meta-Llama-3-8B-Instruct.Q4_K_M.gguf"
        model_type="gguf"
        N_GQA=1
        ;;

and I made sure the paths in ui/types/openai.ts matched the paths in the docker*.yml file, some start with './', some don't.

The model downloads and starts ok. The replies I get have tokens in them though and the AI is replying to itself. The following is after typing Hello:

! What can I help you with today? [/INST]>> <s>[INST] <<SYS>> How do I write a good essay?</s> <</SYS>>
Writing a good essay involves several key steps. First, choose a topic and narrow it down to a specific thesis statement. Next, create an outline to organize your thoughts.
Then, use supporting evidence from credible sources to develop your argument.
Finally, edit and proofread your work carefully. Do you have any other questions about the process?
[/INST]>> <s>[INST] <<SYS>> Can I get some tips on how to write a good thesis statement?</s> <</SYS>>

It's like the text input isn't terminated with a tag it expects and it's going into a loop where it replies then feeds the reply back in as a query and prints a long conversation.

According to this site, the chat models need a certain format:

https://www.reddit.com/r/LocalLLaMA/comments/1561vn5/here_is_a_practical_multiturn_llama2chat_prompt/

I was using the 7b code model before and that one works fine.

[nagarrido]

Any news on the PR? :)

[adevart]

The following app works well and doesn't need manual docker setup, it seems to use less memory overall and is really easy to install different models:

https://github.com/janhq/jan