[Bug]: Qwen2-Audio-7B-support

[freedom-cui]

⚙️ Your current environment

We encountered numerous peculiar issues during the QWen2-Audio-7B conversion process. Do we currently support Qwen2-Audio-7B? If so, could you provide a demo?

Thank you very much！

🐛 Describe the bug

Qwen2-Audio-7B GPTQ W8A8 Quantization

🛠️ Steps to reproduce

No response

[HDCharles]

I don't think its been specifically tested. What do you mean by conversion process? Do you mean applying GPTQ W8A8 Quantization

GPTQ has to trace the model which may not work for certain architectures, does RTN work? What about weight only quantization?

looking at the architecture, it has a normal structure at least in the decoder, if you try to just quantize q_proj, k_proj, v_proj does that work?

[kylesayrs]

#1082

LLM Compressor is capable of supporting Qwen2-Audio! I had a quick start on adding an example script, but other projects took priority. If you or anyone else would like to take a look at reviving the qwen audio example, we can start supporting the model!

[freedom-cui]

I don't think its been specifically tested. What do you mean by conversion process? Do you mean applying GPTQ W8A8 Quantization

GPTQ has to trace the model which may not work for certain architectures, does RTN work? What about weight only quantization?

looking at the architecture, it has a normal structure at least in the decoder, if you try to just quantize q_proj, k_proj, v_proj does that work?
Hello, @HDCharles thank you very much for your reply. I believe RTN is an excellent idea until the relevant audio PR is formally merged. If I want to perform RTN on the decode part of the audio model, is there a reference demo available?

[freedom-cui]

#1082

LLM Compressor is capable of supporting Qwen2-Audio! I had a quick start on adding an example script, but other projects took priority. If you or anyone else would like to take a look at reviving the qwen audio example, we can start supporting the model!

Hello @kylesayrs
Thank you very much for your outstanding contribution. I just reviewed your relevant scripts. Actually, I'm new to llm-compress, and I have a question about the code here. Specifically, in the following location in your code:

https://github.com/vllm-project/llm-compressor/blob/kylesayrs/qwen2_audio/examples/multimodal_audio/qwen2_audio_example.py#L73-L74

The GPTQModifier in llm-compress doesn't seem to be a true GPTQ? If I want to quantize to W8A8, does that mean I need to use it with smoothquant?

Looking forward to your response. Thank you again for your outstanding work.