What models can vLLM actually serve? it fails with GGUF, fails with safetensors

[vmajor]

As per the title,

what models/formats can vLLM actually serve? I am no longer going to iterate through documentation and have it fail, so someone please let me know what vLLM wants.

safetensors error:

ERROR 03-19 09:12:19 [engine.py:448] KeyError: 'layers.26.mlp.down_proj.weight.absmax'
Process SpawnProcess-1:
Traceback (most recent call last):

GGUF error:

ERROR 03-19 07:18:59 [engine.py:160] AttributeError: '_OpNamespace' '_C' object has no attribute 'ggml_dequantize'
ERROR 03-19 07:19:16 [client.py:305] RuntimeError('Engine process (pid 289002) died.')
ERROR 03-19 07:19:16 [client.py:305] NoneType: None
^CINFO 03-19 07:21:16 [launcher.py:74] Shutting down FastAPI HTTP server.
INFO:     Shutting down

[jeejeelee]

Could you plz provide your env details?
For safetensors error: seems to be that you are using BNB, you should append --quantization bitsandbytes --load-format bitsandbytes when starting up

[vmajor]

this worked well --quantization bitsandbytes --load-format bitsandbytes

vllm seems to be running.