[GGUF] Reduce peak RAM usage by casting dequantized tensors early during load

[UsamaKenway]

Optimizes memory usage when loading GGUF models by performing dtype casting immediately after dequantization.

While I was adding the support for Gemma4 in this PR #45296, i noticed this issue that the GGUF tensors are dequantized to float32 by default during the loading process, even if the user intends to load the model in float16 or bfloat16. For large models, this creates a significant RAM spike that can lead to Out Of Memory.

By passing the target torch_dtype directly into the loading utility, we can cast the tensors immediately after dequantization, effectively halving the peak RAM required for the state dict.

Benchmark Results (Gemma 4 26B IT q4_k_m)

I tested the peak RAM (Global Peak RSS) with and without this change using a separate branch for tracking:

- Without this PR (Float32 spike): ~118.7 GB Peak RSS
- With this PR (Early casting):    ~59.4  GB Peak RSS
------------------------------------------------------
Saving:                            ~59.3  GB (50% reduction)

Tests

With the changes

(py312venv) usamakenway@Megatron: RUN_SLOW=1 pytest tests/quantization/ggml/test_ggml.py::GgufModelTests::test_gemma4_26b_it_q4_k_m -s                                                                                                                                                                                                                                                                                                                                                   

[RAM DEBUG] Global Peak RSS (High Water Mark): 2185.59 MB

[RAM DEBUG] Global Peak RSS (High Water Mark): 2197.88 MB

[RAM DEBUG] Global Peak RSS (High Water Mark): 2391.64 MB
Converting and de-quantizing GGUF tensors...: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 658/658 [03:03<00:00,  3.59it/s]

[RAM DEBUG] Global Peak RSS (High Water Mark): 59428.81 MB
Loading weights: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 657/657 [00:00<00:00, 26841.26it/s]
PASSEDtests/quantization/ggml/test_ggml.py::GgufModelTests::test_gemma4_26b_it_q4_k_m [PASSED] 287.34s

Without the changes

(py312venv) usamakenway@Megatron:  RUN_SLOW=1 pytest tests/quantization/ggml/test_ggml.py::GgufModelTests::test_gemma4_26b_it_q4_k_m -s
                                                                                                                                       
tests/quantization/ggml/test_ggml.py::GgufModelTests::test_gemma4_26b_it_q4_k_m 
[RAM DEBUG] Global Peak RSS (High Water Mark): 2259.14 MB

[RAM DEBUG] Global Peak RSS (High Water Mark): 2270.75 MB

[RAM DEBUG] Global Peak RSS (High Water Mark): 2464.60 MB
Converting and de-quantizing GGUF tensors...: 100%|███████████████████████████████████████████████████████████████████████████████| 658/658 [05:46<00:00,  1.90it/s]

[RAM DEBUG] Global Peak RSS (High Water Mark): 118747.33 MB
Loading weights: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 657/657 [00:03<00:00, 195.35it/s]
PASSEDtests/quantization/ggml/test_ggml.py::GgufModelTests::test_gemma4_26b_it_q4_k_m [PASSED] 499.38s

Code Agent Policy

• I confirm that this is not a pure code agent PR.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[Rocketknight1]

cc @SunMarc

[SunMarc]

Thanks, a couple of comments

[SunMarc]

can we move that above so that we don't have to replicate the dtype logic ?

[SunMarc]

do we really need the fp32 check ?