Skip to content

Conversation

@yeahdongcn
Copy link
Contributor

While testing with torch_musa from PyTorch 2.2, I encountered an issue when loading weights to the GPU:

  File "/ws/./ktransformers/util/custom_gguf.py", line 674, in dequantize_q6_k_gpu
    return KTransformersOps.dequantize_q6_k(c_pointer, data.size, block_size, ele_per_blk, device, target_dtype)
TypeError: dequantize_q6_k(): incompatible function arguments. The following argument types are supported:
    1. (data: int, num_bytes: int, blk_size: int, ele_per_blk: int, device: torch.device, target_dtype: torch._C._te.ScalarType) -> torch.Tensor

This PR ensures backward compatibility with Torch 2.2 and fixes a mismatched parameter type in the header file.

Testing Done

  • make dev_install
  • Verified weight loading to CPU/GPU using python ./ktransformers/local_chat.py ...

@Atream
Copy link
Contributor

Atream commented Feb 24, 2025

Merged. Thank you for your contribution!

@Atream Atream merged commit 6f9ea68 into kvcache-ai:main Feb 24, 2025
6 checks passed
Luosuu pushed a commit to Luosuu/ktransformers that referenced this pull request Apr 15, 2025
Ensure backward compatibility with PyTorch 2.2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants