Question about the overall design

[chengchen666]

Hi, I've gone through the source code and the whole TensorFusion project, this is a wonderful project. In terms of the vgpu.rs project itself, I have several questions:

1. It looks like, limiter only ask for permission from hypervisor, and hypervisor only grant permission to limiter, after limiter get the permission, who will actually do the execution? I don't think it's limiter because limiter is the client process, it can only access local GPUs, but you guys did GPU remoting. So I wonder if it's hypervisor do the execution on behalf of client process, or there's other components, which are close-sourced, but do the actual execution on GPU (like launchKernel)?
2. if hypervisor and pod are on same machine, the overhead from benchmark repo shows around 10%, am I understanding the data correctly?
3. Have you guys noticed the project: https://github.com/SJTU-IPADS/PhoenixOS-Remoting/tree/main. This GPU remoting project achieve very good performance in both local and remote, and they've done many optimizations. Hope it helps to your own product.

Thanks for you reply in advance.

[Code2Life]

@chengchen666 Thanks for your comments !

1. vgpu.rs is actually for resource management and multi-process scheduling on single GPU server, while GPU remoting is another project that not open sourced. we open sourced vgpu.rs limiter/hypervisor because it could be used standalone and have similar features to HAMi project
2. The benchmark consists of different modes, for local GPU mode (with fractional GPU / multi-process scheduling features), the overhead is less than 1% in most cases.
3. Yep, we looked into PhoenixOS project, it's an awesome project that also implements CUDA context hot-migration, API remoting, etc. The projects is owned by university rather than the industry, thus the maturity needs to be improved for production-grade usage.

[chengchen666]

Hi, If I understanding correctly, tensorFusion needs vgpu.rs to actually achieve GPU pool, migration, etc, all advanced features in GPUs. So like, tensorFusion is the component that vgpu.rs used to integrate into k8s system. But I wonder does vgpu.rs itself have capability to integrate with k8s system directly? because I saw vgpu.rs has K8S folder, and the code looks like something to integrate into k8s.
I'm asking because I wonder if I want to build some NPU backends, what part of code can I reuse or leverage, can I get some advices?

[0x5457]

Regarding your query about TensorFusion and vgpu.rs integration with Kubernetes: You're correct that TensorFusion leverages vgpu.rs to enable advanced GPU features like pooling and migration. The k8s directory in vgpu.rs contains code that uses kubeclient to interact with Kubernetes, specifically to read GPU allocation details (e.g., GPUUUID, Limit, TFLOPS, memory) from pod annotations. In the TensorFusion architecture, the operator is responsible for assigning GPUs to each pod, and these assignments are reflected in the pod annotations, which vgpu.rs then processes.

[0x5457]

The vgpu.rs itself is a stand-alone component and cannot be used directly without the operator.