Make libvirt assign BDFs to PCI devices that are in sync with CHV #1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit introduces infrastructure for automatically assigning PCI BDF to libvirt devices. We assign a fixed BDF to all devices to guarantee that a guest can find devices on the same bus address after live migration and other use cases. Signed-off-by: Pascal Scholz <[email protected]> On-behalf-of: SAP [email protected]
We need a way to store PCI BDFs for all buses of a domain. Signed-off-by: Pascal Scholz <[email protected]> On-behalf-of: SAP [email protected]
We need to mark BDFs for all devices as reserved whenever we create a domain. Signed-off-by: Pascal Scholz <[email protected]> On-behalf-of: SAP [email protected]
When creating device JSON for transmission, we need to use the BDFs assigned by internal libvirt logic to stay in sync. Signed-off-by: Pascal Scholz <[email protected]> On-behalf-of: SAP [email protected]
If we hotplug a device, we need to assign an static PCI BDF to it, so we can ensure that changes to the PCI device tree are reflected properly in libvirt. Same for detaching devices. Signed-off-by: Pascal Scholz <[email protected]> On-behalf-of: SAP [email protected]
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Configurable BDF Design
Draft, because first the CHV side must b merged!
Background
tl;dr: Same motivation as in cyberus-technology/cloud-hypervisor#32
Configurable PCI BDF (Bus, Device, Function) allows you to choose the address a guest sees for a PCI device. Qemu, for example, implements the
addroption for thedeviceargument. Theaddrargument allows defining the device and function parts of a device's BDF in the formaddr=DD.F, whereDdenotes a device ID in the range [0..32] andFa function ID in the range [0..7]. Qemu denotes the bus number with the optionbusnr, where a valid expression isbusnr=pci.2to denote the second PCI bus, for example. [0]An example of a BDF for Bus=1, Device=3, and Function=5 is
1:03.5. [1]Current status
Currently, the CHV driver generates BDFs on the fly during hotplug of net devices. This is unfortunate, as the PCI slot IDs depend on the order in which network devices are hotplugged. Disk devices are currently not considered for BDF generation. This leaves not only the possibility that device BDF may change when restarting/migrating VMs, but also fails to account for synchronization of BDFs between CHV and libvirt.
Proposed changes
We utilize the given libvirt infrastructure to assign a BDF to a device whenever we add one. Moreover, with the right patches merged to CHV, we now transmit the correct BDFs when communicating with CHV. This solves both problems mentioned before: First, BDFs are in sync between CHV and libvirt by similarly issuing BDF whenever necessary. Second, once the BDFs are assigned by libvirt, both sides respect them when initializing devices, creating save addresses for use cases such as live migration.
References
[0] https://qemu-project.gitlab.io/qemu/system/device-emulation.html#device-buses
[1] https://wiki.xenproject.org/wiki/Bus:Device.Function_(BDF)_Notation