Understanding OSDev for block devices

[bertwesarg]

The docu reads:

Therefore hwloc tries to add software devices (HWLOC_OBJ_OS_DEVICE, also known as OS devices).

So my (individual) block devices attached to an AHCI are "software" devices? This does not make sense to me. For NIC/OPENFABRICS/GPU/COPRO this makes sense and the XMLs are looking what I would expect. Some PCIDev with (possibly) multiple sub OSDevs how this device is exposed by the OS to userspace (opencl0d0, rmsi0, ib0, mlx0_5, ...).

But for block devices this does not fit into the documented and my brain construct.

            <object type="PCIDev" gp_index="969" pci_busid="0000:43:00.0" pci_type="0106 [1022:7901] [1458:1000] 51" pci_link_speed="31.50>
              <object type="OSDev" gp_index="1077" name="sdb" subtype="Disk" osdev_type="0">
                <info name="Size" value="937692504"/>
                <info name="SectorSize" value="512"/>
                <info name="LinuxDeviceID" value="8:16"/>
                <info name="Model" value="INTEL_SSDSC2KG96"/>
                <info name="Revision" value="0121"/>
                <info name="SerialNumber" value="PHYM816002Q0960CGN"/>
              </object>
              <object type="OSDev" gp_index="1078" name="sda" subtype="Disk" osdev_type="0">
                <info name="Size" value="937692504"/>
                <info name="SectorSize" value="512"/>
                <info name="LinuxDeviceID" value="8:0"/>
                <info name="Model" value="INTEL_SSDSC2KG96"/>
                <info name="Revision" value="0121"/>
                <info name="SerialNumber" value="PHYM814402D8960CGN"/>
              </object>
            </object>

For now, I also do not see, that this will change with 3.0, I could only found #540. The XMLs in #399 also look like they fit my mental model, which are 1.11 and 2.1.

Is there something I miss I has there something changed?

Thanks for the feedback.

[bgoglin]

Hello.

Indeed the difference between hardware and software may look vague here. The actual meaning is rather "a software handle to a physical device". "sdb" is the Linux-specific name of that disk, that's why it's considered a software thing. If you had a software-RAID, it would be "md0" (not sure I ever tried that with hwloc). All these are exposed exactly the same by Linux. That's why it's a bit confusing because some are software constructions while some are direct match to the hardware.

Why do nvme disk in #399 look better to you? They are exactly the same as sda and sdb but different technology and Linux names? NVMe may have "namespaces" but nobody uses them anyway.

I'd need to check but I wouldn't be surprised if the block layer gives a way to split a device into two disks, so you'd get two OS Block devices inside a single PCI dev.

There's no plan to change this in 3.0 (at least for now). #540 is only about improving subtypes, names are objects won't change.

One thing that was considered is to change the difference between osdev and pcidev because there is some duplication in some cases (your sda disk). The only solution I'd see is to merge OS and PCI into a single "IO" object type so that "sda" has only one while GPUs have one for the physical device, one for the CUDA handle, etc. However this would require to change the API significantly, I am not sure it's worth the effort both for us and for users.

[bertwesarg]

Thanks, but I think I was not clear in my report. There are only PCIDevs for AHCI controllers and OSDevs for SSD disks. There are no duplicates and a software OSDev for the disks is confusing for me. My expectation based on NIC and GPU would be, that below the PCIDev for the AHCI controller there is an object "DISKDev" (lets call it so to differentiate it from the existing once) for each disk attached to this AHCI controller, and then each DISKDev object has an OSDev as child for the Linux device (sda, sdb). That would keep OSDev as the software device exposed by the OS. (and could potentially extended to also reflect partitions and file systems with new sub objects)

My example was not clear enough, as it misses PCI vendor/device resolution. Here is the topology from my NAS which has 3 spinning disks and one SSD with 1098578:

<object type="Bridge" gp_index="35" id="obj35" bridge_type="0-1" depth="0" bridge_pci="0000:[00-04]">
  <object type="PCIDev" gp_index="31" id="obj31" pci_busid="0000:00:12.0" pci_type="0106 [8086:5ae3] [1849:5ae3] 0b 01" pci_link_speed="0.000000">
    <info name="PCIVendor" value="Intel Corporation"/>
    <info name="PCIDevice" value="Celeron N3350/Pentium N4200/Atom E3900 Series SATA AHCI Controller"/>
    <object type="OSDev" gp_index="37" id="obj37" name="sdb" subtype="Disk" osdev_type="1">
      <info name="Size" value="1953514584KiB"/>
      <info name="SectorSize" value="512"/>
      <info name="LinuxDeviceID" value="8:16"/>
      <info name="Vendor" value="Seagate"/>
      <info name="Model" value="ST2000VN004-2E4164"/>
      <info name="Revision" value="SC60"/>
      <info name="SerialNumber" value="Z52C3F36"/>
    </object>
    <object type="OSDev" gp_index="39" id="obj39" name="sda" subtype="Disk" osdev_type="1">
      <info name="Size" value="3907018584KiB"/>
      <info name="SectorSize" value="512"/>
      <info name="LinuxDeviceID" value="8:0"/>
      <info name="Vendor" value="Seagate"/>
      <info name="Model" value="ST4000VN006-3CW104"/>
      <info name="Revision" value="SC60"/>
      <info name="SerialNumber" value="WW65EEKF"/>
    </object>
  </object>
  <object type="Bridge" gp_index="27" id="obj27" bridge_type="1-1" depth="1" bridge_pci="0000:[03-03]" pci_busid="0000:00:13.2" pci_type="0604 [8086:5ada] [1849:5ada] fb 00" pci_link_speed="0.615385">
    <info name="PCIVendor" value="Intel Corporation"/>
    <info name="PCIDevice" value="Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A #3"/>
    <object type="PCIDev" gp_index="25" id="obj25" pci_busid="0000:03:00.0" pci_type="0106 [1b21:0612] [1849:0612] 02 01" pci_link_speed="0.615385">
      <info name="PCIVendor" value="ASMedia Technology Inc."/>
      <info name="PCIDevice" value="ASM1062 Serial ATA Controller"/>
      <object type="OSDev" gp_index="36" id="obj36" name="sdd" subtype="Disk" osdev_type="1">
        <info name="Size" value="1953514584KiB"/>
        <info name="SectorSize" value="512"/>
        <info name="LinuxDeviceID" value="8:48"/>
        <info name="Vendor" value="Seagate"/>
        <info name="Model" value="ST2000VN004-2E4164"/>
        <info name="Revision" value="SC60"/>
        <info name="SerialNumber" value="Z52BZ8QS"/>
      </object>
      <object type="OSDev" gp_index="38" id="obj38" name="sdc" subtype="Disk" osdev_type="1">
        <info name="Size" value="125034840KiB"/>
        <info name="SectorSize" value="512"/>
        <info name="LinuxDeviceID" value="8:32"/>
        <info name="Model" value="SSD_128GB"/>
        <info name="Revision" value="S1022A0"/>
        <info name="SerialNumber" value="204JPDB128G0000144"/>
      </object>
    </object>
  </object>
</object>

Having Vendor, Model, and SerialNumber info objects on an OSDev should already indicative enough, that its not a software device by the OS.

[bgoglin]

OK I see your point. I am not sure we want yet another object between PCI and OS. I would just be a split of the OSDev into hardware-stuff and software-stuff. Same information but more difficult to retrieve from users/apps.
Maybe we just need to rename/change what OSDevs can be in 3.0. It could be "IO Device" instead.
PCI Device could be IO Devices too, they'd just a PCI "subtype" and a "BDF=0000:02:03.4" as an attribute.
Bridges are already generic, PCI or not, nothing to change there.

However we still need a good way to expose multiple software handles (OpenCL, CUDA, NVML, ...) inside a single physical devices. Either through multiple sub-objects or with just many names+attributes but I don't think that's convenient.
If we keep a dedicated object type for pure software handles (or IODev with a software flag), I wonder if we'll be able to clearly define what's software or not. But does it matter? If we report a "Block" device or an "OpenCL" device, I'd say it's enough for users without an explicit software flag.

[bertwesarg]

I consider the current model for NIC and GPU very natural, so I would propose not to change anything there.

I'm just hallucinating here, but maybe something like

• rename PCIDev to IODev
• introduce an IODev subtype named iodev_type
  • probably remove the pci_ prefix from all the object attributes for PCIDev
• use pci subtype for the current PCIDev
• add sata as a new iodev_type which will be used for SATA devices
  • keep the OSDev for sda and co but spit the properties between hardware device and OSDev
  • optionally add a new OSDev for partitions and use this as subobjects in disk objects

I have not yet saw USB devices in the topology, but that is potential additional iodev_type.

I also saw, that for NVME devices the info objects are all on the OSDev, and not split between the PCIDev and the OSDev

        <object type="PCIDev" gp_index="71" pci_busid="0000:01:00.0" pci_type="0108 [15b7:5006] [15b7:5006] 00" pci_link_speed="3.938462">
          <info name="PCIVendor" value="Sandisk Corp"/>
          <info name="PCIDevice" value="SanDisk Extreme Pro / WD Black SN750 / PC SN730 / Red SN700 NVMe SSD"/>
          <object type="OSDev" gp_index="86" name="nvme0n1" subtype="Disk" osdev_type="0">
            <info name="Size" value="1000204632"/>
            <info name="SectorSize" value="512"/>
            <info name="LinuxDeviceID" value="259:0"/>
            <info name="Vendor" value="Western Digital"/>
            <info name="Model" value="WDC PC SN730 SDBQNTY-1T00-1001"/>
            <info name="Revision" value="11170101"/>
            <info name="SerialNumber" value="21080X802671"/>
          </object>
        </object>

should probably something like

        <object type="PCIDev" gp_index="71" pci_busid="0000:01:00.0" pci_type="0108 [15b7:5006] [15b7:5006] 00" pci_link_speed="3.938462">
          <info name="PCIVendor" value="Sandisk Corp"/>
          <info name="PCIDevice" value="SanDisk Extreme Pro / WD Black SN750 / PC SN730 / Red SN700 NVMe SSD"/>
          <info name="Vendor" value="Western Digital"/>
          <info name="Model" value="WDC PC SN730 SDBQNTY-1T00-1001"/>
          <info name="Revision" value="11170101"/>
          <info name="SerialNumber" value="21080X802671"/>
          <info name="Size" value="1000204632"/>
          <object type="OSDev" gp_index="86" name="nvme0n1" subtype="Disk" osdev_type="0">
            <info name="SectorSize" value="512"/>
            <info name="LinuxDeviceID" value="259:0"/>
          </object>
        </object>

[bgoglin]

Adding SATA means also adding SAS, NVMe, PMEM, etc. Lots of technologies that may change in the future and would require API changes whenever a new one is added. That's why (string) info attributes are better for this kind of things. PCI is different because it's not going to change anything soon. USB isn't implemented because hwloc is for locality and performance, and locality doesn't matter for USB as far as I know. Same reason why hwloc doesn't report an exhaustive hardware inventory. People who care about specific things like USB or Disk have other tools to do so (lsusb, lsblk).

Regarding SATA, NVMe, etc. All these are block devices, there's no need to change the existing Block subtype.

Regarding your last proposal, SectorSize is a hardware thing, but sometimes the OS puts stronger requirements (like 4kB pages instead of 512B sectors). It's hard to know where this attribute should go. Also having both PCIVendor/PCIDevice and Vendor/Device in the same object would be confusing. That's why I believe replacing OSDev with IODev and keeping all osdev attributes in the IODev would be more clear. Attributes starting with "Linux" are obviously software things. The name should be documented as coming from the OS.