Skip to content

VMs with more than 254 vCPUs #956

Open

Description

@iximeow

Going up to 254 vCPUs is straightforward enough, and with a recent Helios change we're free to bump Propolis' MAXCPU up to 254, but going beyond that will involve some work.

  • There's the (easier) control plane side, where the Propolis API currently wants a u8 of vCPUs. That won't do in the bold manymore core future. this is mostly addressed in configurable CPU topology #1237
  • There's the (maybe easy, maybe hard) side, where we'll want guests to be able to direct interrupts to high-number CPUs if they're so inclined.

The obvious and somewhat daunting solution would be for Propolis to grow an IOMMU which can be used to let guests remap interrupts towards high-number cores like you'd do on real hardware.

https://david.woodhou.se/ExtDestId.pdf describes an alternative that might be workable: Hyper-V, KVM, FreeBSD byhve, and Xen all support a hypervisor feature to expand APIC IDs as used in MSI, but guest OS support is more mixed. Reportedly Linux supports this, Windows would be in a weird state either way, and the FreeBSD issue linked there is still open. I assume it would be mildly better for everyone to help get OSes in a good state with this than to build an IOMMU for Propolis, when the need comes..

Additionally, on Turin (and later? time will tell!) there's support for IOMMU virtualization. That is, the MMIO EFR register has vIommuSup set, as well as GTSup. that suggests maybe we can just .. offer a virtualized IOMMU and be done with it? given the IOMMU spec is 313 pages and I'm not too directly familiar with the AMD IOMMU I'd have questions for other knowers before being too sure, but this seems like a promising option for us. presumably this would mean Propolis' device emulation would have to be aware of any address translation the emulated devices ought to be doing? maybe there's something clever we can do about that.

Activity

  1. iximeow commented on May 18, 2026

    @iximeow
    MemberAuthor

    as I'm going through the ACPI table work in #999, the Processor Local APIC Structure definitions make clear that there is some work to do on the ACPI side too for many-processor VMs. probably?

    the Local APIC structure has a one-byte field for APIC ID and for ACPI Processor UID. these clearly won't do when the number overflows. additionally, Local APIC NMI Structure has a one-byte ACPI Processor UID field as well. in both cases, ACPI points out that the Processor declaration operator is deprecated, that there's a compatibility note in the x2APIC forms of the declarations, and to read more there.

    which tells me that in addition to however we're targeting interrupts (ExtDestId or an emulated iommu, as testing guides us) we'll want x2apic tested and working and to generate x2APIC-style processor definitions here.

  2. dancrossnyc commented on May 21, 2026

    @dancrossnyc
    Contributor

    Oh yes. It's painfully tedious, but we'll have to plumb the x2 versions of data for this stuff into the guests to get them to use more than 255 VCPUs. E.g., instead of (or in addition to) using the "Processor Local APIC Structure" of ACPI 6.6 sec 5.2.12.2, we'll have to supply the "Processor Local x2APIC Structure" of sec 5.2.12.12.

    I highly recommend using the enlightments Woodhouse mentioned in the links above; doing a virtualized IOMMU is going to be neither fun nor easy, even just for interrupt remapping, and all the guests we need to worry about should already support them.

  3. iximeow commented on Jun 5, 2026

    @iximeow
    MemberAuthor

    whoops, another place where we don't handle larger vCPU counts: struct Board uses a u8. starting such a large VM will fail on the Nexus side because the u16 we've got for the vCPU count doesn't fit in a u8.

  4. iximeow commented on Oct 30, 2026

    @iximeow
    MemberAuthor

    #1237 is one piece of this that's here in the Propolis repo. this chases out all cases of "cpu is a u8" that I know of

  5. morlandi7 commented on Nov 6, 2026

    @morlandi7

    Pile of illumos issues above now complete upstream in illumos but need to be added to oxide's version. Next up:

    https://www.illumos.org/issues/16721 x2APIC mode should persist across INIT in bhyve
    https://www.illumos.org/issues/16722 allow x2APIC mode changes in bhyve
    https://www.illumos.org/issues/16723 bhyve x2APIC should disallow improper MSR accesses

    Expecting a few more after these + propolis work

  6. luqmana commented on Nov 19, 2026

    @luqmana
    Contributor

    https://www.illumos.org/issues/18681 bhyve: Add CPUID_BHYVE_FEATURES leaf
    https://www.illumos.org/issues/18688 bhyve: Support and advertise 15-bit MSI Extended Destination ID

  7. luqmana commented on Nov 19, 2026

    @luqmana
    Contributor

    Not quite ready yet to launch a VM with >255 vCPUs but confirmed that with x2APIC + the 15-bit Extended MSI Destination ID support, guests can properly address vCPUs with APIC IDs above 254.

    To test that I extended propolis-standalone to allow explicitly assigning vCPU IDs sparsely (#1263 on a host with illumos#18688 + VM_MAXCPU bumped to 4096):

    [main]
    name = "ubuntu"
    bootrom = "OVMF_CODE.fd"
    cpus = 4
    vcpu_ids = [0, 1, 256, 257]
    memory = 4096
    cpuid_profile = "turin_v2_x2apic"
    
    [main.hv_interface]
    type = "bhyve"
    ext_msi_dest_id = true
    
    # ... rest of config

    OVMF upon discovering a CPU with APIC ID >254 switches to x2APIC mode:

    Force x2APIC mode!
    CpuMpPei: 5-Level Paging = 0
    APIC MODE is 2
    MpInitLib: Find 4 processors in system.
    GetMicrocodePatchInfoFromHob: Microcode patch cache HOB is not found.
    CpuMpPei: 5-Level Paging = 0
    CPU[0000]: Microcode revision = 00000000, expected = 00000000
    CPU[0002]: Microcode revision = 00000000, expected = 00000000
    Register PPI Notify: 8F9D4825-797D-48FC-8471-845025792EF6
    Does not find any stored CPU BIST information from PPI!
      APICID - 0x00000000, BIST - 0x00000000
      APICID - 0x00000001, BIST - 0x00000000
      APICID - 0x00000100, BIST - 0x00000000
      APICID - 0x00000101, BIST - 0x00000000
    

    Tracing wrmsr to IA32_APIC_BASE reveals the same:

    4070843415 vm=fffffc68cb898580 vcpu=256 msr    wr 01b APICBASE    <- fee00c00
     APICBASE: base=fee00000 bsp=0 extd=1 en=1 (x2APIC)
    4070849787 vm=fffffc68cb898580 vcpu=256 mode xAPIC -> x2APIC (guest WRMSR)
    
    4070857702 vm=fffffc68cb898580 vcpu=1 msr    wr 01b APICBASE    <- fee00c00
     APICBASE: base=fee00000 bsp=0 extd=1 en=1 (x2APIC)
    4070864745 vm=fffffc68cb898580 vcpu=1 mode xAPIC -> x2APIC (guest WRMSR)
    
    4070885835 vm=fffffc68cb898580 vcpu=257 msr    wr 01b APICBASE    <- fee00c00
     APICBASE: base=fee00000 bsp=0 extd=1 en=1 (x2APIC)
    4070893078 vm=fffffc68cb898580 vcpu=257 mode xAPIC -> x2APIC (guest WRMSR)
    
    4070964462 vm=fffffc68cb898580 vcpu=0 msr    wr 01b APICBASE    <- fee00d00
     APICBASE: base=fee00000 bsp=1 extd=1 en=1 (x2APIC)
    4070971606 vm=fffffc68cb898580 vcpu=0 mode xAPIC -> x2APIC (guest WRMSR)
    

    Reads of the ID register for each return as expected:

    4070865827 vm=fffffc68cb898580 vcpu=256 x2apic rd 802 ID          -> 100
    4070883791 vm=fffffc68cb898580 vcpu=1 x2apic rd 802 ID          -> 1
    4070909499 vm=fffffc68cb898580 vcpu=257 x2apic rd 802 ID          -> 101
    4071107831 vm=fffffc68cb898580 vcpu=0 x2apic rd 802 ID          -> 0
    

    We have our INITs on and SIPI:

    4072164482 vm=fffffc68cb898580 vcpu=0 x2apic wr 830 ICR_LO      <- c4500
     ICR: vector=0 dest=0 (physical) mode=INIT shorthand=all-but-self level=assert trigger=edge
    4072175693 vm=fffffc68cb898580 vcpu=256 lapic  INIT   APICBASE fee00c00 (x2APIC) -> fee00c00 (x2APIC)
    4072184700 vm=fffffc68cb898580 vcpu=1 lapic  INIT   APICBASE fee00c00 (x2APIC) -> fee00c00 (x2APIC)
    4072190591 vm=fffffc68cb898580 vcpu=257 lapic  INIT   APICBASE fee00c00 (x2APIC) -> fee00c00 (x2APIC)
    

    We get to Linux with x2APIC enabled but with stock Ubuntu 24.04, it's running Linux kernel 6.8.0 which is too old to recognize bhyve's ext-msi-dest-id feature bit and switches back to apic:

    ubuntu@ubuntu:~$ lsb_release -a
    No LSB modules are available.
    Distributor ID: Ubuntu
    Description:    Ubuntu 24.04 LTS
    Release:        24.04
    Codename:       noble
    ubuntu@ubuntu:~$ uname -a
    Linux ubuntu 6.8.0-31-generic #31-Ubuntu SMP PREEMPT_DYNAMIC Sat Apr 20 00:40:06 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
    
    ubuntu@ubuntu:~$ sudo dmesg | grep -i -e smpboot -e apic
    [    0.000000] APIC: Static calls initialized
    [    0.000341] x2apic: enabled by BIOS, switching to x2apic ops
    [    0.004755] ACPI: APIC 0x00000000BFB7A000 0000A6 (v01 OVMF   OVMFEDK2 20130221 RVAT 01000000)
    [    0.004762] ACPI: Reserving APIC table memory at [mem 0xbfb7a000-0xbfb7a0a5]
    [    0.004838] APIC: Switched APIC routing to: cluster x2apic
    [    0.051703] ACPI: X2APIC_NMI (uid[0xffffffff] dfl dfl lint[0x1])
    [    0.051737] IOAPIC[0]: apic_id 2, version 17, address 0xfec00000, GSI 0-31
    [    0.051767] smpboot: Allowing 4 CPUs, 0 hotplug CPUs
    [    0.203866] APIC: Switch to symmetric I/O mode setup
    [    0.204675] x2apic: IRQ remapping doesn't support X2APIC mode
    [    0.205315] x2apic disabled
    [    0.210172]  native_apic_msr_read+0x1b/0x60
    [    0.210594]  apic_read_boot_cpu_id+0x69/0x90
    [    0.211030]  apic_set_fixmap+0x75/0x90
    [    0.211403]  enable_IR_x2apic+0x258/0x2b0
    [    0.211794]  x86_64_probe_apic+0xf/0x70
    [    0.212170]  apic_intr_mode_init+0x8f/0x130
    [    0.215765] APIC: Switched APIC routing to: flat
    [    0.217978] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
    [    0.364090] smpboot: CPU0: AMD Oxide Virtual Turin-like Processor (family: 0x1a, model: 0x2, stepping: 0x1)
    [    0.375325] smpboot: native_kick_ap: bad cpu 2
    [    0.375853] smpboot: native_kick_ap: bad cpu 2
    [    0.377403] smpboot: x86: Booting SMP configuration:
    [    0.379423] smpboot: native_kick_ap: bad cpu 3
    [    0.382851] smpboot: native_kick_ap: bad cpu 3
    [    0.384654] smpboot: Max logical packages: 2
    [    0.384838] smpboot: Total of 2 processors activated (10779.41 BogoMIPS)
    [    0.456838] ACPI: Using IOAPIC for interrupt routing
    

    We can at least see the state transitions work properly:

    6347498337 vm=fffffc68cb898580 vcpu=0 msr    wr 01b APICBASE    <- fee00100
     APICBASE: base=fee00000 bsp=1 extd=0 en=0 (disabled)
    6347507053 vm=fffffc68cb898580 vcpu=0 mode x2APIC -> disabled (guest WRMSR)
    6347514918 vm=fffffc68cb898580 vcpu=0 msr    wr 01b APICBASE    <- fee00900
     APICBASE: base=fee00000 bsp=1 extd=0 en=1 (xAPIC)
    6347516992 vm=fffffc68cb898580 vcpu=0 mode disabled -> xAPIC (guest WRMSR)
    

    Install one of the HWE packages like linux-image-virtual-hwe-24.04 though and we get a 7.0.0 kernel and now we're cooking:

    ubuntu@ubuntu:~$ grep BHYVE /boot/config-$(uname -r)
    CONFIG_BHYVE_GUEST=y
    ubuntu@ubuntu:~$ sudo dmesg | grep -i -e smpboot -e apic -e hypervisor
    [    0.000000] APIC: Static calls initialized
    [    0.000000] Hypervisor detected: Bhyve
    [    0.000213] x2apic: enabled by BIOS, switching to x2apic ops
    [    0.004995] ACPI: APIC 0x00000000BFB7A000 0000A6 (v01 OVMF   OVMFEDK2 20130221 RVAT 01000000)
    [    0.005001] ACPI: Reserving APIC table memory at [mem 0xbfb7a000-0xbfb7a0a5]
    [    0.005082] APIC: Switched APIC routing to: cluster x2apic
    [    0.005920] ACPI: X2APIC_NMI (uid[0xffffffff] dfl dfl lint[0x1])
    [    0.005953] IOAPIC[0]: apic_id 2, version 17, address 0xfec00000, GSI 0-31
    [    0.132017] APIC: Switch to symmetric I/O mode setup
    [    0.133259] APIC: Switched APIC routing to: physical x2apic
    [    0.135471] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
    [    0.291607] smpboot: CPU0: AMD Oxide Virtual Turin-like Processor (family: 0x1a, model: 0x2, stepping: 0x1)
    [    0.299129] smpboot: x86: Booting SMP configuration:
    [    0.309298] smpboot: Total of 4 processors activated (21560.32 BogoMIPS)
    [    0.349295] ACPI: Using IOAPIC for interrupt routing
    

    Our sparse vCPU ID allocation is reflected in the guest:

    ubuntu@ubuntu:~$ cat /proc/cpuinfo | grep -Ew -e processor -e apicid -e id
    processor       : 0
    physical id     : 0
    core id         : 0
    apicid          : 0
    initial apicid  : 0
    processor       : 1
    physical id     : 0
    core id         : 0
    apicid          : 1
    initial apicid  : 1
    processor       : 2
    physical id     : 0
    core id         : 128
    apicid          : 256
    initial apicid  : 256
    processor       : 3
    physical id     : 0
    core id         : 128
    apicid          : 257
    initial apicid  : 257
    

    And the latter 2 vCPUs are successfully getting device interrupts demonstrating the 15-bit MSI Destination ID working:

    ubuntu@ubuntu:~$ grep -e CPU -e nvme -e virtio /proc/interrupts
               CPU0       CPU1       CPU2       CPU3       
     32:          0          0          0          0 PCI-MSIX-0000:00:10.0   0-edge      virtio1-config
     33:          0          0          0         65 PCI-MSIX-0000:00:10.0   1-edge      virtio1-req.0
     34:          0          0          0          0 PCI-MSIX-0000:00:05.0   0-edge      virtio0-config
     35:       7546          0          0          0 PCI-MSIX-0000:00:05.0   1-edge      virtio0-input.0
     36:         71          1          0          0 PCI-MSIX-0000:00:05.0   2-edge      virtio0-output.0
     37:          0       8258          1          0 PCI-MSIX-0000:00:05.0   3-edge      virtio0-input.1
     38:          0         37          0          1 PCI-MSIX-0000:00:05.0   4-edge      virtio0-output.1
     39:          1          0       5444          0 PCI-MSIX-0000:00:05.0   5-edge      virtio0-input.2
     40:          0          1         66          0 PCI-MSIX-0000:00:05.0   6-edge      virtio0-output.2
     41:          0          0          1       6776 PCI-MSIX-0000:00:05.0   7-edge      virtio0-input.3
     42:          0          0          0         84 PCI-MSIX-0000:00:05.0   8-edge      virtio0-output.3
     43:          0          0          0          0 PCI-MSIX-0000:00:05.0   9-edge      virtio0-input.4
     44:          0          0          0          0 PCI-MSIX-0000:00:05.0  10-edge      virtio0-output.4
     45:          0          0          0          0 PCI-MSIX-0000:00:05.0  11-edge      virtio0-input.5
     46:          0          0          0          0 PCI-MSIX-0000:00:05.0  12-edge      virtio0-output.5
     47:          0          0          0          0 PCI-MSIX-0000:00:05.0  13-edge      virtio0-input.6
     48:          0          0          0          0 PCI-MSIX-0000:00:05.0  14-edge      virtio0-output.6
     49:          0          0          0          0 PCI-MSIX-0000:00:05.0  15-edge      virtio0-input.7
     50:          0          0          0          0 PCI-MSIX-0000:00:05.0  16-edge      virtio0-output.7
     51:          0          0          0          0 PCI-MSIX-0000:00:05.0  17-edge      virtio0-input.8
     52:          0          0          0          0 PCI-MSIX-0000:00:05.0  18-edge      virtio0-output.8
     53:          0          0          0          0 PCI-MSIX-0000:00:05.0  19-edge      virtio0-input.9
     54:          0          0          0          0 PCI-MSIX-0000:00:05.0  20-edge      virtio0-output.9
     55:          0          0          0          0 PCI-MSIX-0000:00:05.0  21-edge      virtio0-input.10
     56:          0          0          0          0 PCI-MSIX-0000:00:05.0  22-edge      virtio0-output.10
     57:          0         21          0          0 PCI-MSIX-0000:00:06.0   0-edge      nvme0q0
     58:          0          0       1268          0 PCI-MSIX-0000:00:06.0   1-edge      nvme0q1
     59:          0          0          0        990 PCI-MSIX-0000:00:06.0   2-edge      nvme0q2
     60:       1454          0          0          0 PCI-MSIX-0000:00:06.0   3-edge      nvme0q3
     61:          0       2092          0          0 PCI-MSIX-0000:00:06.0   4-edge      nvme0q4
    

    We can also confirm the IOAPIC path works by pinning legacy interrupts for the serial port to the vCPUs 256/257 (CPU2/3 in the guest):

    ubuntu@ubuntu:~$ echo 2-3 | sudo tee /proc/irq/$(grep ttyS0 /proc/interrupts | awk '{sub(/:$/, "", $1); print $1}')/smp_affinity_list
    2-3
    ubuntu@ubuntu:~$ grep -e CPU -e ttyS0 /proc/interrupts
               CPU0       CPU1       CPU2       CPU3       
      4:        725          0        104          0  IO-APIC   4-edge      ttyS0
    

    Tracing the bhyve lapic/ioapic emulation further confirms the guest is making use of the extended destination ID bits for vCPUs above 255:

    === MSI (lapic_intr_msi) ===
      base    ext     dest   vec    count
         0    0x1      256    32        1
         1    0x1      257    35        2
         1    0x1      257    40        3
         1      0        1    41        6
         1      0        1    33       13
         0      0        0    41       32
         1      0        1    43       57
         0    0x1      256    34      541
         1    0x1      257    41      683
         0      0        0    33      757
         1      0        1    42      931
         0      0        0    32     9231
    
    === IOAPIC (vioapic_send_intr) ===
     pin   base    ext     dest   vec masked    count
       4      0    0x1      256    35      1        2
       4      0    0x1      256    35      0     2941
    
    === vlapic_deliver_intr ===
    caller                           dest  phys  delmode   vec    count
    vmm`lapic_intr_msi                256     1        0    32        1
    vmm`lapic_intr_msi                257     1        0    35        2
    vmm`lapic_intr_msi                257     1        0    40        3
    vmm`lapic_intr_msi                  1     1        0    41        6
    vmm`lapic_intr_msi                  1     1        0    33       13
    vmm`lapic_intr_msi                  0     1        0    41       32
    vmm`lapic_intr_msi                  1     1        0    43       57
    vmm`lapic_intr_msi                256     1        0    34      541
    vmm`lapic_intr_msi                257     1        0    41      683
    vmm`lapic_intr_msi                  0     1        0    33      757
    vmm`lapic_intr_msi                  1     1        0    42      931
    vmm`vioapic_send_intr             256     1        0    35     2941
    vmm`lapic_intr_msi                  0     1        0    32     9231
    

    With the Hyper-V enlightenment stack, things work on the stock 6.8 kernel:

    [main.hv_interface]
    type = "hyperv"
    reference_tsc = true
    extended_ioapic_rte = true
    ubuntu@ubuntu:~$ uname -a
    Linux ubuntu 6.8.0-31-generic #31-Ubuntu SMP PREEMPT_DYNAMIC Sat Apr 20 00:40:06 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
    ubuntu@ubuntu:~$ sudo dmesg | grep -i -e smpboot -e apic -e hypervisor
    [    0.000000] APIC: Static calls initialized
    [    0.000000] Hypervisor detected: Microsoft Hyper-V
    [    0.032159] x2apic: enabled by BIOS, switching to x2apic ops
    [    0.036488] ACPI: APIC 0x00000000BFB7A000 0000A6 (v01 OVMF   OVMFEDK2 20130221 RVAT 01000000)
    [    0.036495] ACPI: Reserving APIC table memory at [mem 0xbfb7a000-0xbfb7a0a5]
    [    0.036556] APIC: Switched APIC routing to: physical x2apic
    [    0.039984] ACPI: X2APIC_NMI (uid[0xffffffff] dfl dfl lint[0x1])
    [    0.040012] IOAPIC[0]: apic_id 2, version 17, address 0xfec00000, GSI 0-31
    [    0.040031] smpboot: Allowing 4 CPUs, 0 hotplug CPUs
    [    0.138485] APIC: Switch to symmetric I/O mode setup
    [    0.140946] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
    [    0.278999] smpboot: CPU0: AMD Oxide Virtual Turin-like Processor (family: 0x1a, model: 0x2, stepping: 0x1)
    [    0.286589] smpboot: x86: Booting SMP configuration:
    [    0.289499] smpboot: Max logical packages: 1
    [    0.289873] smpboot: Total of 4 processors activated (21559.68 BogoMIPS)
    [    0.328880] ACPI: Using IOAPIC for interrupt routing
    

    Although the topology is a little different for some reason...

    ubuntu@ubuntu:~$ cat /proc/cpuinfo | grep -Ew -e processor -e apicid -e id
    processor       : 0
    physical id     : 0
    core id         : 0
    apicid          : 0
    initial apicid  : 0
    processor       : 1
    physical id     : 0
    core id         : 0
    apicid          : 1
    initial apicid  : 1
    processor       : 2
    physical id     : 0
    core id         : 0
    apicid          : 256
    initial apicid  : 0
    processor       : 3
    physical id     : 0
    core id         : 0
    apicid          : 257
    initial apicid  : 1
    
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions