You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Going up to 254 vCPUs is straightforward enough, and with a recent Helios change we're free to bump Propolis' MAXCPU up to 254, but going beyond that will involve some work.
There's the (easier) control plane side, where the Propolis API currently wants a u8 of vCPUs. That won't do in the bold manymore core future. this is mostly addressed in configurable CPU topology #1237
There's the (maybe easy, maybe hard) side, where we'll want guests to be able to direct interrupts to high-number CPUs if they're so inclined.
The obvious and somewhat daunting solution would be for Propolis to grow an IOMMU which can be used to let guests remap interrupts towards high-number cores like you'd do on real hardware.
https://david.woodhou.se/ExtDestId.pdf describes an alternative that might be workable: Hyper-V, KVM, FreeBSD byhve, and Xen all support a hypervisor feature to expand APIC IDs as used in MSI, but guest OS support is more mixed. Reportedly Linux supports this, Windows would be in a weird state either way, and the FreeBSD issue linked there is still open. I assume it would be mildly better for everyone to help get OSes in a good state with this than to build an IOMMU for Propolis, when the need comes..
Additionally, on Turin (and later? time will tell!) there's support for IOMMU virtualization. That is, the MMIO EFR register has vIommuSup set, as well as GTSup. that suggests maybe we can just .. offer a virtualized IOMMU and be done with it? given the IOMMU spec is 313 pages and I'm not too directly familiar with the AMD IOMMU I'd have questions for other knowers before being too sure, but this seems like a promising option for us. presumably this would mean Propolis' device emulation would have to be aware of any address translation the emulated devices ought to be doing? maybe there's something clever we can do about that.
as I'm going through the ACPI table work in #999, the Processor Local APIC Structure definitions make clear that there is some work to do on the ACPI side too for many-processor VMs. probably?
the Local APIC structure has a one-byte field for APIC ID and for ACPI Processor UID. these clearly won't do when the number overflows. additionally, Local APIC NMI Structure has a one-byte ACPI Processor UID field as well. in both cases, ACPI points out that the Processor declaration operator is deprecated, that there's a compatibility note in the x2APIC forms of the declarations, and to read more there.
which tells me that in addition to however we're targeting interrupts (ExtDestId or an emulated iommu, as testing guides us) we'll want x2apic tested and working and to generate x2APIC-style processor definitions here.
Oh yes. It's painfully tedious, but we'll have to plumb the x2 versions of data for this stuff into the guests to get them to use more than 255 VCPUs. E.g., instead of (or in addition to) using the "Processor Local APIC Structure" of ACPI 6.6 sec 5.2.12.2, we'll have to supply the "Processor Local x2APIC Structure" of sec 5.2.12.12.
I highly recommend using the enlightments Woodhouse mentioned in the links above; doing a virtualized IOMMU is going to be neither fun nor easy, even just for interrupt remapping, and all the guests we need to worry about should already support them.
whoops, another place where we don't handle larger vCPU counts: struct Board uses a u8. starting such a large VM will fail on the Nexus side because the u16 we've got for the vCPU count doesn't fit in a u8.
Not quite ready yet to launch a VM with >255 vCPUs but confirmed that with x2APIC + the 15-bit Extended MSI Destination ID support, guests can properly address vCPUs with APIC IDs above 254.
To test that I extended propolis-standalone to allow explicitly assigning vCPU IDs sparsely (#1263 on a host with illumos#18688 + VM_MAXCPU bumped to 4096):
[main]
name = "ubuntu"bootrom = "OVMF_CODE.fd"cpus = 4vcpu_ids = [0, 1, 256, 257]
memory = 4096cpuid_profile = "turin_v2_x2apic"
[main.hv_interface]
type = "bhyve"ext_msi_dest_id = true# ... rest of config
OVMF upon discovering a CPU with APIC ID >254 switches to x2APIC mode:
Force x2APIC mode!
CpuMpPei: 5-Level Paging = 0
APIC MODE is 2
MpInitLib: Find 4 processors in system.
GetMicrocodePatchInfoFromHob: Microcode patch cache HOB is not found.
CpuMpPei: 5-Level Paging = 0
CPU[0000]: Microcode revision = 00000000, expected = 00000000
CPU[0002]: Microcode revision = 00000000, expected = 00000000
Register PPI Notify: 8F9D4825-797D-48FC-8471-845025792EF6
Does not find any stored CPU BIST information from PPI!
APICID - 0x00000000, BIST - 0x00000000
APICID - 0x00000001, BIST - 0x00000000
APICID - 0x00000100, BIST - 0x00000000
APICID - 0x00000101, BIST - 0x00000000
We get to Linux with x2APIC enabled but with stock Ubuntu 24.04, it's running Linux kernel 6.8.0 which is too old to recognize bhyve's ext-msi-dest-id feature bit and switches back to apic:
ubuntu@ubuntu:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 24.04 LTS
Release: 24.04
Codename: noble
ubuntu@ubuntu:~$ uname -a
Linux ubuntu 6.8.0-31-generic #31-Ubuntu SMP PREEMPT_DYNAMIC Sat Apr 20 00:40:06 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@ubuntu:~$ sudo dmesg | grep -i -e smpboot -e apic
[ 0.000000] APIC: Static calls initialized
[ 0.000341] x2apic: enabled by BIOS, switching to x2apic ops
[ 0.004755] ACPI: APIC 0x00000000BFB7A000 0000A6 (v01 OVMF OVMFEDK2 20130221 RVAT 01000000)
[ 0.004762] ACPI: Reserving APIC table memory at [mem 0xbfb7a000-0xbfb7a0a5]
[ 0.004838] APIC: Switched APIC routing to: cluster x2apic
[ 0.051703] ACPI: X2APIC_NMI (uid[0xffffffff] dfl dfl lint[0x1])
[ 0.051737] IOAPIC[0]: apic_id 2, version 17, address 0xfec00000, GSI 0-31
[ 0.051767] smpboot: Allowing 4 CPUs, 0 hotplug CPUs
[ 0.203866] APIC: Switch to symmetric I/O mode setup
[ 0.204675] x2apic: IRQ remapping doesn't support X2APIC mode
[ 0.205315] x2apic disabled
[ 0.210172] native_apic_msr_read+0x1b/0x60
[ 0.210594] apic_read_boot_cpu_id+0x69/0x90
[ 0.211030] apic_set_fixmap+0x75/0x90
[ 0.211403] enable_IR_x2apic+0x258/0x2b0
[ 0.211794] x86_64_probe_apic+0xf/0x70
[ 0.212170] apic_intr_mode_init+0x8f/0x130
[ 0.215765] APIC: Switched APIC routing to: flat
[ 0.217978] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[ 0.364090] smpboot: CPU0: AMD Oxide Virtual Turin-like Processor (family: 0x1a, model: 0x2, stepping: 0x1)
[ 0.375325] smpboot: native_kick_ap: bad cpu 2
[ 0.375853] smpboot: native_kick_ap: bad cpu 2
[ 0.377403] smpboot: x86: Booting SMP configuration:
[ 0.379423] smpboot: native_kick_ap: bad cpu 3
[ 0.382851] smpboot: native_kick_ap: bad cpu 3
[ 0.384654] smpboot: Max logical packages: 2
[ 0.384838] smpboot: Total of 2 processors activated (10779.41 BogoMIPS)
[ 0.456838] ACPI: Using IOAPIC for interrupt routing
We can at least see the state transitions work properly:
Activity
iximeow commentedon May 18, 2026
as I'm going through the ACPI table work in #999, the
Processor Local APIC Structuredefinitions make clear that there is some work to do on the ACPI side too for many-processor VMs. probably?the Local APIC structure has a one-byte field for
APIC IDand forACPI Processor UID. these clearly won't do when the number overflows. additionally,Local APIC NMI Structurehas a one-byteACPI Processor UIDfield as well. in both cases, ACPI points out that theProcessor declaration operator is deprecated, that there's a compatibility note in the x2APIC forms of the declarations, and to read more there.which tells me that in addition to however we're targeting interrupts (ExtDestId or an emulated iommu, as testing guides us) we'll want x2apic tested and working and to generate x2APIC-style processor definitions here.
dancrossnyc commentedon May 21, 2026
Oh yes. It's painfully tedious, but we'll have to plumb the x2 versions of data for this stuff into the guests to get them to use more than 255 VCPUs. E.g., instead of (or in addition to) using the "Processor Local APIC Structure" of ACPI 6.6 sec 5.2.12.2, we'll have to supply the "Processor Local x2APIC Structure" of sec 5.2.12.12.
I highly recommend using the enlightments Woodhouse mentioned in the links above; doing a virtualized IOMMU is going to be neither fun nor easy, even just for interrupt remapping, and all the guests we need to worry about should already support them.
iximeow commentedon Jun 5, 2026
whoops, another place where we don't handle larger vCPU counts: struct Board uses a u8. starting such a large VM will fail on the Nexus side because the u16 we've got for the vCPU count doesn't fit in a u8.
simoneepperson commentedon Oct 29, 2026
Illumos related work
in progress: complete 11/6https://www.illumos.org/issues/18522
https://www.illumos.org/issues/18523
https://www.illumos.org/issues/18524
https://code.illumos.org/c/illumos-gate/+/5100
https://www.illumos.org/issues/18525
https://code.illumos.org/c/illumos-gate/+/5101
https://www.illumos.org/issues/18526
https://www.illumos.org/issues/18538
https://www.illumos.org/issues/18527
https://www.illumos.org/issues/18528
https://code.illumos.org/c/illumos-gate/+/5102
https://www.illumos.org/issues/18539
https://code.illumos.org/c/illumos-gate/+/5103
iximeow commentedon Oct 30, 2026
#1237 is one piece of this that's here in the Propolis repo. this chases out all cases of "cpu is a u8" that I know of
morlandi7 commentedon Nov 6, 2026
Pile of illumos issues above now complete upstream in illumos but need to be added to oxide's version. Next up:
https://www.illumos.org/issues/16721 x2APIC mode should persist across INIT in bhyve
https://www.illumos.org/issues/16722 allow x2APIC mode changes in bhyve
https://www.illumos.org/issues/16723 bhyve x2APIC should disallow improper MSR accesses
Expecting a few more after these + propolis work
luqmana commentedon Nov 19, 2026
https://www.illumos.org/issues/18681 bhyve: Add CPUID_BHYVE_FEATURES leaf
https://www.illumos.org/issues/18688 bhyve: Support and advertise 15-bit MSI Extended Destination ID
luqmana commentedon Nov 19, 2026
Not quite ready yet to launch a VM with >255 vCPUs but confirmed that with x2APIC + the 15-bit Extended MSI Destination ID support, guests can properly address vCPUs with APIC IDs above 254.
To test that I extended propolis-standalone to allow explicitly assigning vCPU IDs sparsely (#1263 on a host with illumos#18688 +
VM_MAXCPUbumped to4096):OVMF upon discovering a CPU with APIC ID >254 switches to x2APIC mode:
Tracing
wrmsrtoIA32_APIC_BASEreveals the same:Reads of the ID register for each return as expected:
We have our INITs on and SIPI:
We get to Linux with x2APIC enabled but with stock Ubuntu 24.04, it's running Linux kernel 6.8.0 which is too old to recognize bhyve's ext-msi-dest-id feature bit and switches back to apic:
We can at least see the state transitions work properly:
Install one of the HWE packages like
linux-image-virtual-hwe-24.04though and we get a 7.0.0 kernel and now we're cooking:Our sparse vCPU ID allocation is reflected in the guest:
And the latter 2 vCPUs are successfully getting device interrupts demonstrating the 15-bit MSI Destination ID working:
We can also confirm the IOAPIC path works by pinning legacy interrupts for the serial port to the vCPUs 256/257 (CPU2/3 in the guest):
Tracing the bhyve lapic/ioapic emulation further confirms the guest is making use of the extended destination ID bits for vCPUs above 255:
With the Hyper-V enlightenment stack, things work on the stock 6.8 kernel:
Although the topology is a little different for some reason...