[nvme_disk] MSI-X broken

[waddlesplash]

(I am leaving this an NVMe ticket for now, but I suspect it may be an issue in our PCI bus.)

MSI-X interrupt mode is broken in recent QEMUs for nvme_disk. (NVMe's emulated controller does not support MSI, apparently, only legacy interrupts or MSI-X.) I made a series of custom builds of QEMU in an attempt to diagnose and then fix this problem, but was ultimately mystified, hence this ticket.

What I did determine is that there may be two problems: first, that QEMU may expect MSIs to already be configured at the time qpairs are created (this is fixable by moving "set up interrupt" block above the "allocate qpairs" block.)

The second, and major, problem is that "msix_notify" in QEMU's hw/pci/msix.c always returns due to "msix_is_masked" returning true. This appears, on further tracing, to be because PCI_MSIX_ENTRY_CTRL_MASKBIT is set on the vector. That should have been unset by Haiku's pci_configure_msix (we call that bit PCI_msix_vctrl_mask, which I think should have triggered QEMU's msix_table_mmio_write; but that function is never invoked.

I tested with an "e1000e" network device in QEMU, which also supports MSI-X, and there everything seemed to work fine. So, why do MSI-X interrupts fail to unmask for NVMe, then?

[waddlesplash]

I disabled MSI-X for nvme_disk in hrev55520 until we can figure out what is going on here.

[waddlesplash]

korli: I think you know more about MSI-X in Haiku than I do, any idea what might be going wrong here or what else can be investigated?

[korli]

I submitted ​https://review.haiku-os.org/c/haiku/+/4613 for review. I doubt it helps in this case.

[waddlesplash]

Apparently #15874 also started working after the commit that disabled MSI-X, so, this is not restricted just to QEMU.

[waddlesplash]

The mystery deepens: someone has opened a ticket in which nvme_disk interrupts were apparently broken by disabling MSI-X in the driver; it seems they worked under MSI-X but (awaiting confirmation) are broken under MSI: #17352.

[waddlesplash]

FreeBSD's MSI/X capability reading: ​https://xref.landonf.org/source/xref/freebsd-current/sys/dev/pci/pci.c#923

[waddlesplash]

Well, after re-ordering interrupts initialization again, it now seems to work, so I suppose your changes to support 64-bit addresses actually fixed it after all.