virtio devices often timeout (never receive interrupts) under guarded heap

[waddlesplash]

I built an image with the guarded heap enabled (USE_GUARDED_HEAP_FOR_MALLOC=1, USE_GUARDED_HEAP_FOR_OBJECT_CACHE=0). When booting it in QEMU with software emulation (no KVM), virtio-scsi and virtio-net often fail to initialize. Sometimes virtio-scsi will work but virtio-net will not; sometimes virtio-scsi will fail to initialize with these messages:

msix configured for 4 vectors
msi-x enabled: 0x8003
virtio_pci: using MSI-X count 4 starting at 24
virtio_scsi: wait failed with status: 0x80000009

(and thus the boot will fail.)

When the devices have failed to initialize, the interrupt count as displayed by "ints" always seems to be 0.

When not using the guarded heap, I haven't gotten equivalent failures yet. The fact that it only happens sometimes under the guarded heap sounds like it could be a race condition.

This is with current master and QEMU 7.0. QEMU invocation:

qemu-system-x86_64 -display sdl -m 4G -nic model=virtio -device virtio-scsi-pci -device scsi-hd,drive=x0 -drive file=haiku-minimum.image,format=raw,if=none,id=x0 -serial stdio -s

[kallisti5]

reproduced with:

diff --git a/build/config_headers/kernel_debug_config.h b/build/config_headers/kernel_debug_config.h
index 22f0a40b90..23ff2972d4 100644
--- a/build/config_headers/kernel_debug_config.h
+++ b/build/config_headers/kernel_debug_config.h
@@ -104,14 +104,14 @@
 // Use the selected allocator as generic memory allocator (malloc()/free()).
 #define USE_DEBUG_HEAP_FOR_MALLOC              0
        // Heap implementation with additional debugging facilities.
-#define USE_GUARDED_HEAP_FOR_MALLOC            0
+#define USE_GUARDED_HEAP_FOR_MALLOC            1
        // Heap implementation that allocates memory so that the end of the
        // allocation always coincides with a page end and is followed by a guard
        // page which is marked non-present. Out of bounds access (both read and
        // write) therefore cause a crash (unhandled page fault). Note that this
        // allocator is neither speed nor space efficient, indeed it wastes huge
        // amounts of pages and address space so it is quite easy to hit limits.
-#define USE_SLAB_ALLOCATOR_FOR_MALLOC  1
+#define USE_SLAB_ALLOCATOR_FOR_MALLOC  0
        // Heap implementation based on the slab allocator (for production use).
 
 // Replace the object cache with the guarded heap to force debug features. Also

Plus ​https://review.haiku-os.org/c/haiku/+/8369

[kallisti5]

debug log. Guarded heap + debug in virtio_pci / virtio_scsi

[kallisti5]

virtio interrupts (added panic to the wait timeout)

[kallisti5]

guarded heap area info. interrupt data addresses

[korli]

by bisecting, it looks like it started with hrev57612. hrev57611 doesn't expose the problem.