Kernel Panic on Multiple user_xsi_msgget Syscall Invocations

[thosewhowork]

Hello,

It is possible to cause a kernel panic by invoking the xsi_msgget syscall with fuzzed low integer inputs.

I believe the issue is a NULL pointer dereference here: ​https://github.com/haiku/haiku/blob/master/src/system/kernel/posix/xsi_message_queue.cpp#L685-L686

Where messageQueue is still NULL after calling .Lookup

This test program should reproduce the issue:

#include <dlfcn.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

typedef int (*syscall_kern_xsi_msgget)(int, int);

int
main(int argc, char** argv) {
    int i = 0;
    syscall_kern_xsi_msgget p_kern_xsi_msgget = NULL;

    if ((p_kern_xsi_msgget = dlsym(RTLD_DEFAULT, "_kern_xsi_msgget")) == NULL) {
        return 1;
    }
    for (; i < 0xffff; i++) {
        printf("%d\n", i);
        fflush(stdout);
        usleep(500);
        p_kern_xsi_msgget(i, i);
        p_kern_xsi_msgget(i, i);
    }
    return 0;
}

Compiled with:

gcc -o xsi_msgget_syscall src/X-xsi_msgget_syscall.c

System Details

Haiku Version: Haiku Nightly rev54891

Guest Machine: VMWare Fusion 8.5.3

Host Machine: Mac OS Mojave

I've attached the backtrace from the kernel debugger.

Thanks!

[thosewhowork]

xsi_msgget trace

[korli]

Thanks! ​https://review.haiku-os.org/c/haiku/+/3643

[waddlesplash]

Fix merged. Thanks again for reporting!