mmap a file twice with MAP_FIXED on overlapping regions crashes the kernel

[trungnt2910]

#include <unistd.h>
#include <sys/mman.h>
#include <fcntl.h>

int main()
{
    int fd = open("./mmap", O_RDWR);
    
    mmap((void*)0xdeadbeef000, 4096, PROT_NONE, MAP_SHARED | MAP_FIXED, fd, 0);
    mmap((void*)0xdeadbeef000, 4096, PROT_NONE, MAP_SHARED | MAP_FIXED, fd, 0);
}

0xdeadbeef000 can be replaced with any valid userland address, PROT_NONE can be replaced with any valid protection, "./mmap" can be replaced with any path to an existing file, both 4096 can be replaced with any valid area sizes, and both 0 can be replaced with any non-negative offset.

​https://github.com/haiku/haiku/blob/ec97248c2a823c226ba71214692ce99b4a0e4c56/src/system/kernel/vm/vm.cpp#LL1005C1-L1012C1

The code above crashes the kernel because the new cache (the one pointed by fd) and the old cache (the one held by the first mmaped area) is the same. When map_backing_store is called with the new cache, unmap_address_range is called and tries to lock the old cache in order to free the old area. This results in a double lock, making the kernel panic at _mutex_lock.

[waddlesplash]

Some more discussion about fixes in ​https://review.haiku-os.org/c/haiku/+/6392

[waddlesplash]

Was fixed in hrev57062.