Deadlock in error case of vm.cpp:map_backing_store()

[bonefish]

Revision hrev21887. Executing the runtime_loader causes a deadlock in the kernel. When the runtime_loader tries to map its own executable, this fails in map_backing_store(), which in the error case deletes the cache it created. The cache is a consumer of the supplied cache, which in turn has been locked by the caller. Eventually vm_cache_remove_consumer() is invoked with the already locked cache, thus causing the deadlock.

stack trace for thread 0x9d "runtime_loader"
    kernel stack: 0x93491000 to 0x93495000
      user stack: 0x7efe7000 to 0x7ffe7000
frame            caller     <image>:function + offset
934946d4 (+  32) 80032562   <kernel>:context_switch__FP6threadT0 + 0x0026
934946f4 (+  64) 800327bf   <kernel>:scheduler_reschedule + 0x023b
93494734 (+  96) 80033653   <kernel>:switch_sem_etc + 0x03a7
93494794 (+  48) 80033254   <kernel>:acquire_sem + 0x001c
934947c4 (+  48) 8002d2ad   <kernel>:mutex_lock + 0x002d
934947f4 (+  80) 8005d820   <kernel>:vm_cache_remove_consumer + 0x0018
93494844 (+  48) 8005d497   <kernel>:vm_cache_release_ref + 0x0143
93494874 (+  64) 80056d18   <kernel>:map_backing_store__FP16vm_address_spaceP8vm_cachePPvxUlUliiiPP7vm_areaPCc + 0x01f4
934948b4 (+ 112) 80057974   <kernel>:_vm_map_file__FlPCcPPvUlUlUlUlT1xb + 0x0104
93494924 (+1136) 8005c9a1   <kernel>:_user_vm_map_file + 0x00cd
93494d94 (+ 368) 80036c6b   <kernel>:syscall_dispatcher + 0x0e57
93494f04 (+ 160) 80092da8   <kernel>:i386_handle_trap + 0x02b0
iframe at 0x93494fac (end = 0x93495000)
 eax 0x86           ebx 0x116ecc        ecx 0x1          edx 0x119004
 esi 0x0            edi 0x0             ebp 0x7ffe56fc   esp 0x93494fdc
 eip 0xffff0102  eflags 0x246
 vector: 0x63, error code: 0x0
93494fa4 (+   0) ffff0102
7ffe56fc (+6256) 00101f04   1286:runtime_loader_seg0ro@0x00100000 + 0x1f04
7ffe6f6c (+  64) 001024ee   1286:runtime_loader_seg0ro@0x00100000 + 0x24ee
7ffe6fac (+  48) 00100782   1286:runtime_loader_seg0ro@0x00100000 + 0x782
7ffe6fdc (+   0) 7ffe6fec   1285:/boot/beos/system/runtime_loade@0x7efe7000 + 0xffffec

[bonefish]

Fixed in hrev21914. It's now just unlocking the source cache temporarily, which doesn't seem to be a problem for any caller.

[axeld]

My local solution has been to move the add consumer call to a place after everything else went okay. Since the cache is locked, this would have been safe, too. What do you think?

[bonefish]

Replying to axeld:

My local solution has been to move the add consumer call to a place after everything else went okay. Since the cache is locked, this would have been safe, too. What do you think?

I considered this first too, but was sceptical about inserting the area with a cache that has been connected correctly. It shouldn't matter, of course, since the caller has the address space write lock. So, well, change it, if you like that better.