Opened 11 years ago

Closed 11 years ago

Last modified 11 years ago

#2552 closed bug (fixed)

PANIC: _mutex_lock(): double lock of 0x90cc4ed0 by thread 3595

Reported by: emitrax Owned by: axeld
Priority: normal Milestone: R1
Component: - General Version: R1/pre-alpha1
Keywords: Cc:
Blocked By: Blocking:
Has a Patch: no Platform: All

Description

It actually happens a few revision ago and I don't know how to trigger it again.

PANIC: _mutex_lock(): double lock of 0x90cc4ed0 by thread 3595
Welcome to Kernel Debugging Land...
Running on CPU 0
kdebug> bt
stack trace for thread 3595 "grep"
    kernel stack: 0x90b50000 to 0x90b54000
      user stack: 0x7efef000 to 0x7ffef000
frame            caller     <image>:function + offset
 0 90b53644 (+  48) 800542cd   <kernel>:invoke_debugger_command + 0x00ed
 1 90b53674 (+  64) 800540c5   <kernel>:invoke_pipe_segment__FP21debugger_command_pipelPc + 0x0079
 2 90b536b4 (+  64) 8005440d   <kernel>:invoke_debugger_command_pipe + 0x009d
 3 90b536f4 (+  48) 800552e8   <kernel>:_ParseCommandPipe__16ExpressionParserRi + 0x0234
 4 90b53724 (+  48) 80054c9e   <kernel>:EvaluateCommand__16ExpressionParserPCcRi + 0x01de
 5 90b53754 (+ 224) 800566b4   <kernel>:evaluate_debug_command + 0x0088
 6 90b53834 (+  64) 800528de   <kernel>:kernel_debugger_loop__Fv + 0x017a
 7 90b53874 (+  48) 8005330f   <kernel>:kernel_debugger + 0x0117
 8 90b538a4 (+ 192) 800531ed   <kernel>:panic + 0x0029
 9 90b53964 (+  64) 80037758   <kernel>:_mutex_lock + 0x00e0
10 90b539a4 (+  48) 8003763f   <kernel>:mutex_switch_lock + 0x005b
11 90b539d4 (+  48) 800a98b1   <kernel>:vm_cache_acquire_locked_page_cache + 0x0085
12 90b53a04 (+  64) 800ace95   <kernel>:steal_page__FP7vm_pageb + 0x0019
13 90b53a44 (+ 128) 800ad076   <kernel>:steal_pages__FPP7vm_pageUlb + 0x0066
14 90b53ac4 (+  48) 800adec7   <kernel>:vm_page_reserve_pages + 0x0097
15 90b53af4 (+ 256) 800a6a0d   <kernel>:vm_soft_fault__FUlbT1 + 0x0225
16 90b53bf4 (+  64) 800a6586   <kernel>:vm_page_fault + 0x002e
17 90b53c34 (+  64) 800b341d   <kernel>:page_fault_exception + 0x00b1
18 90b53c74 (+  12) 800b6976   <kernel>:int_bottom + 0x0036 (nearest)
iframe at 0x90b53c80 (end = 0x90b53cd8)
 eax 0x0            ebx 0x18034000      ecx 0x400        edx 0x914ceaec
 esi 0x8ef80000     edi 0x18034000      ebp 0x90b53d04   esp 0x90b53cb4
 eip 0x800b6608  eflags 0x210206   
 vector: 0xe, error code: 0x2
19 90b53c80 (+ 132) 800b6608   <kernel>:arch_cpu_user_memcpy + 0x001e (nearest)
20 90b53d04 (+ 176) 8002e326   <kernel>:cache_io__FPvT0xUlPUlb + 0x03d2
21 90b53db4 (+  64) 8002ea1d   <kernel>:file_cache_read + 0x002d
22 90b53df4 (+  80) 8059246e   <bfs>:ReadAt__5InodexPUcPUl + 0x00da
23 90b53e44 (+  64) 8059ed6e   <bfs>:bfs_read__FP9fs_volumeP8fs_vnodePvxT2PUl + 0x0076
24 90b53e84 (+  64) 8008a340   <kernel>:file_read__FP15file_descriptorxPvPUl + 0x0044
25 90b53ec4 (+  80) 8007ad8d   <kernel>:common_user_io__FixPvUlb + 0x017d
26 90b53f14 (+  48) 8007b228   <kernel>:_user_read + 0x0028
27 90b53f44 (+ 100) 800b6ba2   <kernel>:pre_syscall_debug_done + 0x0002 (nearest)
iframe at 0x90b53fa8 (end = 0x90b54000)
 eax 0x7c           ebx 0x2be908        ecx 0x7ffeee20   edx 0xffff0104
 esi 0x8000         edi 0x0             ebp 0x7ffeee5c   esp 0x90b53fdc
 eip 0xffff0104  eflags 0x203      
 vector: 0x63, error code: 0x0
28 90b53fa8 (+   0) ffff0104
29 7ffeee5c (+  80) 002028a0   <_APP_>:_term_after + 0x04b8 (nearest)
30 7ffeeeac (+  64) 00203416   <_APP_>:_term_after + 0x102e (nearest)
31 7ffeeeec (+  48) 00203883   <_APP_>:_term_after + 0x149b (nearest)
32 7ffeef1c (+  96) 00204eba   <_APP_>:main + 0x0ef2
33 7ffeef7c (+  48) 00202307   <_APP_>:_start + 0x005b
34 7ffeefac (+  48) 001008ea   80289:runtime_loader_seg0ro@0x00100000 + 0x8ea
35 7ffeefdc (+   0) 7ffeefec   80288:grep_main_stack@0x7efef000 + 0xffffec
kdebug> mutex 0x90cc4ed0
mutex 0x90cc4ed0:
  name:            vm_cache
  flags:           0x0
  holder:          3595
  waiting threads:
kdebug> thread bonnie++
THREAD: 0x914b2000
id:                 1375 (0x55f)
name:               "bonnie++"
all_next:           0x91480000
team_next:          0x00000000
q_next:             0x800fdc00
priority:           10 (next 10)
state:              waiting
next_state:         waiting
cpu:                0x00000000 
sig_pending:        0x0 (blocked: 0x0)
in_kernel:          1
waiting for:        mutex 0x90b8e2d4
fault_handler:      0x00000000
args:               0x90bbeca8 0x00000000
entry:              0x80047f6c
team:               0x90c05744, "bonnie++"
  exit.sem:         15474
  exit.status:      0x0 (No error)
  exit.reason:      0x0
  exit.signal:      0x0
  exit.waiters:
kernel_stack_area:  30812
kernel_stack_base:  0x90b20000
user_stack_area:    30814
user_stack_base:    0x7efef000
user_local_storage: 0x7ffef000
kernel_errno:       0x0 (No error)
kernel_time:        46889683
user_time:          31341082
flags:              0x200
architecture dependant section:
        esp: 0x90b23c68
        ss: 0x00000010
        fpu_state at 0x914b2380
kdebug> bt 1375  
stack trace for thread 1375 "bonnie++"
    kernel stack: 0x90b20000 to 0x90b24000
      user stack: 0x7efef000 to 0x7ffef000
frame            caller     <image>:function + offset
 0 90b23cc4 (+  32) 800404a2   <kernel>:context_switch__FP6threadT0 + 0x0026
 1 90b23ce4 (+  64) 8004070c   <kernel>:scheduler_reschedule + 0x0248
 2 90b23d24 (+  48) 8003780c   <kernel>:_mutex_lock + 0x0194
 3 90b23d54 (+  48) 80036c43   <kernel>:recursive_lock_lock + 0x0053
 4 90b23d84 (+  48) 80597a1f   <bfs>:Lock__7JournalP11Transaction + 0x0023
 5 90b23db4 (+  48) 80597de5   <bfs>:Start__11TransactionP6Volumex + 0x0035
 6 90b23de4 (+  64) 805925d5   <bfs>:WriteAt__5InodeR11TransactionxPCUcPUl + 0x013d
 7 90b23e24 (+  96) 8059ee44   <bfs>:bfs_write__FP9fs_volumeP8fs_vnodePvxPCvPUl + 0x00cc
 8 90b23e84 (+  64) 8008a398   <kernel>:file_write__FP15file_descriptorxPCvPUl + 0x0044
 9 90b23ec4 (+  80) 8007ad8d   <kernel>:common_user_io__FixPvUlb + 0x017d
10 90b23f14 (+  48) 8007b280   <kernel>:_user_write + 0x0028
11 90b23f44 (+ 100) 800b6ba2   <kernel>:pre_syscall_debug_done + 0x0002 (nearest)
iframe at 0x90b23fa8 (end = 0x90b24000)
 eax 0x7e           ebx 0x2f4908        ecx 0x7ffeeb90   edx 0xffff0104
 esi 0x7ffeec3c     edi 0xb4b           ebp 0x7ffeebcc   esp 0x90b23fdc
 eip 0xffff0104  eflags 0x203      
 vector: 0x63, error code: 0x0
12 90b23fa8 (+   0) ffff0104
13 7ffeebcc (+  48) 00204cde   </boot/home/bonnie++-1.03d/bonnie++@0x00200000>:unknown + 0x4cde
14 7ffeebfc (+ 128) 00203a17   </boot/home/bonnie++-1.03d/bonnie++@0x00200000>:unknown + 0x3a17
15 7ffeec7c (+ 768) 0020356b   </boot/home/bonnie++-1.03d/bonnie++@0x00200000>:unknown + 0x356b
16 7ffeef7c (+  48) 0020290f   </boot/home/bonnie++-1.03d/bonnie++@0x00200000>:unknown + 0x290f
17 7ffeefac (+  48) 001008ea   30815:runtime_loader_seg0ro@0x00100000 + 0x8ea
18 7ffeefdc (+   0) 7ffeefec   30814:bonnie++_main_stack@0x7efef000 + 0xffffec
kdebug> mutex 0x90b8e2d4
mutex 0x90b8e2d4:
  name:            bfs journal
  flags:           0x0
  holder:          1377
  waiting threads: 1375 1376
kdebug> bt 1376
stack trace for thread 1376 "bonnie++"
    kernel stack: 0x90b24000 to 0x90b28000
      user stack: 0x7efef000 to 0x7ffef000
frame            caller     <image>:function + offset
 0 90b27cc4 (+  32) 800404a2   <kernel>:context_switch__FP6threadT0 + 0x0026
 1 90b27ce4 (+  64) 8004070c   <kernel>:scheduler_reschedule + 0x0248
 2 90b27d24 (+  48) 8003780c   <kernel>:_mutex_lock + 0x0194
 3 90b27d54 (+  48) 80036c43   <kernel>:recursive_lock_lock + 0x0053
 4 90b27d84 (+  48) 80597a1f   <bfs>:Lock__7JournalP11Transaction + 0x0023
 5 90b27db4 (+  48) 80597de5   <bfs>:Start__11TransactionP6Volumex + 0x0035
 6 90b27de4 (+  64) 805925d5   <bfs>:WriteAt__5InodeR11TransactionxPCUcPUl + 0x013d
 7 90b27e24 (+  96) 8059ee44   <bfs>:bfs_write__FP9fs_volumeP8fs_vnodePvxPCvPUl + 0x00cc
 8 90b27e84 (+  64) 8008a398   <kernel>:file_write__FP15file_descriptorxPCvPUl + 0x0044
 9 90b27ec4 (+  80) 8007ad8d   <kernel>:common_user_io__FixPvUlb + 0x017d
10 90b27f14 (+  48) 8007b280   <kernel>:_user_write + 0x0028
11 90b27f44 (+ 100) 800b6ba2   <kernel>:pre_syscall_debug_done + 0x0002 (nearest)
iframe at 0x90b27fa8 (end = 0x90b28000)
 eax 0x7e           ebx 0x2f4908        ecx 0x7ffeeb90   edx 0xffff0104
 esi 0x7ffeec3c     edi 0x96d           ebp 0x7ffeebcc   esp 0x90b27fdc
 eip 0xffff0104  eflags 0x203      
 vector: 0x63, error code: 0x0
12 90b27fa8 (+   0) ffff0104
13 7ffeebcc (+  48) 00204cde   </boot/home/bonnie++-1.03d/bonnie++@0x00200000>:unknown + 0x4cde
14 7ffeebfc (+ 128) 00203a17   </boot/home/bonnie++-1.03d/bonnie++@0x00200000>:unknown + 0x3a17
15 7ffeec7c (+ 768) 0020356b   </boot/home/bonnie++-1.03d/bonnie++@0x00200000>:unknown + 0x356b
16 7ffeef7c (+  48) 0020290f   </boot/home/bonnie++-1.03d/bonnie++@0x00200000>:unknown + 0x290f
17 7ffeefac (+  48) 001008ea   30846:runtime_loader_seg0ro@0x00100000 + 0x8ea
18 7ffeefdc (+   0) 7ffeefec   30845:bonnie++_main_stack@0x7efef000 + 0xffffec
kdebug> bt 1377
stack trace for thread 1377 "bonnie++"
    kernel stack: 0x90b28000 to 0x90b2c000
      user stack: 0x7efef000 to 0x7ffef000
frame            caller     <image>:function + offset
 0 90b2b6e4 (+  32) 800404a2   <kernel>:context_switch__FP6threadT0 + 0x0026
 1 90b2b704 (+  64) 8004070c   <kernel>:scheduler_reschedule + 0x0248
 2 90b2b744 (+  64) 80041a54   <kernel>:switch_sem_etc + 0x0390
 3 90b2b784 (+  48) 8004166c   <kernel>:acquire_sem + 0x001c
 4 90b2b7b4 (+ 112) 806dfe96   <scsi_periph>:periph_read_write + 0x020a
 5 90b2b824 (+  80) 806dfc33   <scsi_periph>:periph_read + 0x003f
 6 90b2b874 (+  64) 806db974   <scsi_disk>:das_read__FP15das_handle_infoPC9phys_vecsxUlUlPUl + 0x0040
 7 90b2b8b4 (+ 160) 805ab964   <block_io>:block_io_readv_int + 0x042c
 8 90b2b954 (+  80) 805ac140   <block_io>:block_io_read + 0x0040
 9 90b2b9a4 (+  80) 800623ad   <kernel>:devfs_read__FP9fs_volumeP8fs_vnodePvxT2PUl + 0x0121
10 90b2b9f4 (+  64) 8008a340   <kernel>:file_read__FP15file_descriptorxPvPUl + 0x0044
11 90b2ba34 (+  96) 8007b665   <kernel>:_kern_read + 0x0145
12 90b2ba94 (+  64) 800bfc5a   <kernel>:read_pos + 0x0036
13 90b2bad4 (+  80) 8002a06f   <kernel>:get_cached_block__FP11block_cachexPbb + 0x010b
14 90b2bb24 (+  80) 8002ce0e   <kernel>:block_cache_get_etc + 0x004e
15 90b2bb74 (+  64) 80583a9a   <bfs>:SetTo__15AllocationBlockR15AllocationGroupUs + 0x00ae
16 90b2bbb4 (+ 128) 80584b09   <bfs>:AllocateBlocks__14BlockAllocatorR11TransactionlUsUsUsR9block_run + 0x017d
17 90b2bc34 (+  80) 80584f5f   <bfs>:Allocate__14BlockAllocatorR11TransactionP5InodexR9block_runUs + 0x018f
18 90b2bc84 (+ 304) 80592c50   <bfs>:_GrowStream__5InodeR11Transactionx + 0x0300
19 90b2bdb4 (+  48) 80594249   <bfs>:SetFileSize__5InodeR11Transactionx + 0x0069
20 90b2bde4 (+  64) 80592643   <bfs>:WriteAt__5InodeR11TransactionxPCUcPUl + 0x01ab
21 90b2be24 (+  96) 8059ee44   <bfs>:bfs_write__FP9fs_volumeP8fs_vnodePvxPCvPUl + 0x00cc
22 90b2be84 (+  64) 8008a398   <kernel>:file_write__FP15file_descriptorxPCvPUl + 0x0044
23 90b2bec4 (+  80) 8007ad8d   <kernel>:common_user_io__FixPvUlb + 0x017d
24 90b2bf14 (+  48) 8007b280   <kernel>:_user_write + 0x0028
25 90b2bf44 (+ 100) 800b6ba2   <kernel>:pre_syscall_debug_done + 0x0002 (nearest)
iframe at 0x90b2bfa8 (end = 0x90b2c000)
 eax 0x7e           ebx 0x2f4908        ecx 0x7ffeeb90   edx 0xffff0104
 esi 0x7ffeec3c     edi 0xaf9           ebp 0x7ffeebcc   esp 0x90b2bfdc
 eip 0xffff0104  eflags 0x203      
 vector: 0x63, error code: 0x0
26 90b2bfa8 (+   0) ffff0104
27 7ffeebcc (+  48) 00204cde   </boot/home/bonnie++-1.03d/bonnie++@0x00200000>:unknown + 0x4cde
28 7ffeebfc (+ 128) 00203a17   </boot/home/bonnie++-1.03d/bonnie++@0x00200000>:unknown + 0x3a17
29 7ffeec7c (+ 768) 0020356b   </boot/home/bonnie++-1.03d/bonnie++@0x00200000>:unknown + 0x356b
30 7ffeef7c (+  48) 0020290f   </boot/home/bonnie++-1.03d/bonnie++@0x00200000>:unknown + 0x290f
31 7ffeefac (+  48) 001008ea   30877:runtime_loader_seg0ro@0x00100000 + 0x8ea
32 7ffeefdc (+   0) 7ffeefec   30876:bonnie++_main_stack@0x7efef000 + 0xffffec
kdebug> thread 1377
THREAD: 0x914b6800
id:                 1377 (0x561)
name:               "bonnie++"
all_next:           0x91487800
team_next:          0x00000000
q_next:             0x800fdc00
priority:           10 (next 10)
state:              ready
next_state:         ready
cpu:                0x00000000 
sig_pending:        0x0 (blocked: 0x0)
in_kernel:          1
waiting for:        fault_handler:      0x00000000
args:               0x90b71618 0x00000000
entry:              0x80047f6c
team:               0x90c05e88, "bonnie++"
  exit.sem:         15504
  exit.status:      0x0 (No error)
  exit.reason:      0x0
  exit.signal:      0x0
  exit.waiters:
kernel_stack_area:  30874
kernel_stack_base:  0x90b28000
user_stack_area:    30876
user_stack_base:    0x7efef000
user_local_storage: 0x7ffef000
kernel_errno:       0x0 (No error)
kernel_time:        47384764
user_time:          31419292
flags:              0x0
architecture dependant section:
        esp: 0x90b2b688
        ss: 0x00000010
        fpu_state at 0x914b6b80
kdebug> continue

Attachments (1)

file_cache.diff (824 bytes) - added by emitrax 11 years ago.

Download all attachments as: .zip

Change History (6)

comment:1 Changed 11 years ago by bonefish

The problem is relatively simple: cache_io() uses user_memcpy() with the cache lock held. The user memory is not yet mapped, and vm_page_reserve_pages() coincidently tries to steal a page from the same cache. Possible solutions:

  • Lock the memory upfront.
  • Mark the pages busy, unlock the cache, user_memcpy(), lock the cache, mark pages unbusy.

comment:2 Changed 11 years ago by emitrax

I actually mananaged to triggered it again by simply running bonnie++. I tried with the attached patch as you said. Please review.

Changed 11 years ago by emitrax

Attachment: file_cache.diff added

comment:3 Changed 11 years ago by bonefish

Resolution: fixed
Status: newclosed

Should be fixed in hrev26700.

comment:4 in reply to:  2 Changed 11 years ago by scottmc

Replying to emitrax:

I actually mananaged to triggered it again by simply running bonnie++. I tried with the attached patch as you said. Please review.

@emitrax, Does that mean that you fixed #1412?

comment:5 Changed 11 years ago by emitrax

@scottmc: Partially. The sem.h has been implemented but not heavely tested.

Note: See TracTickets for help on using tickets.