Opened 10 years ago

Last modified 10 years ago

#11032 assigned bug

Panic with message "failed to acquire spinlock %p for a long time" in usleep

Reported by: zhuowei Owned by: pdziepak
Priority: normal Milestone: R1
Component: System/Kernel Version: R1/Development
Keywords: Cc:
Blocked By: Blocking: #11314, #11344
Platform: All

Description

Similar to #10934, but the system panicked in usleep() instead.

This happened on hrev47380, on a gcc2 hybrid running in VMWare Player 6.0.1.

I was running the tests for the Golang toolchain; I left the computer for a while, and when I returned the display was frozen.

The last window displayed was an "application has crashed" dialog for one of the Golang tests.

After attaching a serial console, the backtrace showed that the system had panicked with the message "failed to acquire spinlock %p for a long time".

Attached is the backtrace from the serial console.

Attachments (4)

kdl_log.txt (4.0 KB ) - added by zhuowei 10 years ago.
backtrace from the serial console
kdl_log2.txt (2.8 KB ) - added by zhuowei 10 years ago.
Another KDL log of the same crash
kdl_log3.txt (11.5 KB ) - added by zhuowei 10 years ago.
output of running, thread, sc, and spinlock commands in the kernel debugger
pprof.test (3.3 MB ) - added by zhuowei 10 years ago.
The executable that causes the crash

Change History (12)

by zhuowei, 10 years ago

Attachment: kdl_log.txt added

backtrace from the serial console

by zhuowei, 10 years ago

Attachment: kdl_log2.txt added

Another KDL log of the same crash

comment:1 by bonefish, 10 years ago

Please also add stack traces for the threads running on the other CPUs (running shows the running threads). Furthermore, if that doesn't show the culprit, please enter the spinlock with the spinlock address from the panic message. It should print a "locked from" address which you can feed to ls to get the name of the function that acquired the spinlock.

by zhuowei, 10 years ago

Attachment: kdl_log3.txt added

output of running, thread, sc, and spinlock commands in the kernel debugger

by zhuowei, 10 years ago

Attachment: pprof.test added

The executable that causes the crash

comment:2 by zhuowei, 10 years ago

I attached the executable that causes the crash - it should be able to run on any 32-bit GCC2 hybrid; hopefully this helps to track down the cause.

comment:3 by bonefish, 10 years ago

Owner: changed from axeld to pdziepak
Status: newassigned

comment:4 by mmlr, 10 years ago

Blocking: 11314 added

(In #11314) This is the same as #11032.

comment:5 by mmlr, 10 years ago

Blocking: 11344 added

(In #11344) This is the same as #11032.

comment:6 by mmlr, 10 years ago

A workaround for this has been pushed as hrev48139. Leaving ticket open for pdziepak to rework the code properly.

comment:7 by jessicah, 10 years ago

FYI, the workaround only half fixes the issue. It still panics in BKernel::TeamSignalEvent::DoDPC.

in reply to:  7 comment:8 by mmlr, 10 years ago

Replying to jessicah:

FYI, the workaround only half fixes the issue. It still panics in BKernel::TeamSignalEvent::DoDPC.

No, that must be a different issue and is not tracked here. DoDPC uses a different spinlock (the team signal spinlock) and none of the 6 total backtraces in these tickets point to it. The panic message is pointing out a general failure to lock a spinlock, any number of places with a deadlock/double lock can be responsible for that and each of those places are different issues. Please open a new ticket with a backtrace and steps to reproduce for the DoDPC case.

Note: See TracTickets for help on using tickets.