Opened 5 years ago

Last modified 5 years ago

#11032 assigned bug

Panic with message "failed to acquire spinlock %p for a long time" in usleep

Reported by: zhuowei Owned by: pdziepak
Priority: normal Milestone: R1
Component: System/Kernel Version: R1/Development
Keywords: Cc:
Blocked By: Blocking: #11314, #11344
Has a Patch: no Platform: All

Description

Similar to #10934, but the system panicked in usleep() instead.

This happened on hrev47380, on a gcc2 hybrid running in VMWare Player 6.0.1.

I was running the tests for the Golang toolchain; I left the computer for a while, and when I returned the display was frozen.

The last window displayed was an "application has crashed" dialog for one of the Golang tests.

After attaching a serial console, the backtrace showed that the system had panicked with the message "failed to acquire spinlock %p for a long time".

Attached is the backtrace from the serial console.

Attachments (4)

kdl_log.txt (4.0 KB) - added by zhuowei 5 years ago.
backtrace from the serial console
kdl_log2.txt (2.8 KB) - added by zhuowei 5 years ago.
Another KDL log of the same crash
kdl_log3.txt (11.5 KB) - added by zhuowei 5 years ago.
output of running, thread, sc, and spinlock commands in the kernel debugger
pprof.test (3.3 MB) - added by zhuowei 5 years ago.
The executable that causes the crash

Change History (12)

Changed 5 years ago by zhuowei

Attachment: kdl_log.txt added

backtrace from the serial console

Changed 5 years ago by zhuowei

Attachment: kdl_log2.txt added

Another KDL log of the same crash

comment:1 Changed 5 years ago by bonefish

Please also add stack traces for the threads running on the other CPUs (running shows the running threads). Furthermore, if that doesn't show the culprit, please enter the spinlock with the spinlock address from the panic message. It should print a "locked from" address which you can feed to ls to get the name of the function that acquired the spinlock.

Changed 5 years ago by zhuowei

Attachment: kdl_log3.txt added

output of running, thread, sc, and spinlock commands in the kernel debugger

Changed 5 years ago by zhuowei

Attachment: pprof.test added

The executable that causes the crash

comment:2 Changed 5 years ago by zhuowei

I attached the executable that causes the crash - it should be able to run on any 32-bit GCC2 hybrid; hopefully this helps to track down the cause.

comment:3 Changed 5 years ago by bonefish

Owner: changed from axeld to pdziepak
Status: newassigned

comment:4 Changed 5 years ago by mmlr

Blocking: 11314 added

(In #11314) This is the same as #11032.

comment:5 Changed 5 years ago by mmlr

Blocking: 11344 added

(In #11344) This is the same as #11032.

comment:6 Changed 5 years ago by mmlr

A workaround for this has been pushed as hrev48139. Leaving ticket open for pdziepak to rework the code properly.

comment:7 Changed 5 years ago by jessicah

FYI, the workaround only half fixes the issue. It still panics in BKernel::TeamSignalEvent::DoDPC.

comment:8 in reply to:  7 Changed 5 years ago by mmlr

Replying to jessicah:

FYI, the workaround only half fixes the issue. It still panics in BKernel::TeamSignalEvent::DoDPC.

No, that must be a different issue and is not tracked here. DoDPC uses a different spinlock (the team signal spinlock) and none of the 6 total backtraces in these tickets point to it. The panic message is pointing out a general failure to lock a spinlock, any number of places with a deadlock/double lock can be responsible for that and each of those places are different issues. Please open a new ticket with a backtrace and steps to reproduce for the DoDPC case.

Note: See TracTickets for help on using tickets.