Debug button doesn't always work

[anevilyak]

In some circumstances, the Debug button doesn't seem to stop a thread. This started after enabling post syscall debug events for the purposes of output capture. Currently, all that's known for certain is that the actual debug_thread() call makes it down to the kernel and does in fact return success.

[anevilyak]

Confirmed that it's the presence of the B_TEAM_DEBUG_POST_SYSCALL flag that causes the problem, but I'm currently at a loss as to why. Adding tracing to _user_debug_thread() confirms that it indeeds decides all is well and calls thread_interrupt() on the thread as it should, but the expected B_DEBUGGER_MESSAGE_THREAD_DEBUGGED never seems to arrive, and the thread happily keeps executing as best I can tell. Interrupting a thread via a breakpoint, however, consistently still behaves as expected.

An example of such an undebuggable thread is one belonging to a BLooper, which was simply sitting in the kernel waiting for a port read when attempting to debug it. Any ideas?

[anevilyak]

After some further investigation, what seems to happen is that the stop flag on the thread gets trapped by the syscall it was blocked in being interrupted, and consequently the userland debugger simply gets sent the post syscall event alone. As such, unless we want to complicate the set of debug flags stored on the kernel thread structures, it'd appear that the simplest approach would be for Debugger to track that it has a stop request pending, and if it does, then enter debug mode upon the next syscall event it gets that causes the thread to stop.

A quick test seems to confirm that this works, as demonstrated in the above patch. Is that approach acceptable, or should it be generalized to other types of debug events as well?

[bonefish]

Looks good.

[anevilyak]

Thanks! Applied in hrev45819.