Opened 7 years ago

Closed 7 years ago

#6288 closed bug (fixed)

popen deadlocks Paladin on multicore machines

Reported by: DarkWyrm Owned by: Ingo Weinhold
Priority: normal Milestone: R1
Component: System/POSIX Version: R1/alpha2
Keywords: Cc:
Blocked By: Blocking:
Has a Patch: no Platform: All

Description

This only occurs when multithreaded builds are enabled. Here are the steps to reproduce the error:

1) Load a project on an SMP machine. 2) Force a rebuild of the project 3) Run the build.

On my quad core system, perhaps 1 file might get built. Typically not, though. It deadlocks. Killing Paladin from the Deskbar at this point causes a crash with the following backtrace:

0x00356031 in _ZL7do_execPKcPKPcS3_b () from /boot/system/lib/libroot.so
(gdb) bt
#0  0x00356031 in _ZL7do_execPKcPKPcS3_b () from /boot/system/lib/libroot.so
#1  0x0035623f in execl () from /boot/system/lib/libroot.so
#2  0x00318de2 in _IO_proc_open () from /boot/system/lib/libroot.so
#3  0x00319000 in popen () from /boot/system/lib/libroot.so
#4  0x0028a53c in SourceFileC::Compile (this=0x1802c468, info=@0x18069174, 
    options=0x18189de8 "-O0 -I '/boot/home/projects/BeVexed' -I '/boot/home/projects/BeVexed/.' -I '/boot/develop/headers/be' -I '/boot/develop/headers/cpp' -I '/boot/develop/headers/posix' -I '/boot/home/config/include' ")
    at /boot/home/projects/Paladin-related/Paladin/SourceTypeC.cpp:309
#5  0x0024968d in Project::CompileFile (this=0x18069050, file=0x1802c468)
    at /boot/home/projects/Paladin-related/Paladin/Project.cpp:701
#6  0x002872e6 in ProjectBuilder::BuildThread (data=0x18062318)
    at /boot/home/projects/Paladin-related/Paladin/ProjectBuilder.cpp:343
#7  0x002ef4ec in thread_entry () from /boot/system/lib/libroot.so
#8  0x70184fec in ?? ()

Attachments (3)

syslog (15.8 KB) - added by DarkWyrm 7 years ago.
Updated syslog with stack crawl of project window thread and all 4 build threads
PaladinQuadBuild.zip (263.8 KB) - added by DarkWyrm 7 years ago.
A development build of Paladin which is hardwired to use 4 threads to build projects
TestProject.zip (46.9 KB) - added by DarkWyrm 7 years ago.
The test project I've used for testing -- BeVexed

Download all attachments as: .zip

Change History (12)

comment:1 Changed 7 years ago by Ingo Weinhold

When it hangs a stack trace of the child process (the one fork()ed off from Paladin) from the kernel debugger would interesting. Also a thread <id> for that team.

The problem I see in the exec*() family is that memory is allocated from the heap (in __parse_invoke_line() and __flatten_process_args(), which should better not be done, if the fork()ed process was multi-threaded (until we fix fork() that is).

comment:2 Changed 7 years ago by DarkWyrm

I'd be happy to provide one if you could tell me how to get it. :)

comment:3 in reply to:  2 ; Changed 7 years ago by Ingo Weinhold

Replying to darkwyrm:

I'd be happy to provide one if you could tell me how to get it. :)

Enter the kernel debugger, type teams. Find Paladin's child (same name, greater ID) and type sc <id> and thread <id>.

comment:4 in reply to:  3 Changed 7 years ago by Ingo Weinhold

Forgot to mention: A few seconds after leaving the kernel debugger, the KDL session will appear in the syslog.

comment:5 Changed 7 years ago by Ingo Weinhold

The most interesting stack trace would have been the one of the main thread of the new team (348 in this case). Although also having the stack traces of all build threads wouldn't harm either.

Changed 7 years ago by DarkWyrm

Attachment: syslog added

Updated syslog with stack crawl of project window thread and all 4 build threads

comment:6 Changed 7 years ago by Ingo Weinhold

Owner: changed from Nobody to Ingo Weinhold
Status: newassigned

Thanks! It seems that it is not exec*() that hangs, but fork(). I guess this needs some more digging. It would be great, if you could provide a Paladin executable that always uses 4 build threads (I suppose it currently depends on the number of CPUs), and a project to reproduce the problem.

Changed 7 years ago by DarkWyrm

Attachment: PaladinQuadBuild.zip added

A development build of Paladin which is hardwired to use 4 threads to build projects

Changed 7 years ago by DarkWyrm

Attachment: TestProject.zip added

The test project I've used for testing -- BeVexed

comment:7 Changed 7 years ago by DarkWyrm

I've uploaded both the project I've been using for reproducing the bug on my machine and a quick-and-dirty zipfile of Paladin's sources modified to always build with 4 threads. This can be modified by changing the value in line 128 of ProjectBuilder.cpp. A binary is not included in the zipfile because of my penchant for goofing things up, but buildhaikugcc2.sh and buildhaikugcc4.sh will get the job done with a minimum of fuss. E-mail me privately if you run into problems. :)

comment:8 Changed 7 years ago by Ingo Weinhold

Status: assignedin-progress

Thanks! I can reproduce the problem in qemu (ThreadManager also limits the number other threads, BTW).

comment:9 Changed 7 years ago by Ingo Weinhold

Resolution: fixed
Status: in-progressclosed

Fixed in hrev37460.

Note: See TracTickets for help on using tickets.