popen deadlocks Paladin on multicore machines

[darkwyrm]

This only occurs when multithreaded builds are enabled. Here are the steps to reproduce the error:

1) Load a project on an SMP machine. 2) Force a rebuild of the project 3) Run the build.

On my quad core system, perhaps 1 file might get built. Typically not, though. It deadlocks. Killing Paladin from the Deskbar at this point causes a crash with the following backtrace:

0x00356031 in _ZL7do_execPKcPKPcS3_b () from /boot/system/lib/libroot.so
(gdb) bt
#0  0x00356031 in _ZL7do_execPKcPKPcS3_b () from /boot/system/lib/libroot.so
#1  0x0035623f in execl () from /boot/system/lib/libroot.so
#2  0x00318de2 in _IO_proc_open () from /boot/system/lib/libroot.so
#3  0x00319000 in popen () from /boot/system/lib/libroot.so
#4  0x0028a53c in SourceFileC::Compile (this=0x1802c468, info=@0x18069174, 
    options=0x18189de8 "-O0 -I '/boot/home/projects/BeVexed' -I '/boot/home/projects/BeVexed/.' -I '/boot/develop/headers/be' -I '/boot/develop/headers/cpp' -I '/boot/develop/headers/posix' -I '/boot/home/config/include' ")
    at /boot/home/projects/Paladin-related/Paladin/SourceTypeC.cpp:309
#5  0x0024968d in Project::CompileFile (this=0x18069050, file=0x1802c468)
    at /boot/home/projects/Paladin-related/Paladin/Project.cpp:701
#6  0x002872e6 in ProjectBuilder::BuildThread (data=0x18062318)
    at /boot/home/projects/Paladin-related/Paladin/ProjectBuilder.cpp:343
#7  0x002ef4ec in thread_entry () from /boot/system/lib/libroot.so
#8  0x70184fec in ?? ()

[bonefish]

When it hangs a stack trace of the child process (the one fork()ed off from Paladin) from the kernel debugger would interesting. Also a thread <id> for that team.

The problem I see in the exec*() family is that memory is allocated from the heap (in __parse_invoke_line() and __flatten_process_args(), which should better not be done, if the fork()ed process was multi-threaded (until we fix fork() that is).

[darkwyrm]

I'd be happy to provide one if you could tell me how to get it. :)

[bonefish]

Replying to darkwyrm:

I'd be happy to provide one if you could tell me how to get it. :)

Enter the kernel debugger, type teams. Find Paladin's child (same name, greater ID) and type sc <id> and thread <id>.

[bonefish]

Forgot to mention: A few seconds after leaving the kernel debugger, the KDL session will appear in the syslog.

[bonefish]

The most interesting stack trace would have been the one of the main thread of the new team (348 in this case). Although also having the stack traces of all build threads wouldn't harm either.

[darkwyrm]

Updated syslog with stack crawl of project window thread and all 4 build threads

[bonefish]

Thanks! It seems that it is not exec*() that hangs, but fork(). I guess this needs some more digging. It would be great, if you could provide a Paladin executable that always uses 4 build threads (I suppose it currently depends on the number of CPUs), and a project to reproduce the problem.

[darkwyrm]

A development build of Paladin which is hardwired to use 4 threads to build projects

[darkwyrm]

The test project I've used for testing -- BeVexed

[darkwyrm]

I've uploaded both the project I've been using for reproducing the bug on my machine and a quick-and-dirty zipfile of Paladin's sources modified to always build with 4 threads. This can be modified by changing the value in line 128 of ProjectBuilder.cpp. A binary is not included in the zipfile because of my penchant for goofing things up, but buildhaikugcc2.sh and buildhaikugcc4.sh will get the job done with a minimum of fuss. E-mail me privately if you run into problems. :)

[bonefish]

Thanks! I can reproduce the problem in qemu (ThreadManager also limits the number other threads, BTW).

[bonefish]

Fixed in hrev37460.