"vfork: Invalid Argument" while executing processes that involve large amounts of fork/exec

[anevilyak]

Since the ASLR changes, attempting a jam -qj8 build of Haiku under Haiku itself frequently fails with the above error message. Rerunning the command works for a while longer, then eventually fails again the same way. Nothing of interest shows up in syslog though, and I've been unable to reproduce it while running with strace so I assume a race of some form is involved. Hardware here is a quad core i7 w/ hyperthreading and 8GB of RAM, ergo PAE is active.

[pdziepak]

I have a real problem reproducing that error (only one out of several builds failed). Hopefully fixed in hrev45550. Please confirm.

[siarzhuk]

Replying to pdziepak:

I have a real problem reproducing that error (only one out of several builds failed). Hopefully fixed in hrev45550. Please confirm.

Unfortunately the problem is still here on hrev45550. Clean build of haiku sources from scratch on i7 observes this issue 3 times. :-(

[hamish]

Since the ASLR changes, mmap will map over existing areas (particularly those at the top of the user address space -- commpage and thread stacks) if you request a large enough mapping. If the area insertion logic is broken, that could be the cause of both these issues.

You can reproduce this by mapping a few large anonymous areas (500MB+).

[hamish]

(In #9752) I also get a crash when using the server VM, though with a different error message to yours. Regardless, I suspect this is caused by #9686. You can use the client VM until that gets fixed: java -client <args>.

[pdziepak]

Thanks for this information. Hopefully, hrev45704 fixes both this bug and #9752.

[siarzhuk]

Unfortunately it is still observed on hrev45711. :-(

PS: building Haiku on Haiku platform with -j 8 is the way to provoke this issue.

[bonefish]

Having a closer look.

[bonefish]

Fixed in hrev45847.