NULL pointer access inside fork()

[bhaible]

While running a configure script in tight memory situation, I got an application crash. Clicking on the "Debug" button of the error dialog, I was presented a gdb session the shell that was executing the configure script. At the top, function fork().

To reproduce:

0. Start Haiku in QEMU with 256 MB RAM allocated to it,

1. Open a Terminal.

2. $ wget ​http://www.haible.de/bruno/gnu/testdir3.tar.gz

3. $ tar xvfz testdir3.tar.gz

4. $ cd testdir3

5. $ ./configure --host=i586-pc-haiku --build=i586-pc-haiku

--prefix=/boot/home/config CPPFLAGS="-Wall" 2>&1 | tee log1

6. Wait a couple of hours.

[axeld]

I would guess that's not too surprising - if it runs out of memory there is little it can do. But I keep it open to have a deeper look at it first.

[bonefish]

The general memory tightness is apparently caused by a kernel memory leak (cf. #3128). Generally fork() should report ENOMEM instead of crashing, but if the virtual memory is really exhausted, touching previously unused stack will definitely crash programs, since the stacks are over-committing. And there's little else we can do in such a case. The syslog (/var/log/syslog) should help to verify whether the page fault happened on the stack.

[axeld]

We could let fork() reserve one or two stack pages for the new team, so that it could fail with ENOMEM in most of these cases as well.

[bonefish]

Replying to axeld:

We could let fork() reserve one or two stack pages for the new team, so that it could fail with ENOMEM in most of these cases as well.

That doesn't make much sense. fork() is not supposed to return an error code in the child. Crashing really is the only option I'm afraid.

[axeld]

Replying to bonefish:

That doesn't make much sense. fork() is not supposed to return an error code in the child. Crashing really is the only option I'm afraid.

Who said it would do that? The areas of the new team are allocated in the parent, so we can also commit memory there for the child; if that fails, we can let fork() fail which would cause it to return ENOMEM in the parent process, before the child has been started.

[waddlesplash]

This ticket is 16 years old, there's been a lot of changes since then; I have routinely seen "fork: Out of memory" errors appear in Terminal, so clearly fork() is capable of returning ENOMEM properly now. Any issues which remain (if any) should be in another ticket.