MMAP: process address mapping uses all avail memory due to no ENOMEM result

[cocobean]

Open POSIX Test Suite fails MMAP 24-1 test. Does not provided ENOMEM result when available or free memory resource(s) is near exhaustion (i.e. insufficient room in the address space for continued memory mapping).

The test will hold used memory resources even if the thread is killed. Tested on Haiku R1B3 x86_64 (+56).

  The mmap() function shall fail if:
  [ENOMEM] MAP_FIXED was specified, and the range [addr,addr+len)
  exceeds that allowed for the address space of a process;
  or, if MAP_FIXED was not specified and
  there is insufficient room in the address space to effect the mapping.
 
  Test Steps:
  1. In a very long loop, keep mapping a shared memory object,
     until there this insufficient room in the address space;
  2. Should get ENOMEM.

Ref: ​​https://github.com/haiku/open_posix_testsuite/blob/master/conformance/interfaces/mmap/24-1.c