Heap corruption after AreaTest is run

[KapiX]

AreaTest is part of our test suite for Media Kit. [1]

It's fairly simple but has disastrous effect on memory. BApplication tests crash in hoard later. If it is disabled, everything is fine.

Minimal reproducer:

Add AreaTest contents to TBApplicationTester::BApplication1() at the beginning and merge TBApplicationTester::BApplication2() with it so that they are one test case. [2]

It can also be reproduced by running UnitTester with [3] applied (and AreaTest enabled). It should crash in QuitTest or BApplicationTest.

I suspect it's a VM subsystem issue - clone_area is not used a lot in Haiku (if GitHub search is to be believed only accelerants, BBitmaps and Media Kit use it).

I tried to debug, but did not get very far:

the crash happens when hoard is traversed, for example here [4]. chunk->next is a bogus value (in my tests 0x09090808 - this one surrounded by other bogus stuff, 0x22 or 0x2222222 - these look like some memory write didn't go where it should).

[1] ​https://git.haiku-os.org/haiku/tree/src/tests/kits/media/AreaTest.cpp

[2] ​https://git.haiku-os.org/haiku/tree/src/tests/kits/app/bapplication/BApplicationTester.cpp#n55

[3] ​https://review.haiku-os.org/c/haiku/+/465

[4] ​https://git.haiku-os.org/haiku/tree/src/system/libroot/posix/malloc/arch-specific.cpp#n159

[KapiX]

BApplication tests use pipes a lot and most of the crashes happen there (in popen for example).

[CodeforEvolution]

Since we’re talking about memory and heap corruption anyway, I’d like to bring up that our current memory allocator, hoard, has been written up in articles as inefficient with memory compared to competitors such as jemalloc. Maybe a future todo could be replacing hoard? Maybe even a GSoC task depending?

[korli]

Replying to CodeforEvolution:

Since we’re talking about memory and heap corruption anyway, I’d like to bring up that our current memory allocator, hoard, has been written up in articles as inefficient with memory

It seems to me like you didn't search too long before posting such a comment. #13554

[CodeforEvolution]

Ah, sorry about that, didn't see that ticket.

[waddlesplash]

KapiX, now that we've switched to rpmalloc, can you look into this one again?

[KapiX]

I have run the entire test suite and it still crashes in malloc (NetworkUrlTest), but it's way farther and disabling AreaTest doesn't seem to affect it. I guess there is a heap corruption somewhere else in the tests.

[waddlesplash]

It would still be nice to see if there actually is some clone area bug that AreaTest was somehow hitting. But maybe that's impossible.