Memory leak when allocationg and freeing areas

[X512]

This is hrev54290.

Test program attached. When allocationg and freeing areas, used memory continuously increases. Increased item is right side of "System resources and caches" in ProcessController.

Memory is released when team is terminated.

[X512]

Test program.

[waddlesplash]

What does "listarea" on the team after it has finished freeing areas show?

[X512]

listarea then test is started, but not yet allocate areas

[X512]

listarea when test is done, but process is still running

[X512]

listarea when test process is exited

[waddlesplash]

I don't see any obvious problems in the logs, so mmlr's VM changes are the most likely culprit here.

[axeld]

I just ran the test program in hrev54069, and the test application used 4MB when done, but before exit. listarea-post, and listarea-exit don't really show much of a difference, and don't explain at all where the memory has gone to. That sounds like an interesting issue :-)

[mmlr]

It's a feature!

So where the memory goes is the only place we don't separately track: The translation map. The change in hrev54275 is what triggers this, although it doesn't exactly cause it.

What happens with this change is that we move from a linear search from the front of the address space to allocating from the end of the last allocated area. That means even if there's space at the beginning of the address space, we don't use/reuse it but just continue to fill elsewhere. This makes the address space allocations a lot less "compact" and they will spread all over the entire 128TiB of address space we have available. That causes massive amounts of page directory tables, page directories and page tables to be allocated for the individual page mappings. As we never clean them up on unmap, they stay around until the address space and all of it's paging structures eventually are deleted on team exit. Totaling hundreds of megabytes, but not actually leaking anything. You can see them reflected as part of the wired pages count of page_stats, which correspondingly drops on team exit. It is also bounded, it will stop increasing once everything has been mapped at least once, so at somewhere around 256GiB...

So this is actually a general problem that existed before and you would run into it whenever you spread allocations over a large chunk of address space. It is now just easy to trigger by allocating and releasing massive amounts of areas. Specifying a B_BASE_ADDRESS address spec with 0 as the base should restore the previous behavior in this specific case.

The question is how to handle this:

• Make the insertion hinting more clever, or replace it with free range tracking as is already done for the kernel address space, to make address space allocations more compact again
• Release paging structures when they become unused on unmap
• Make the address space smaller

The second solution would be the most general fix that would also work when you for example just waste address space (guarded heap with disabled memory reuse and MADV_FREE, once the supporting changes land). It will obviously come with a performance impact as you'd have to check the fill level of the paging structures on unmap. This could be optimized by keeping track of fill levels or using bitmaps, but when looking at tracking structures for the entire address space, it's again quite some memory that would be needed (4GiB for the page table bitmaps at the leaf level). Obviously not all of it would have to be present at all times.

On the other hand having a more compact address space may be desirable in any case. There's address space fragmentation to consider, although that isn't much of an issue with 48 bits available as it is now. Then it could also be an advantage in caching of the paging structures / TLB when things are closer together, but I haven't actually investigated whether or not this is really the case or if it's relevant at all.

Shrinking the address space would also reduce the issue and the entire 128TiB seems somewhat excessive, but it's obviously not a real solution.