Optimize packagefs memory usage

[s40in]

Haiku Beta1 consumes 262MB
When booted in unpacked version Beta1 consumes only 148MB
Which is almost 120MB less RAM.
​See screenshots

[waddlesplash]

I'm not sure that's a memory leak. packagefs seems to use a lot of memory for internal data structures; while this could be optimized, I don't know of any leaks here.

[axeld]

Same thoughts here. Would be nice to investigate where this memory is going to, though, just to know. I find the difference acceptable in principle, at least.

BTW: How would I build a Haiku version without PM? That sounds interesting to improve the turn-around times when doing kernel development.

[cb88]

120MB is bloaty to the point of ridiculous though... even if it isn't a memory leak per se.

I assume this has something to do with how data structures that are needed to implement a single coherent filesystem out of all the packages are implemented... perhaps the packages themselves could restructured so that the data structures that are needed can just be mmaped directly from the hpkgs and leave the rest up to cache instead of actually loading things into memory. If it already does that then it seems there is some other problem...

I guess any shine through R/W files might cause overhead as well?

[diver]

Replying to axeld:

BTW: How would I build a Haiku version without PM? That sounds interesting to improve the turn-around times when doing kernel development.

IIUC, you need to copy your /system folder to another bfs partition, delete everything in system/packages except for haiku_loader hpkg. I havn't tried it myself, tho.

[waddlesplash]

After the O_NOCACHE change, is there any improvement here?

Also, what's the cached vs. used memory? (See ActivityMonitor.) We only really care about more "used" memory, "cached" memory usually can be reused.

[diver]

Haiku uses 290MB in hrev53259 right after boot.

[diver]

Cached memory is at 430MB.

[kallisti5]

As of hrev54509 on x86_64, here are the top consumers:

• 75 MiB app_server (one terminal window open)
• 70 MiB package_daemon
• 64 MiB Tracker
• 16 MiB 1x sshd connection
• several 1-2 MiB processes

~ 256 MiB of ram seems like the *absolute* bare minimum. ~ 384 MiB to avoid swapping running minimal applications ~ 512 MiB or more preferred

Haiku could be stripped down to run on < 128 MiB of RAM in theory if we could boot without app_server and tracker.

[waddlesplash]

64MB for Tracker sounds very wrong, uses far less here (and elsewhere as seen in screenshots.) Same for app_server.

You also do not seem to account for kernel memory usage here which is far greater than userland in this case. packagefs is the largest consumer there, IIRC a sans-packagefs Haiku boots using 100MB+ less RAM.