Optimize packagefs memory usage

[s40in]

Haiku Beta1 consumes 262MB
When booted in unpacked version Beta1 consumes only 148MB
Which is almost 120MB less RAM.
​See screenshots

[waddlesplash]

I'm not sure that's a memory leak. packagefs seems to use a lot of memory for internal data structures; while this could be optimized, I don't know of any leaks here.

[axeld]

Same thoughts here. Would be nice to investigate where this memory is going to, though, just to know. I find the difference acceptable in principle, at least.

BTW: How would I build a Haiku version without PM? That sounds interesting to improve the turn-around times when doing kernel development.

[cb88]

120MB is bloaty to the point of ridiculous though... even if it isn't a memory leak per se.

I assume this has something to do with how data structures that are needed to implement a single coherent filesystem out of all the packages are implemented... perhaps the packages themselves could restructured so that the data structures that are needed can just be mmaped directly from the hpkgs and leave the rest up to cache instead of actually loading things into memory. If it already does that then it seems there is some other problem...

Memory wasted to packagefs is memory you don't have for cache...it seems to make the installer unbootable on my 512MB system also (need to retest since beta).

I guess any shine through R/W files might cause overhead as well?

[diver]

Replying to axeld:

BTW: How would I build a Haiku version without PM? That sounds interesting to improve the turn-around times when doing kernel development.

IIUC, you need to copy your /system folder to another bfs partition, delete everything in system/packages except for haiku_loader hpkg. I havn't tried it myself, tho.

[waddlesplash]

After the O_NOCACHE change, is there any improvement here?

Also, what's the cached vs. used memory? (See ActivityMonitor.) We only really care about more "used" memory, "cached" memory usually can be reused.

[diver]

Haiku uses 290MB in hrev53259 right after boot.

[diver]

Cached memory is at 430MB.

[kallisti5]

As of hrev54509 on x86_64, here are the top consumers:

• 75 MiB app_server (one terminal window open)
• 70 MiB package_daemon
• 64 MiB Tracker
• 16 MiB 1x sshd connection
• several 1-2 MiB processes

~ 256 MiB of ram seems like the *absolute* bare minimum. ~ 384 MiB to avoid swapping running minimal applications ~ 512 MiB or more preferred

Haiku could be stripped down to run on < 128 MiB of RAM in theory if we could boot without app_server and tracker.

[waddlesplash]

64MB for Tracker sounds very wrong, uses far less here (and elsewhere as seen in screenshots.) Same for app_server.

You also do not seem to account for kernel memory usage here which is far greater than userland in this case. packagefs is the largest consumer there, IIRC a sans-packagefs Haiku boots using 100MB+ less RAM.

[waddlesplash]

Following the tweaks in hrev58039, here's the current statistics I see after boot on my main development VM:

           address                   name  objsize    align    usage  empty  usedobj    total    flags
0xffffffff82bf7800     pkgfs heap buffers    65536        8  1048576      0       14       16        0
0xffffffff82fe3c50   pkgfs TKAVLTreeNodes       40        8 16146432      0   398139   398142        0
0xffffffff8087de18 pkgfs PackageDirectors      128        8  1728512      0    13069    13082        0
0xffffffff8087dc58 pkgfs PackageNodeAttrs       64        8 11018240      0   169437   169470        0
0xffffffff8087da98     pkgfs PackageFiles      144        8 17113088      0   116966   116984        0
0xffffffff8d211e08 pkgfs UnpackingDirs         208        8  1527808      0     7085     7087        0
0xffffffff8d211c48 pkgfs UnpackingLNodes       144        8 18300928      0   125093   125104        0

That's a total of 63.7 MB. Total system memory usage is 465 MB. I didn't investigate whether there are other significant memory usages in packagefs that aren't in packagefs-specific allocators (the string heap for example; it appears to be using only a few MB though), but at least I don't see anything in the block allocator with a usedobj in the hundreds of thousands.

The first optimization we can do here is to have only UnpackingDirectories be read-write locked instead of all UnpackingNodes, which will save 40+ bytes (on x86_64 at least) per file. I have a WIP change for that.

[waddlesplash]

Actually, that "heap buffers" usage looks suspicious. Those buffers should only be used very temporarily for uncompressing heaps. We shouldn't have 14 "used" objects unless there are 7 separate threads reading from packagefs at once, which doesn't appear to be the case at the time I captured this output. There may be a leak, then; but I don't see how the code in question could leak anything...

[waddlesplash]

Ah, the usage is probably in the object depot, actually. So that makes sense then, and there's no leak after all.

[waddlesplash]

There are indeed some other allocations not accounted for there. One of them is that Directory objects use a hash table for name lookups, which can get large if there are a lot of child nodes. Given that we also have entry caching, a hash table here is probably overkill. We can likely replace it with an AVL tree, which will allow us to also get rid of the linked list and just iterate over the AVL tree instead, saving some memory that way.

[waddlesplash]

Another set of changes in hrev58072 to reduce the size of PackageData and PackageNodeAttribute a bit.

[waddlesplash]

I made a build of the full system with allocation tracking. It appears after the boot is finished, packagefs is has allocated somewhere around 13MB of memory outside its object_caches (note that this is purely the sizes from calls to malloc() and doesn't account for wasted memory, which is surely significant, too.)

The next largest consumer after the string pool is the Node ID hash table, which is using 2 MB. Then the directory child name hash tables are using around 1.8MB.

[waddlesplash]

Another small savings in StringPool in hrev58073.

[waddlesplash]

Three more changes in hrev58080 and hrev58082: giving PackageSymlink an object_cache, reducing the size of PackageNode by 8 bytes, and then by another 8 bytes.

[waddlesplash]

Another 16 bytes savings to PackageNode in hrev58083.

[waddlesplash]

Here's where we're at now:

           address                   name  objsize    align    usage  empty  usedobj    total    flags
0xffffffff82bfb800     pkgfs heap buffers    65536        8  1048576      0       14       16        0
0xffffffff82fe8c50   pkgfs TKAVLTreeNodes       40        8 16138240      0   397935   397940  8000000
0xffffffff80cb3e18 pkgfs PackageDirectorys      88        8  1191936      0    13072    13095  8000000
0xffffffff80cb3c58 pkgfs PackageNodeAttrs       48        8  8261632      0   169366   169428  8000000
0xffffffff80cb3a98     pkgfs PackageFiles       96        8 11403264      0   116909   116928  8000000
0xffffffff80cb3558  pkgfs PackageSymlinks       80        8  1032192      0    12586    12600  8000000
0xffffffff8c884c48 pkgfs UnpackingDirs         208        8  1531904      0     7093     7106  8000000
0xffffffff8c884a88 pkgfs UnpackingLNodes       144        8 18288640      0   125017   125020  8000000

So, despite adding a new object_cache for PackageSymlinks, the overall total has still gone down, to 56.2 MB (without PackageSymlinks it's 55.2 MB.) Though it appears my installed files total has also gone down, but not by much (by about ~75).

[waddlesplash]

Reduced the Unpacking nodes by 8 bytes in hrev58084.

[waddlesplash]

rwlocking changes merged in hrev58203, reducing UnpackingLeafNode by 40 bytes.

[waddlesplash]

Diver did some tests on hrev58204 booting a packaged and then an "unpackaged" Haiku. He reports that the packaged version had 350MB used after the boot finished, and the "unpackaged" 214MB. The total "slabs" usage (across all object caches, not just packagefs ones) was 50,098,176 for the packaged version, and 31,776,768 for "unpackaged", a difference of 17.5 MB.

So, how and where is the rest of the 136 MB difference in usage coming from? Some of it is surely heap allocations larger than the slab caches account for (there's at least a few MB from packagefs in there), but I can't see how it'd be 100+ MB.

[waddlesplash]

The packagefs "cached data reader" should be using cached pages, so that shouldn't be the difference either (they won't be counted in "used" pages). But something with pages seems to be the only real possibility here, I think...

[waddlesplash]

packaged boot in ActivityMonitor: Used 369.61 MB, Cached 470.75 MB. "unpackaged" boot: Used 227.83 MB, Cached 209.51 MB.

[waddlesplash]

On my system (> 400MB used after boot), package_daemon appears to be using around 10MB at most.

[waddlesplash]

The problem was a memory leak in the bootloader. After hrev58211, post-boot memory usage is down by over 100 MB on my system. There's still more work to be done in the bootloader here, but at least I think we can consider this fixed.

[waddlesplash]

For reference, here's the final result of packagefs caches:

0xffffffff82bff800     pkgfs heap buffers    65536        8  1048576      0       12       16        0
0xffffffff82ff5c50   pkgfs TKAVLTreeNodes       40        8 15794176      0   389451   389456  8000000
0xffffffff82ff5010 pkgfs PackageDirs            88        8  1171456      0    12860    12870  8000000
0xffffffff80911e18 pkgfs PackageNodeAttrs       48        8  7970816      0   163416   163464  8000000
0xffffffff80911c58  pkgfs PackageSymlinks       80        8  1036288      0    12643    12650  8000000
0xffffffff80911a98     pkgfs PackageFiles       96        8 10838016      0   111115   111132  8000000
0xffffffff8cf9da88 pkgfs UnpackingDirs         200        8  1425408      0     6953     6960  8000000
0xffffffff8cf9d8c8 pkgfs UnpackingLeafs         96        8 11931648      0   122326   122346  8000000

Note that "UnpackingLeafs" has gone down by about ~2700 files since the last time I checked (regular churn of installation and uninstallation, I suppose).

[diver]

Just updated to hrev58228.

Memory usage went down from 350MB to 250MB with 2GB of RAM.
Reducing RAM to 1GB (to compare with original ticket) memory usage decreased to 220MB (42MB less than beta1)
"Unpackaged" decreased from 214MB to 188MB (40MB more than "unpackaged" beta1 which is kinda concerning).
It would be nice to monitor mem usage after every nightly build (using CI) to spot memory usage increase.

[diver]

With 1GB RAM:

beta1	beta1 unpackaged
262MB	148MB
beta5 (hrev58228)	beta5 (hrev58228, unpackaged)
220MB	188MB

[waddlesplash]

The output of the KDL "slabs" command from beta1 unpackaged vs. beta5 unpackaged would be interesting. Also "listarea"'s output from both of the same.