#14831 closed bug (fixed)
Optimize packagefs memory usage
Reported by: | s40in | Owned by: | bonefish |
---|---|---|---|
Priority: | normal | Milestone: | R1/beta6 |
Component: | File Systems/packagefs | Version: | R1/Development |
Keywords: | Cc: | ||
Blocked By: | Blocking: | #10163 | |
Platform: | All |
Description (last modified by )
Haiku Beta1 consumes 262MB
When booted in unpacked version Beta1 consumes only 148MB
Which is almost 120MB less RAM.
See screenshots
Attachments (4)
Change History (37)
by , 6 years ago
Attachment: | Haiku unpacked mem.jpg added |
---|
by , 6 years ago
Attachment: | Haiku hpkg_mem.jpg added |
---|
comment:1 by , 6 years ago
Component: | File Systems → File Systems/packagefs |
---|---|
Description: | modified (diff) |
Keywords: | packagefs hpkg removed |
Owner: | changed from | to
Summary: | Packagefs memory leak → packagefs memory leak |
by , 6 years ago
Attachment: | Haiku hpkg.jpg added |
---|
by , 6 years ago
Attachment: | Haiku unpacked.jpg added |
---|
comment:2 by , 6 years ago
comment:3 by , 6 years ago
Summary: | packagefs memory leak → Optimize packagefs memory usage |
---|
follow-up: 6 comment:4 by , 6 years ago
Same thoughts here. Would be nice to investigate where this memory is going to, though, just to know. I find the difference acceptable in principle, at least.
BTW: How would I build a Haiku version without PM? That sounds interesting to improve the turn-around times when doing kernel development.
comment:5 by , 6 years ago
120MB is bloaty to the point of ridiculous though... even if it isn't a memory leak per se.
I assume this has something to do with how data structures that are needed to implement a single coherent filesystem out of all the packages are implemented... perhaps the packages themselves could restructured so that the data structures that are needed can just be mmaped directly from the hpkgs and leave the rest up to cache instead of actually loading things into memory. If it already does that then it seems there is some other problem...
Memory wasted to packagefs is memory you don't have for cache...it seems to make the installer unbootable on my 512MB system also (need to retest since beta).
I guess any shine through R/W files might cause overhead as well?
comment:6 by , 6 years ago
Replying to axeld:
BTW: How would I build a Haiku version without PM? That sounds interesting to improve the turn-around times when doing kernel development.
IIUC, you need to copy your /system folder to another bfs partition, delete everything in system/packages except for haiku_loader hpkg. I havn't tried it myself, tho.
comment:7 by , 6 years ago
After the O_NOCACHE change, is there any improvement here?
Also, what's the cached vs. used memory? (See ActivityMonitor.) We only really care about more "used" memory, "cached" memory usually can be reused.
comment:8 by , 5 years ago
Blocking: | 10163 added |
---|
comment:11 by , 4 years ago
As of hrev54509 on x86_64, here are the top consumers:
- 75 MiB app_server (one terminal window open)
- 70 MiB package_daemon
- 64 MiB Tracker
- 16 MiB 1x sshd connection
- several 1-2 MiB processes
~ 256 MiB of ram seems like the *absolute* bare minimum. ~ 384 MiB to avoid swapping running minimal applications ~ 512 MiB or more preferred
Haiku could be stripped down to run on < 128 MiB of RAM in theory if we could boot without app_server and tracker.
comment:12 by , 4 years ago
64MB for Tracker sounds very wrong, uses far less here (and elsewhere as seen in screenshots.) Same for app_server.
You also do not seem to account for kernel memory usage here which is far greater than userland in this case. packagefs is the largest consumer there, IIRC a sans-packagefs Haiku boots using 100MB+ less RAM.
comment:13 by , 3 months ago
Following the tweaks in hrev58039, here's the current statistics I see after boot on my main development VM:
address name objsize align usage empty usedobj total flags 0xffffffff82bf7800 pkgfs heap buffers 65536 8 1048576 0 14 16 0 0xffffffff82fe3c50 pkgfs TKAVLTreeNodes 40 8 16146432 0 398139 398142 0 0xffffffff8087de18 pkgfs PackageDirectors 128 8 1728512 0 13069 13082 0 0xffffffff8087dc58 pkgfs PackageNodeAttrs 64 8 11018240 0 169437 169470 0 0xffffffff8087da98 pkgfs PackageFiles 144 8 17113088 0 116966 116984 0 0xffffffff8d211e08 pkgfs UnpackingDirs 208 8 1527808 0 7085 7087 0 0xffffffff8d211c48 pkgfs UnpackingLNodes 144 8 18300928 0 125093 125104 0
That's a total of 63.7 MB. Total system memory usage is 465 MB. I didn't investigate whether there are other significant memory usages in packagefs that aren't in packagefs-specific allocators (the string heap for example; it appears to be using only a few MB though), but at least I don't see anything in the block allocator with a usedobj in the hundreds of thousands.
The first optimization we can do here is to have only UnpackingDirectories be read-write locked instead of all UnpackingNodes, which will save 40+ bytes (on x86_64 at least) per file. I have a WIP change for that.
comment:14 by , 3 months ago
Actually, that "heap buffers" usage looks suspicious. Those buffers should only be used very temporarily for uncompressing heaps. We shouldn't have 14 "used" objects unless there are 7 separate threads reading from packagefs at once, which doesn't appear to be the case at the time I captured this output. There may be a leak, then; but I don't see how the code in question could leak anything...
comment:15 by , 3 months ago
Ah, the usage is probably in the object depot, actually. So that makes sense then, and there's no leak after all.
comment:16 by , 3 months ago
There are indeed some other allocations not accounted for there. One of them is that Directory objects use a hash table for name lookups, which can get large if there are a lot of child nodes. Given that we also have entry caching, a hash table here is probably overkill. We can likely replace it with an AVL tree, which will allow us to also get rid of the linked list and just iterate over the AVL tree instead, saving some memory that way.
comment:17 by , 3 months ago
Another set of changes in hrev58072 to reduce the size of PackageData and PackageNodeAttribute a bit.
comment:18 by , 3 months ago
I made a build of the full system with allocation tracking. It appears after the boot is finished, packagefs is has allocated somewhere around 13MB of memory outside its object_caches (note that this is purely the sizes from calls to malloc() and doesn't account for wasted memory, which is surely significant, too.)
The next largest consumer after the string pool is the Node ID hash table, which is using 2 MB. Then the directory child name hash tables are using around 1.8MB.
comment:20 by , 3 months ago
comment:22 by , 3 months ago
Here's where we're at now:
address name objsize align usage empty usedobj total flags 0xffffffff82bfb800 pkgfs heap buffers 65536 8 1048576 0 14 16 0 0xffffffff82fe8c50 pkgfs TKAVLTreeNodes 40 8 16138240 0 397935 397940 8000000 0xffffffff80cb3e18 pkgfs PackageDirectorys 88 8 1191936 0 13072 13095 8000000 0xffffffff80cb3c58 pkgfs PackageNodeAttrs 48 8 8261632 0 169366 169428 8000000 0xffffffff80cb3a98 pkgfs PackageFiles 96 8 11403264 0 116909 116928 8000000 0xffffffff80cb3558 pkgfs PackageSymlinks 80 8 1032192 0 12586 12600 8000000 0xffffffff8c884c48 pkgfs UnpackingDirs 208 8 1531904 0 7093 7106 8000000 0xffffffff8c884a88 pkgfs UnpackingLNodes 144 8 18288640 0 125017 125020 8000000
So, despite adding a new object_cache for PackageSymlinks, the overall total has still gone down, to 56.2 MB (without PackageSymlinks it's 55.2 MB.) Though it appears my installed files total has also gone down, but not by much (by about ~75).
comment:24 by , 6 weeks ago
rwlocking changes merged in hrev58203, reducing UnpackingLeafNode by 40 bytes.
comment:25 by , 6 weeks ago
Diver did some tests on hrev58204 booting a packaged and then an "unpackaged" Haiku. He reports that the packaged version had 350MB used after the boot finished, and the "unpackaged" 214MB. The total "slabs" usage (across all object caches, not just packagefs ones) was 50,098,176 for the packaged version, and 31,776,768 for "unpackaged", a difference of 17.5 MB.
So, how and where is the rest of the 136 MB difference in usage coming from? Some of it is surely heap allocations larger than the slab caches account for (there's at least a few MB from packagefs in there), but I can't see how it'd be 100+ MB.
comment:26 by , 6 weeks ago
The packagefs "cached data reader" should be using cached pages, so that shouldn't be the difference either (they won't be counted in "used" pages). But something with pages seems to be the only real possibility here, I think...
comment:27 by , 6 weeks ago
packaged boot in ActivityMonitor: Used 369.61 MB, Cached 470.75 MB. "unpackaged" boot: Used 227.83 MB, Cached 209.51 MB.
comment:28 by , 6 weeks ago
On my system (> 400MB used after boot), package_daemon appears to be using around 10MB at most.
comment:29 by , 6 weeks ago
Milestone: | Unscheduled → R1/beta6 |
---|---|
Resolution: | → fixed |
Status: | new → closed |
The problem was a memory leak in the bootloader. After hrev58211, post-boot memory usage is down by over 100 MB on my system. There's still more work to be done in the bootloader here, but at least I think we can consider this fixed.
comment:30 by , 6 weeks ago
For reference, here's the final result of packagefs caches:
0xffffffff82bff800 pkgfs heap buffers 65536 8 1048576 0 12 16 0 0xffffffff82ff5c50 pkgfs TKAVLTreeNodes 40 8 15794176 0 389451 389456 8000000 0xffffffff82ff5010 pkgfs PackageDirs 88 8 1171456 0 12860 12870 8000000 0xffffffff80911e18 pkgfs PackageNodeAttrs 48 8 7970816 0 163416 163464 8000000 0xffffffff80911c58 pkgfs PackageSymlinks 80 8 1036288 0 12643 12650 8000000 0xffffffff80911a98 pkgfs PackageFiles 96 8 10838016 0 111115 111132 8000000 0xffffffff8cf9da88 pkgfs UnpackingDirs 200 8 1425408 0 6953 6960 8000000 0xffffffff8cf9d8c8 pkgfs UnpackingLeafs 96 8 11931648 0 122326 122346 8000000
Note that "UnpackingLeafs" has gone down by about ~2700 files since the last time I checked (regular churn of installation and uninstallation, I suppose).
comment:31 by , 6 weeks ago
Just updated to hrev58228.
Memory usage went down from 350MB to 250MB with 2GB of RAM.
Reducing RAM to 1GB (to compare with original ticket) memory usage decreased to 220MB (42MB less than beta1)
"Unpackaged" decreased from 214MB to 188MB (40MB more than "unpackaged" beta1 which is kinda concerning).
It would be nice to monitor mem usage after every nightly build (using CI) to spot memory usage increase.
comment:32 by , 6 weeks ago
comment:33 by , 6 weeks ago
The output of the KDL "slabs" command from beta1 unpackaged vs. beta5 unpackaged would be interesting. Also "listarea"'s output from both of the same.
I'm not sure that's a memory leak. packagefs seems to use a lot of memory for internal data structures; while this could be optimized, I don't know of any leaks here.