Opened 6 years ago

Closed 3 weeks ago

Last modified 3 weeks ago

#14831 closed bug (fixed)

Optimize packagefs memory usage

Reported by: s40in Owned by: bonefish
Priority: normal Milestone: R1/beta6
Component: File Systems/packagefs Version: R1/Development
Keywords: Cc:
Blocked By: Blocking: #10163
Platform: All

Description (last modified by diver)

Haiku Beta1 consumes 262MB
When booted in unpacked version Beta1 consumes only 148MB
Which is almost 120MB less RAM.
See screenshots

Attachments (4)

Haiku unpacked mem.jpg (110.3 KB ) - added by s40in 6 years ago.
Haiku hpkg_mem.jpg (81.2 KB ) - added by s40in 6 years ago.
Haiku hpkg.jpg (65.4 KB ) - added by s40in 6 years ago.
Haiku unpacked.jpg (69.0 KB ) - added by s40in 6 years ago.

Download all attachments as: .zip

Change History (37)

by s40in, 6 years ago

Attachment: Haiku unpacked mem.jpg added

by s40in, 6 years ago

Attachment: Haiku hpkg_mem.jpg added

comment:1 by diver, 6 years ago

Component: File SystemsFile Systems/packagefs
Description: modified (diff)
Keywords: packagefs hpkg removed
Owner: changed from nobody to bonefish
Summary: Packagefs memory leakpackagefs memory leak

by s40in, 6 years ago

Attachment: Haiku hpkg.jpg added

by s40in, 6 years ago

Attachment: Haiku unpacked.jpg added

comment:2 by waddlesplash, 6 years ago

I'm not sure that's a memory leak. packagefs seems to use a lot of memory for internal data structures; while this could be optimized, I don't know of any leaks here.

comment:3 by diver, 6 years ago

Summary: packagefs memory leakOptimize packagefs memory usage

comment:4 by axeld, 6 years ago

Same thoughts here. Would be nice to investigate where this memory is going to, though, just to know. I find the difference acceptable in principle, at least.

BTW: How would I build a Haiku version without PM? That sounds interesting to improve the turn-around times when doing kernel development.

comment:5 by cb88, 6 years ago

120MB is bloaty to the point of ridiculous though... even if it isn't a memory leak per se.

I assume this has something to do with how data structures that are needed to implement a single coherent filesystem out of all the packages are implemented... perhaps the packages themselves could restructured so that the data structures that are needed can just be mmaped directly from the hpkgs and leave the rest up to cache instead of actually loading things into memory. If it already does that then it seems there is some other problem...

Memory wasted to packagefs is memory you don't have for cache...it seems to make the installer unbootable on my 512MB system also (need to retest since beta).

I guess any shine through R/W files might cause overhead as well?

Last edited 6 years ago by cb88 (previous) (diff)

in reply to:  4 comment:6 by diver, 6 years ago

Replying to axeld:

BTW: How would I build a Haiku version without PM? That sounds interesting to improve the turn-around times when doing kernel development.

IIUC, you need to copy your /system folder to another bfs partition, delete everything in system/packages except for haiku_loader hpkg. I havn't tried it myself, tho.

comment:7 by waddlesplash, 5 years ago

After the O_NOCACHE change, is there any improvement here?

Also, what's the cached vs. used memory? (See ActivityMonitor.) We only really care about more "used" memory, "cached" memory usually can be reused.

comment:8 by diver, 5 years ago

Blocking: 10163 added

comment:9 by diver, 5 years ago

Haiku uses 290MB in hrev53259 right after boot.

comment:10 by diver, 5 years ago

Cached memory is at 430MB.

comment:11 by kallisti5, 4 years ago

As of hrev54509 on x86_64, here are the top consumers:

  • 75 MiB app_server (one terminal window open)
  • 70 MiB package_daemon
  • 64 MiB Tracker
  • 16 MiB 1x sshd connection
  • several 1-2 MiB processes

~ 256 MiB of ram seems like the *absolute* bare minimum. ~ 384 MiB to avoid swapping running minimal applications ~ 512 MiB or more preferred

Haiku could be stripped down to run on < 128 MiB of RAM in theory if we could boot without app_server and tracker.

comment:12 by waddlesplash, 4 years ago

64MB for Tracker sounds very wrong, uses far less here (and elsewhere as seen in screenshots.) Same for app_server.

You also do not seem to account for kernel memory usage here which is far greater than userland in this case. packagefs is the largest consumer there, IIRC a sans-packagefs Haiku boots using 100MB+ less RAM.

comment:13 by waddlesplash, 2 months ago

Following the tweaks in hrev58039, here's the current statistics I see after boot on my main development VM:

           address                   name  objsize    align    usage  empty  usedobj    total    flags
0xffffffff82bf7800     pkgfs heap buffers    65536        8  1048576      0       14       16        0
0xffffffff82fe3c50   pkgfs TKAVLTreeNodes       40        8 16146432      0   398139   398142        0
0xffffffff8087de18 pkgfs PackageDirectors      128        8  1728512      0    13069    13082        0
0xffffffff8087dc58 pkgfs PackageNodeAttrs       64        8 11018240      0   169437   169470        0
0xffffffff8087da98     pkgfs PackageFiles      144        8 17113088      0   116966   116984        0
0xffffffff8d211e08 pkgfs UnpackingDirs         208        8  1527808      0     7085     7087        0
0xffffffff8d211c48 pkgfs UnpackingLNodes       144        8 18300928      0   125093   125104        0

That's a total of 63.7 MB. Total system memory usage is 465 MB. I didn't investigate whether there are other significant memory usages in packagefs that aren't in packagefs-specific allocators (the string heap for example; it appears to be using only a few MB though), but at least I don't see anything in the block allocator with a usedobj in the hundreds of thousands.

The first optimization we can do here is to have only UnpackingDirectories be read-write locked instead of all UnpackingNodes, which will save 40+ bytes (on x86_64 at least) per file. I have a WIP change for that.

comment:14 by waddlesplash, 2 months ago

Actually, that "heap buffers" usage looks suspicious. Those buffers should only be used very temporarily for uncompressing heaps. We shouldn't have 14 "used" objects unless there are 7 separate threads reading from packagefs at once, which doesn't appear to be the case at the time I captured this output. There may be a leak, then; but I don't see how the code in question could leak anything...

comment:15 by waddlesplash, 2 months ago

Ah, the usage is probably in the object depot, actually. So that makes sense then, and there's no leak after all.

comment:16 by waddlesplash, 2 months ago

There are indeed some other allocations not accounted for there. One of them is that Directory objects use a hash table for name lookups, which can get large if there are a lot of child nodes. Given that we also have entry caching, a hash table here is probably overkill. We can likely replace it with an AVL tree, which will allow us to also get rid of the linked list and just iterate over the AVL tree instead, saving some memory that way.

comment:17 by waddlesplash, 2 months ago

Another set of changes in hrev58072 to reduce the size of PackageData and PackageNodeAttribute a bit.

comment:18 by waddlesplash, 2 months ago

I made a build of the full system with allocation tracking. It appears after the boot is finished, packagefs is has allocated somewhere around 13MB of memory outside its object_caches (note that this is purely the sizes from calls to malloc() and doesn't account for wasted memory, which is surely significant, too.)

The next largest consumer after the string pool is the Node ID hash table, which is using 2 MB. Then the directory child name hash tables are using around 1.8MB.

comment:19 by waddlesplash, 2 months ago

Another small savings in StringPool in hrev58073.

comment:20 by waddlesplash, 2 months ago

Three more changes in hrev58080 and hrev58082: giving PackageSymlink an object_cache, reducing the size of PackageNode by 8 bytes, and then by another 8 bytes.

comment:21 by waddlesplash, 2 months ago

Another 16 bytes savings to PackageNode in hrev58083.

comment:22 by waddlesplash, 2 months ago

Here's where we're at now:

           address                   name  objsize    align    usage  empty  usedobj    total    flags
0xffffffff82bfb800     pkgfs heap buffers    65536        8  1048576      0       14       16        0
0xffffffff82fe8c50   pkgfs TKAVLTreeNodes       40        8 16138240      0   397935   397940  8000000
0xffffffff80cb3e18 pkgfs PackageDirectorys      88        8  1191936      0    13072    13095  8000000
0xffffffff80cb3c58 pkgfs PackageNodeAttrs       48        8  8261632      0   169366   169428  8000000
0xffffffff80cb3a98     pkgfs PackageFiles       96        8 11403264      0   116909   116928  8000000
0xffffffff80cb3558  pkgfs PackageSymlinks       80        8  1032192      0    12586    12600  8000000
0xffffffff8c884c48 pkgfs UnpackingDirs         208        8  1531904      0     7093     7106  8000000
0xffffffff8c884a88 pkgfs UnpackingLNodes       144        8 18288640      0   125017   125020  8000000

So, despite adding a new object_cache for PackageSymlinks, the overall total has still gone down, to 56.2 MB (without PackageSymlinks it's 55.2 MB.) Though it appears my installed files total has also gone down, but not by much (by about ~75).

comment:23 by waddlesplash, 2 months ago

Reduced the Unpacking nodes by 8 bytes in hrev58084.

comment:24 by waddlesplash, 4 weeks ago

rwlocking changes merged in hrev58203, reducing UnpackingLeafNode by 40 bytes.

comment:25 by waddlesplash, 3 weeks ago

Diver did some tests on hrev58204 booting a packaged and then an "unpackaged" Haiku. He reports that the packaged version had 350MB used after the boot finished, and the "unpackaged" 214MB. The total "slabs" usage (across all object caches, not just packagefs ones) was 50,098,176 for the packaged version, and 31,776,768 for "unpackaged", a difference of 17.5 MB.

So, how and where is the rest of the 136 MB difference in usage coming from? Some of it is surely heap allocations larger than the slab caches account for (there's at least a few MB from packagefs in there), but I can't see how it'd be 100+ MB.

comment:26 by waddlesplash, 3 weeks ago

The packagefs "cached data reader" should be using cached pages, so that shouldn't be the difference either (they won't be counted in "used" pages). But something with pages seems to be the only real possibility here, I think...

comment:27 by waddlesplash, 3 weeks ago

packaged boot in ActivityMonitor: Used 369.61 MB, Cached 470.75 MB. "unpackaged" boot: Used 227.83 MB, Cached 209.51 MB.

comment:28 by waddlesplash, 3 weeks ago

On my system (> 400MB used after boot), package_daemon appears to be using around 10MB at most.

comment:29 by waddlesplash, 3 weeks ago

Milestone: UnscheduledR1/beta6
Resolution: fixed
Status: newclosed

The problem was a memory leak in the bootloader. After hrev58211, post-boot memory usage is down by over 100 MB on my system. There's still more work to be done in the bootloader here, but at least I think we can consider this fixed.

comment:30 by waddlesplash, 3 weeks ago

For reference, here's the final result of packagefs caches:

0xffffffff82bff800     pkgfs heap buffers    65536        8  1048576      0       12       16        0
0xffffffff82ff5c50   pkgfs TKAVLTreeNodes       40        8 15794176      0   389451   389456  8000000
0xffffffff82ff5010 pkgfs PackageDirs            88        8  1171456      0    12860    12870  8000000
0xffffffff80911e18 pkgfs PackageNodeAttrs       48        8  7970816      0   163416   163464  8000000
0xffffffff80911c58  pkgfs PackageSymlinks       80        8  1036288      0    12643    12650  8000000
0xffffffff80911a98     pkgfs PackageFiles       96        8 10838016      0   111115   111132  8000000
0xffffffff8cf9da88 pkgfs UnpackingDirs         200        8  1425408      0     6953     6960  8000000
0xffffffff8cf9d8c8 pkgfs UnpackingLeafs         96        8 11931648      0   122326   122346  8000000

Note that "UnpackingLeafs" has gone down by about ~2700 files since the last time I checked (regular churn of installation and uninstallation, I suppose).

comment:31 by diver, 3 weeks ago

Just updated to hrev58228.

Memory usage went down from 350MB to 250MB with 2GB of RAM.
Reducing RAM to 1GB (to compare with original ticket) memory usage decreased to 220MB (42MB less than beta1)
"Unpackaged" decreased from 214MB to 188MB (40MB more than "unpackaged" beta1 which is kinda concerning).
It would be nice to monitor mem usage after every nightly build (using CI) to spot memory usage increase.

Last edited 3 weeks ago by diver (previous) (diff)

comment:32 by diver, 3 weeks ago

With 1GB RAM:

beta1 262MB
beta1 (unpackaged)148MB
beta5 (hrev58228)220MB
beta5 (hrev58228, unpackaged)188MB
Last edited 3 weeks ago by diver (previous) (diff)

comment:33 by waddlesplash, 3 weeks ago

The output of the KDL "slabs" command from beta1 unpackaged vs. beta5 unpackaged would be interesting. Also "listarea"'s output from both of the same.

Note: See TracTickets for help on using tickets.