Memory pressure causes high latency

[bbjimmy]

Performing a jam on the haiku trunk with 479mb available ram and 997mb Virtual Memory swap file causes extreem latency issues after about 10 minutes of operation. Stopping the jsm job will not ease the latency issues. The activity monitor shows about 20% used memory, but the rest is devoted to cashed memory and I beleive the swap file is being constantly accessed ( constant disk activity, and remaining above 50% utilized.) This state will continue untill the system is re-booted. then the used memory will remain around 20%, but the cashed memory will not use the rest of available memory and the swap space remains below 20% usage.

This is 100% repeatable on hrev35788 and recent earlier revisions.

Increasing the swap file space makes the problem worse, not better.

decreasing the swap file space solves the issue, but leves not enough memory to jam the tree.

I believe Haiku is attempting to use virtual memory for cached memory and making the system unusable by constantly accessing the swap file.

[bonefish]

Failed to reproduce with 512 MB RAM on my Core i7, hrev35799. The build just completed normally after 34 min.

Please provide more information from within the kernel debugger. The output of "page_stats" and "avail" commands would be of interest.

[bbjimmy]

How do I do this?

[bonefish]

Replying to bbjimmy:

How do I do this?

• Enter the kernel debugger via Alt-SysReq-D. You will be greeted with a prompt that allows you to enter commands.
• Execute "page_stats".
• Execute "avail".
• The output of the "slabs" command might be interesting, too. It will be longer and might overwrite previous output on screen.

If you have a serial port and a second machine you can connect it to, you can capture the output this way (configuration: 115200 8N1). Alternatively take a picture of the output.

[bbjimmy]

image of debugger output

[bbjimmy]

The image is from before the system became completely unresponsive. A short while later after continuing, the mouse cursor froze and the screen stopped updating.

[bbjimmy]

debug info after the mouse froze

[bonefish]

The "page_stats" output confirms that the system is actively paging. All pages (save for a small system reserve) are in use, none are cached. "avail" basically agrees. Someone has allocated lots of memory and from the looks of it it is some kernel component. The output of "slabs" would be helpful as well as the output of "areas 1".

[bbjimmy]

slabs info

[bbjimmy]

slab info again -- better image

[bbjimmy]

slabs information added

[bonefish]

Something seems to be wrong with the block cache. It eats up 380 MB. Handing the ticket over to Axel.

[bbjimmy]

areas 1 output: ​http://coquilletkd.com/bbjimmy/areas1.zip

[axeld]

The block cache itself is unlikely to be the problem, though. Anyway, can you add the following debug output when this happens:

• low_resource
• block_caches
• And then "block_cache <address>" with the addresses you got from the previous command? (the output of block_caches itself isn't interesting)

Thanks!

[bbjimmy]

low resources and block cache info

[bbjimmy]

• added

low resources and block cache info

[bonefish]

So the block caches think they don't have any blocks anymore while the "block cache buffers" object caches still has lots of allocated objects (DSCN1482.JPG). Interestingly the "cached blocks" object cache is empty, too. Also interesting: The slabs listing doesn't start with "block cache: 16", which means the low resource manager is currently working on the object caches. More precisely, it doesn't seem to be done with "cached blocks" yet.

bbjimmy, can you get a stack trace of the "low resource manager" thread please. I.e. check with "threads 1" which ID it has and then run "sc <id>".

[axeld]

While I looked over it several times already, I wouldn't rule out a memory leak in the block cache. I also have seen a "cache still has partial slabs" panic on unmounting BFS recently which might hint into that direction, too.

[bbjimmy]

low resource manager info

[axeld]

Since it isn't running this time, it was probably just a coincidence it did before. So I would assume there has to be a leak in the block cache.

Thanks for being so helpful, bbjimmy, it's really appreciated!

[bonefish]

Replying to axeld:

While I looked over it several times already, I wouldn't rule out a memory leak in the block cache. I also have seen a "cache still has partial slabs" panic on unmounting BFS recently which might hint into that direction, too.

I can reproduce this panic on unmounting a ReiserFS volume after a larger recursive grep (with limited RAM). I've already verified that all block_cache_get() calls are balanced by block_cache_put()s. Apparently an issue in the low resource handler of the block cache or the slab. Will have a look.

[bonefish]

Should be fixed in hrev35918. Please reopen, if you can still reproduce the problem.

[bbjimmy]

Imdeed this seems to have fixed the issue. The block cache is staying about mb and not continually growing during the jam.