Context Navigation

#7976 assigned enhancement

Improve memory exhaustion process

Reported by:	kallisti5	Owned by:	nobody
Priority:	normal	Milestone:	R1.1
Component:	System/Kernel	Version:	R1/Development
Keywords:		Cc:	ttcoder
Blocked By:		Blocking:
Platform:	All

Description

This may be invalid, if so feel free to mark it as such.

During a critical physical memory exhaustion, it looks as though we don't take any steps to mitigate the situation such as killing top memory consumers etc.

While the argument that killing processes may lose data may exist... if the system is seconds away from crashing then killing one top consumer may be better then losing everything.

Here is an example of everything running a-muck in VirtualBox when 512 MB memory is fully consumed momentarily...

KERN: slab memory manager: created area 0x8a801000 (4927)
KERN: slab memory manager: created area 0x8b001000 (4938)
KERN: low resource memory: normal -> note
KERN: low resource memory: note -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> note
KERN: low resource memory: note -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> note
KERN: low resource memory: note -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: 0x840157b0->VMAnonymousCache::_Commit(423034880): Failed to reserve 131072 bytes of RAM
KERN: vm_soft_fault: va 0x0 not covered by area in address space
KERN: vm_page_fault: vm_soft_fault returned error 'Bad address' on fault at 0x4, ip 0x20c325, write 1, user 1, thread 0x101
KERN: vm_page_fault: thread "jam" (257) in team "jam" (257) tried to write address 0x4, ip 0x20c325 ("jam_seg0ro" +0xc325)
KERN: low resource memory: critical -> warning
KERN: debug_server: Thread 257 entered the debugger: Segment violation
KERN: low resource memory: warning -> critical
KERN: 0x80fe75b0->VMAnonymousCache::_Commit(122880): Failed to reserve 122880 bytes of RAM
KERN: 0x80fe75b0->VMAnonymousCache::_Commit(98304): Failed to reserve 98304 bytes of RAM
KERN: 0x838ca008->VMAnonymousCache::_Commit(393216): Failed to reserve 131072 bytes of RAM
KERN: vm_soft_fault: va 0x2000 not covered by area in address space
KERN: vm_page_fault: vm_soft_fault returned error 'Bad address' on fault at 0x2000, ip 0x2b0fee, write 1, user 1, thread 0x119
KERN: vm_page_fault: thread "team 257 handler" (281) in team "debug_server" (62) tried to write address 0x2000, ip 0x2b0fee ("libroot.so_seg0ro" +0xa7fee)
KERN: 0x80fe75b0->VMAnonymousCache::Fault(): Failed to reserve 4096 bytes of RAM.
KERN: vm_page_fault: vm_soft_fault returned error 'Out of memory' on fault at 0x700c0f54, ip 0x278a9d, write 1, user 1, thread 0x11b
KERN: vm_page_fault: thread "team 62 handler" (283) in team "debug_server" (62) tried to write address 0x700c0f54, ip 0x278a9d ("libroot.so_seg0ro" +0x6fa9d)
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: 0x840097a8->VMAnonymousCache::_Commit(2752512): Failed to reserve 131072 bytes of RAM
KERN: vm_soft_fault: va 0x2000 not covered by area in address space
KERN: vm_page_fault: vm_soft_fault returned error 'Bad address' on fault at 0x2000, ip 0x805fee, write 1, user 1, thread 0xe5
KERN: vm_page_fault: thread "w>Terminal 1: haiku: jam" (229) in team "Terminal" (221) tried to write address 0x2000, ip 0x805fee ("libroot.so_seg0ro" +0xa7fee)
KERN: vm_soft_fault: va 0x4000 not covered by area in address space
KERN: vm_page_fault: vm_soft_fault returned error 'Bad address' on fault at 0x4000, ip 0x2b0fee, write 1, user 1, thread 0x44
KERN: vm_page_fault: thread "kernel listener" (68) in team "debug_server" (62) tried to write address 0x4000, ip 0x2b0fee ("libroot.so_seg0ro" +0xa7fee)
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: 0x83c4c6e8->VMAnonymousCache::_Commit(28672): Failed to reserve 28672 bytes of RAM
KERN: runtime_loader: /boot/system/lib/libroot.so: Could not map image: Out of memory
KERN: low resource memory: critical -> warning
KERN: 0x83c25dc0->VMAnonymousCache::_Commit(28672): Failed to reserve 28672 bytes of RAM
KERN: runtime_loader: /boot/system/lib/libroot.so: Could not map image: Out of memory
KERN: low resource memory: warning -> critical
KERN: 0x83d19850->VMAnonymousCache::_Commit(65536): Failed to reserve 65536 bytes of RAM
KERN: 0x83d19350->VMAnonymousCache::_Commit(65536): Failed to reserve 65536 bytes of RAM
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: 0x83d19850->VMAnonymousCache::Fault(): Failed to reserve 4096 bytes of RAM.
KERN: vm_page_fault: vm_soft_fault returned error 'Out of memory' on fault at 0x780324c4, ip 0x631ec8, write 1, user 1, thread 0x13c
KERN: vm_page_fault: thread "w>TrackerWindow" (316) in team "Tracker" (115) tried to write address 0x780324c4, ip 0x631ec8 ("libroot.so_seg0ro" +0x2cec8)
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: 0x83d197b0->VMAnonymousCache::_Commit(8192): Failed to reserve 8192 bytes of RAM
Last message repeated 1 time
KERN: low resource memory: warning -> normal

At this point the system was unable to launch any further applications. (see screenshot attached)

Attachments (1)

Screenshot-1.png (97.8 KB ) - added by kallisti5 13 years ago.

Download all attachments as: .zip

Change History (11)

by kallisti5, 13 years ago

Attachment:	Screenshot-1.png added

comment:1 by kallisti5, 13 years ago

code:

https://dev.haiku-os.org/browser/haiku/trunk/src/system/kernel/low_resource_manager.cpp

comment:2 by axeld, 13 years ago

If you ran out of memory, it's actually sane to not start new processes in favor of killing some random team. Besides yet, yes, we don't take such drastic measurements yet, and our low memory handling could surely be further improved.

I'm not sure it's a good idea to randomly kill teams (or rather, I definitely know it's a bad idea, it's just the question of what alternatives you have). One way to deal with things like these is to notify the userland from the problem, and let it deal by taking down not so important services (and also notifying the user) -- unless you currently record something, this could be the media server, for example. Or player only apps like MediaPlayer or ShowImage.

Once we have proper session management in place, a good alternative would be to just quit non-used applications that could be relaunched as they were when needed or possible.

comment:3 by kallisti5, 13 years ago

Those are good points.

If expected behavior is to kill the process attempting to malloc more memory then whats available... shouldn't the system remain stable?

In the example above, It looks as though the failed malloc was incompletely subtracted from available memory. I could no longer open new processes... even though the memory usage is low as per the memory bar in the top right side of the screenshot.

I also noticed jam was still running, even though the terminal was locked up. Once I forcibly killed jam (which didn't effect free memory) the system still wouldn't spawn new processes.

comment:4 by axeld, 7 years ago

Owner:	changed from axeld to nobody
Status:	new → assigned

comment:5 by pulkomandy, 4 years ago

What I can see in the log is:

Jam crashed on a NULL pointer dereference. That's a bug in Jam, it didn't bother to check if malloc actually returned some memory
Then, we try to handle this in the debug_server. We also allocate memory for that. This also fails, and debug_server also crashes.
Finally, Tracker crashes as well.

I don't think there is much to be done on the system side. Killing random apps is always a bad idea, better let the user make the right decisions on what to close. However, we should fix Tracker, debug_server and jam to check for allocation failures, and not crash, but cleanly stop what they are doing instead (jam could just exit, debug_server could just kill the crashed app without trying to show the debug alert, and Tracker stop whatever it was doing).

One thing we may improve on is user feedback. If the memory is completely filled, there may not even be enough memory left for showing an error alert, since that also needs memory (spawning two threads in app_server and the app doing it, at least). If even that can't be done, it iwll appear as if the system is just not doing anything when trying to interact with it.

comment:6 by ttcoder, 4 years ago

Cc:	ttcoder added

comment:7 by diver, 4 years ago

IIRC in macOS and SerenityOS there is such a feature as purgeable memory which I think is pretty cool, take a look https://www.youtube.com/watch?v=9l0nWEUpg7s

comment:8 by diver, 4 years ago

Another video about the same topic https://www.youtube.com/watch?v=GX1VqoZWCjA

comment:9 by pulkomandy, 4 years ago

Is that different from what our low resource manager does? Flushing disk caches and the like?

In macOS it appears to be an on-disk thing, apparently a kind of hidden trashcan?

comment:10 by pulkomandy, 4 years ago

Milestone:	R1 → R1.1

Note: See TracTickets for help on using tickets.

Download in other formats: