Opened 8 years ago

Last modified 3 years ago

#7976 assigned enhancement

Improve memory exhaustion process

Reported by: kallisti5 Owned by: nobody
Priority: normal Milestone: R1
Component: System/Kernel Version: R1/Development
Keywords: Cc:
Blocked By: Blocking:
Has a Patch: no Platform: All

Description

This may be invalid, if so feel free to mark it as such.

During a critical physical memory exhaustion, it looks as though we don't take any steps to mitigate the situation such as killing top memory consumers etc.

While the argument that killing processes may lose data may exist... if the system is seconds away from crashing then killing one top consumer may be better then losing everything.

Here is an example of everything running a-muck in VirtualBox when 512 MB memory is fully consumed momentarily...

KERN: slab memory manager: created area 0x8a801000 (4927)
KERN: slab memory manager: created area 0x8b001000 (4938)
KERN: low resource memory: normal -> note
KERN: low resource memory: note -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> note
KERN: low resource memory: note -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> note
KERN: low resource memory: note -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: 0x840157b0->VMAnonymousCache::_Commit(423034880): Failed to reserve 131072 bytes of RAM
KERN: vm_soft_fault: va 0x0 not covered by area in address space
KERN: vm_page_fault: vm_soft_fault returned error 'Bad address' on fault at 0x4, ip 0x20c325, write 1, user 1, thread 0x101
KERN: vm_page_fault: thread "jam" (257) in team "jam" (257) tried to write address 0x4, ip 0x20c325 ("jam_seg0ro" +0xc325)
KERN: low resource memory: critical -> warning
KERN: debug_server: Thread 257 entered the debugger: Segment violation
KERN: low resource memory: warning -> critical
KERN: 0x80fe75b0->VMAnonymousCache::_Commit(122880): Failed to reserve 122880 bytes of RAM
KERN: 0x80fe75b0->VMAnonymousCache::_Commit(98304): Failed to reserve 98304 bytes of RAM
KERN: 0x838ca008->VMAnonymousCache::_Commit(393216): Failed to reserve 131072 bytes of RAM
KERN: vm_soft_fault: va 0x2000 not covered by area in address space
KERN: vm_page_fault: vm_soft_fault returned error 'Bad address' on fault at 0x2000, ip 0x2b0fee, write 1, user 1, thread 0x119
KERN: vm_page_fault: thread "team 257 handler" (281) in team "debug_server" (62) tried to write address 0x2000, ip 0x2b0fee ("libroot.so_seg0ro" +0xa7fee)
KERN: 0x80fe75b0->VMAnonymousCache::Fault(): Failed to reserve 4096 bytes of RAM.
KERN: vm_page_fault: vm_soft_fault returned error 'Out of memory' on fault at 0x700c0f54, ip 0x278a9d, write 1, user 1, thread 0x11b
KERN: vm_page_fault: thread "team 62 handler" (283) in team "debug_server" (62) tried to write address 0x700c0f54, ip 0x278a9d ("libroot.so_seg0ro" +0x6fa9d)
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: 0x840097a8->VMAnonymousCache::_Commit(2752512): Failed to reserve 131072 bytes of RAM
KERN: vm_soft_fault: va 0x2000 not covered by area in address space
KERN: vm_page_fault: vm_soft_fault returned error 'Bad address' on fault at 0x2000, ip 0x805fee, write 1, user 1, thread 0xe5
KERN: vm_page_fault: thread "w>Terminal 1: haiku: jam" (229) in team "Terminal" (221) tried to write address 0x2000, ip 0x805fee ("libroot.so_seg0ro" +0xa7fee)
KERN: vm_soft_fault: va 0x4000 not covered by area in address space
KERN: vm_page_fault: vm_soft_fault returned error 'Bad address' on fault at 0x4000, ip 0x2b0fee, write 1, user 1, thread 0x44
KERN: vm_page_fault: thread "kernel listener" (68) in team "debug_server" (62) tried to write address 0x4000, ip 0x2b0fee ("libroot.so_seg0ro" +0xa7fee)
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: 0x83c4c6e8->VMAnonymousCache::_Commit(28672): Failed to reserve 28672 bytes of RAM
KERN: runtime_loader: /boot/system/lib/libroot.so: Could not map image: Out of memory
KERN: low resource memory: critical -> warning
KERN: 0x83c25dc0->VMAnonymousCache::_Commit(28672): Failed to reserve 28672 bytes of RAM
KERN: runtime_loader: /boot/system/lib/libroot.so: Could not map image: Out of memory
KERN: low resource memory: warning -> critical
KERN: 0x83d19850->VMAnonymousCache::_Commit(65536): Failed to reserve 65536 bytes of RAM
KERN: 0x83d19350->VMAnonymousCache::_Commit(65536): Failed to reserve 65536 bytes of RAM
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: 0x83d19850->VMAnonymousCache::Fault(): Failed to reserve 4096 bytes of RAM.
KERN: vm_page_fault: vm_soft_fault returned error 'Out of memory' on fault at 0x780324c4, ip 0x631ec8, write 1, user 1, thread 0x13c
KERN: vm_page_fault: thread "w>TrackerWindow" (316) in team "Tracker" (115) tried to write address 0x780324c4, ip 0x631ec8 ("libroot.so_seg0ro" +0x2cec8)
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: low resource memory: warning -> critical
KERN: low resource memory: critical -> warning
KERN: 0x83d197b0->VMAnonymousCache::_Commit(8192): Failed to reserve 8192 bytes of RAM
Last message repeated 1 time
KERN: low resource memory: warning -> normal

At this point the system was unable to launch any further applications. (see screenshot attached)

Attachments (1)

Screenshot-1.png (97.8 KB ) - added by kallisti5 8 years ago.

Download all attachments as: .zip

Change History (5)

by kallisti5, 8 years ago

Attachment: Screenshot-1.png added

comment:2 by axeld, 8 years ago

If you ran out of memory, it's actually sane to not start new processes in favor of killing some random team. Besides yet, yes, we don't take such drastic measurements yet, and our low memory handling could surely be further improved.

I'm not sure it's a good idea to randomly kill teams (or rather, I definitely know it's a bad idea, it's just the question of what alternatives you have). One way to deal with things like these is to notify the userland from the problem, and let it deal by taking down not so important services (and also notifying the user) -- unless you currently record something, this could be the media server, for example. Or player only apps like MediaPlayer or ShowImage.

Once we have proper session management in place, a good alternative would be to just quit non-used applications that could be relaunched as they were when needed or possible.

comment:3 by kallisti5, 8 years ago

Those are good points.

If expected behavior is to kill the process attempting to malloc more memory then whats available... shouldn't the system remain stable?

In the example above, It looks as though the failed malloc was incompletely subtracted from available memory. I could no longer open new processes... even though the memory usage is low as per the memory bar in the top right side of the screenshot.

I also noticed jam was still running, even though the terminal was locked up. Once I forcibly killed jam (which didn't effect free memory) the system still wouldn't spawn new processes.

comment:4 by axeld, 3 years ago

Owner: changed from axeld to nobody
Status: newassigned
Note: See TracTickets for help on using tickets.