Opened 9 years ago

Closed 9 years ago

#5816 closed bug (fixed)

Kernel out of memory

Reported by: axeld Owned by: bonefish
Priority: normal Milestone: R1/alpha3
Component: System/Kernel Version: R1/Development
Keywords: Cc:
Blocked By: Blocking: #4963
Has a Patch: no Platform: All

Description

When running checkfs on a volume with many files, or when copying these files to another volume, the slabs keep an insane amount of memory back.

This works for a while, but ultimately, the kernel runs out of memory during the above actions. The test case was a 2 GB machine with around 400,000 files on it.

I'm currently trying to reproduce the problem in VMware with a lower amount of memory (and files).

Change History (9)

comment:1 Changed 9 years ago by michael.weirauch

Can confirm this on a volume with >500k files on it. Haven't just found the time filing the issue. (4GB (3GB) machine here)

comment:2 Changed 9 years ago by michael.weirauch

Still with us on hrev37154 gcc4h. (Btw, #4963 looks like a dup, respectively vice versa.)

checkfs run with additional avail, page_stats, swap:

2010-06-16 18:52:11 KERN: slab memory manager: created area 0xdf800000 (7983)
2010-06-16 18:52:13 KERN: slab memory manager: created area 0xe0000000 (7984)
2010-06-16 18:52:16 KERN: slab memory manager: created area 0xe0800000 (7985)
2010-06-16 18:52:17 KERN: heap_add_area: area 7986 added to small heap 0xccd0c000 - usable range 0xdec07000 - 0xdf000000
2010-06-16 18:52:20 KERN: slab memory manager: created area 0xe1000000 (7987)
2010-06-16 18:52:21 KERN: slab memory manager: created area 0xe1800000 (7988)
2010-06-16 18:52:22 KERN: heap_add_area: area 7989 added to small heap 0xccd0c000 - usable range 0xe2007000 - 0xe2400000
2010-06-16 18:52:22 KERN: slab memory manager: created area 0xe2800000 (7990)
2010-06-16 18:52:24 KERN: slab memory manager: created area 0xe3000000 (7991)
2010-06-16 18:52:24 KERN: heap_add_area: area 7992 added to small heap 0xccd0c000 - usable range 0xe2407000 - 0xe2800000
2010-06-16 18:52:25 KERN: slab memory manager: created area 0xe3800000 (7993)
2010-06-16 18:52:28 KERN: slab memory manager: created area 0xe4000000 (7994)
2010-06-16 18:52:31 KERN: heap_add_area: area 7995 added to small heap 0xccd0c000 - usable range 0xe4807000 - 0xe4c00000
2010-06-16 18:52:31 KERN: slab memory manager: created area 0xe5000000 (7996)
2010-06-16 18:52:34 KERN: slab memory manager: created area 0xe5800000 (7997)
2010-06-16 18:52:37 KERN: slab memory manager: created area 0xe6000000 (7998)
2010-06-16 18:52:38 KERN: heap_add_area: area 7999 added to small heap 0xccd0c000 - usable range 0xe4c07000 - 0xe5000000
2010-06-16 18:52:39 KERN: slab memory manager: created area 0xe6800000 (8000)
2010-06-16 18:52:41 KERN: slab memory manager: created area 0xe7000000 (8001)
2010-06-16 18:52:43 KERN: heap_add_area: area 8002 added to small heap 0xccd0c000 - usable range 0xe7807000 - 0xe7c00000
2010-06-16 18:52:44 KERN: slab memory manager: created area 0xe8000000 (8003)
2010-06-16 18:52:46 KERN: slab memory manager: created area 0xe8800000 (8004)
2010-06-16 18:52:47 KERN: heap_add_area: area 8005 added to small heap 0xccd0c000 - usable range 0xe7c07000 - 0xe8000000
2010-06-16 18:52:47 KERN: slab memory manager: created area 0xe9000000 (8006)
2010-06-16 18:52:48 KERN: slab memory manager: created area 0xe9800000 (8007)
2010-06-16 18:52:49 KERN: heap_add_area: area 8008 added to small heap 0xccd0c000 - usable range 0xea007000 - 0xea400000
2010-06-16 18:52:49 KERN: slab memory manager: created area 0xea800000 (8009)
2010-06-16 18:52:51 KERN: slab memory manager: created area 0xeb000000 (8010)
2010-06-16 18:52:51 KERN: heap_add_area: area 8011 added to small heap 0xccd0c000 - usable range 0xea407000 - 0xea800000
2010-06-16 18:52:52 KERN: slab memory manager: created area 0xeb800000 (8012)
2010-06-16 18:52:53 KERN: slab memory manager: created area 0xec000000 (8013)
2010-06-16 18:52:54 KERN: heap_add_area: area 8014 added to small heap 0xccd0c000 - usable range 0xec807000 - 0xecc00000
2010-06-16 18:52:54 KERN: slab memory manager: created area 0xed000000 (8015)
2010-06-16 18:52:59 KERN: slab memory manager: created area 0xed800000 (8016)
2010-06-16 18:53:06 KERN: slab memory manager: created area 0xee000000 (8017)
2010-06-16 18:53:06 KERN: heap_add_area: area 8018 added to small heap 0xccd0c000 - usable range 0xecc07000 - 0xed000000
2010-06-16 18:53:11 KERN: slab memory manager: created area 0xee800000 (8019)
2010-06-16 18:53:12 KERN: slab memory manager: created area 0xef000000 (8020)
2010-06-16 18:53:13 KERN: heap_add_area: area 8021 added to small heap 0xccd0c000 - usable range 0xef807000 - 0xefc00000
2010-06-16 18:53:14 KERN: slab memory manager: created area 0xf0000000 (8022)
2010-06-16 18:53:16 KERN: slab memory manager: created area 0xf0800000 (8023)
2010-06-16 18:53:20 KERN: heap_add_area: area 8024 added to small heap 0xccd0c000 - usable range 0xefc07000 - 0xf0000000
2010-06-16 18:53:21 KERN: slab memory manager: created area 0xf1000000 (8025)
2010-06-16 18:53:23 KERN: slab memory manager: created area 0xf1800000 (8026)
2010-06-16 18:53:25 KERN: heap_add_area: area 8027 added to small heap 0xccd0c000 - usable range 0xf2007000 - 0xf2400000
2010-06-16 18:53:26 KERN: slab memory manager: created area 0xf2800000 (8028)
2010-06-16 18:53:31 KERN: slab memory manager: created area 0xf3000000 (8029)
2010-06-16 18:53:35 KERN: slab memory manager: created area 0xf3800000 (8030)
2010-06-16 18:53:37 KERN: heap_add_area: area 8031 added to small heap 0xccd0c000 - usable range 0xf2407000 - 0xf2800000
2010-06-16 18:53:39 KERN: slab memory manager: created area 0xf4000000 (8032)
2010-06-16 18:53:43 KERN: slab memory manager: created area 0xf4800000 (8033)
2010-06-16 18:53:48 KERN: heap_add_area: area 8034 added to small heap 0xccd0c000 - usable range 0xf5007000 - 0xf5400000
2010-06-16 18:53:48 KERN: slab memory manager: created area 0xf5800000 (8035)
2010-06-16 18:53:53 KERN: slab memory manager: created area 0xf6000000 (8036)
2010-06-16 18:53:57 KERN: slab memory manager: created area 0xf6800000 (8037)
2010-06-16 18:54:00 KERN: heap_add_area: area 8038 added to small heap 0xccd0c000 - usable range 0xf5407000 - 0xf5800000
2010-06-16 18:54:01 KERN: slab memory manager: created area 0xf7000000 (8039)
2010-06-16 18:54:04 KERN: slab memory manager: created area 0xf7800000 (8040)
2010-06-16 18:54:06 KERN: slab memory manager: created area 0xf8000000 (8041)
2010-06-16 18:54:09 KERN: heap_add_area: area 8042 added to small heap 0xccd0c000 - usable range 0xf8807000 - 0xf8c00000
2010-06-16 18:54:10 KERN: slab memory manager: created area 0xf9000000 (8043)
2010-06-16 18:54:14 KERN: slab memory manager: created area 0xf9800000 (8044)
2010-06-16 18:54:19 KERN: heap_add_area: area 8045 added to small heap 0xccd0c000 - usable range 0xf8c07000 - 0xf9000000
2010-06-16 18:54:20 KERN: slab memory manager: created area 0xfa000000 (8046)
2010-06-16 18:54:25 KERN: slab memory manager: created area 0xfa800000 (8047)
2010-06-16 18:54:28 KERN: slab memory manager: created area 0xfb000000 (8048)
2010-06-16 18:54:28 KERN: heap_add_area: area 8049 added to small heap 0xccd0c000 - usable range 0xfb807000 - 0xfbc00000
2010-06-16 18:54:32 KERN: slab memory manager: created area 0xfc000000 (8050)
2010-06-16 18:54:34 KERN: slab memory manager: created area 0xfc800000 (8051)
2010-06-16 18:54:35 KERN: heap_add_area: area 8052 added to small heap 0xccd0c000 - usable range 0xfbc07000 - 0xfc000000
2010-06-16 18:54:37 KERN: slab memory manager: created area 0xfd000000 (8053)
2010-06-16 18:54:39 KERN: slab memory manager: created area 0xfd800000 (8054)
2010-06-16 18:54:41 KERN: heap_add_area: area 8055 added to small heap 0xccd0c000 - usable range 0xfe007000 - 0xfe400000
2010-06-16 18:54:44 KERN: slab memory manager: created area 0xfe800000 (8056)
2010-06-16 18:54:48 KERN: slab memory manager: created area 0xff000000 (8057)
2010-06-16 18:54:49 KERN: heap_add_area: area 8058 added to small heap 0xccd0c000 - usable range 0xfe407000 - 0xfe800000
2010-06-16 18:54:52 KERN: slab memory manager: created area 0x86800000 (8059)
2010-06-16 18:54:55 KERN: slab memory manager: created area 0x87000000 (8060)
2010-06-16 18:54:59 KERN: heap_add_area: area 8061 added to small heap 0xccd0c000 - usable range 0xff807000 - 0xffc00000
2010-06-16 18:54:59 KERN: slab memory manager: created area 0x87800000 (8062)
2010-06-16 18:55:00 KERN: slab memory manager: created area 0x88000000 (8063)
2010-06-16 18:55:02 KERN: heap_add_area: area 8064 added to small heap 0xccd0c000 - usable range 0x86407000 - 0x86800000
2010-06-16 18:55:02 KERN: slab memory manager: created area 0x88800000 (8065)
2010-06-16 18:55:05 KERN: slab memory manager: created area 0x89000000 (8066)
2010-06-16 18:55:09 KERN: heap_add_area: area 8067 added to small heap 0xccd0c000 - usable range 0x89807000 - 0x89c00000
2010-06-16 18:55:10 KERN: slab memory manager: created area 0x8a000000 (8068)
2010-06-16 18:55:15 KERN: slab memory manager: created area 0x8a800000 (8069)
2010-06-16 18:55:20 KERN: slab memory manager: created area 0x8b000000 (8070)
2010-06-16 18:55:21 KERN: heap_add_area: area 8071 added to small heap 0xccd0c000 - usable range 0x89c07000 - 0x8a000000
2010-06-16 18:55:23 KERN: slab memory manager: created area 0x8b800000 (8072)
2010-06-16 18:55:27 KERN: slab memory manager: created area 0x8c000000 (8073)
2010-06-16 18:55:30 KERN: heap_add_area: area 8074 added to small heap 0xccd0c000 - usable range 0x8c807000 - 0x8cc00000
2010-06-16 18:55:30 KERN: slab memory manager: created area 0x8d000000 (8075)
2010-06-16 18:55:34 KERN: slab memory manager: created area 0x8d800000 (8076)
2010-06-16 18:55:38 KERN: slab memory manager: created area 0x8e000000 (8077)
2010-06-16 18:55:40 KERN: heap_add_area: area 8078 added to small heap 0xccd0c000 - usable range 0x8cc07000 - 0x8d000000
2010-06-16 18:55:41 KERN: slab memory manager: created area 0x8e800000 (8079)
2010-06-16 18:55:44 KERN: slab memory manager: created area 0x8f800000 (8081)
2010-06-16 18:55:48 KERN: slab memory manager: created area 0x90000000 (8082)
2010-06-16 18:55:51 KERN: heap_add_area: area 8083 added to small heap 0xccd0c000 - usable range 0x8f407000 - 0x8f800000
2010-06-16 18:55:53 KERN: slab memory manager: created area 0x90800000 (8084)
2010-06-16 18:55:58 KERN: slab memory manager: created area 0x91000000 (8085)
2010-06-16 18:56:03 KERN: slab memory manager: created area 0x91800000 (8086)
2010-06-16 18:56:04 KERN: heap_add_area: area 8087 added to small heap 0xccd0c000 - usable range 0x92007000 - 0x92400000
2010-06-16 18:56:05 KERN: slab memory manager: created area 0x92800000 (8088)
2010-06-16 18:56:07 KERN: slab memory manager: created area 0x93000000 (8089)
2010-06-16 18:56:07 KERN: heap_add_area: area 8090 added to small heap 0xccd0c000 - usable range 0x92407000 - 0x92800000
2010-06-16 18:56:08 KERN: slab memory manager: created area 0x93800000 (8091)
2010-06-16 18:56:09 KERN: slab memory manager: created area 0x94000000 (8092)
2010-06-16 18:56:10 KERN: heap_add_area: area 8093 added to small heap 0xccd0c000 - usable range 0x94807000 - 0x94c00000
2010-06-16 18:56:11 KERN: slab memory manager: created area 0x95000000 (8094)
2010-06-16 18:56:14 KERN: slab memory manager: created area 0x95800000 (8095)
2010-06-16 18:56:15 KERN: heap_add_area: area 8096 added to small heap 0xccd0c000 - usable range 0x94c07000 - 0x95000000
2010-06-16 18:56:18 KERN: slab memory manager: created area 0x96000000 (8097)
2010-06-16 18:56:21 KERN: slab memory manager: created area 0x96800000 (8098)
2010-06-16 18:56:25 KERN: heap_add_area: area 8099 added to small heap 0xccd0c000 - usable range 0x97007000 - 0x97400000
2010-06-16 18:56:25 KERN: slab memory manager: created area 0x97800000 (8100)
2010-06-16 18:56:31 KERN: slab memory manager: created area 0x98000000 (8101)
2010-06-16 18:56:35 KERN: slab memory manager: created area 0x98800000 (8102)
2010-06-16 18:56:36 KERN: heap_add_area: area 8103 added to small heap 0xccd0c000 - usable range 0x97407000 - 0x97800000
2010-06-16 18:56:37 KERN: slab memory manager: created area 0x99000000 (8104)
2010-06-16 18:56:38 KERN: slab memory manager: created area 0x99800000 (8105)
2010-06-16 18:56:40 KERN: slab memory manager: created area 0x9a000000 (8106)
2010-06-16 18:56:41 KERN: heap_add_area: area 8107 added to small heap 0xccd0c000 - usable range 0x9a807000 - 0x9ac00000
2010-06-16 18:56:41 KERN: slab memory manager: created area 0x9b000000 (8108)
2010-06-16 18:56:43 KERN: slab memory manager: created area 0x9b800000 (8109)
2010-06-16 18:56:43 KERN: heap_add_area: area 8110 added to small heap 0xccd0c000 - usable range 0x9ac07000 - 0x9b000000
2010-06-16 18:56:44 KERN: slab memory manager: created area 0x9c000000 (8111)
2010-06-16 18:56:45 KERN: slab memory manager: created area 0x9c800000 (8112)
2010-06-16 18:56:47 KERN: slab memory manager: created area 0x9d000000 (8113)
2010-06-16 18:56:47 KERN: heap_add_area: area 8114 added to small heap 0xccd0c000 - usable range 0x9d807000 - 0x9dc00000
2010-06-16 18:56:48 KERN: slab memory manager: created area 0x9e000000 (8115)
2010-06-16 18:56:50 KERN: slab memory manager: created area 0x9e800000 (8116)
2010-06-16 18:56:50 KERN: heap_add_area: area 8117 added to small heap 0xccd0c000 - usable range 0x9dc07000 - 0x9e000000
2010-06-16 18:56:51 KERN: slab memory manager: created area 0x9f000000 (8118)
2010-06-16 18:56:53 KERN: slab memory manager: created area 0x9f800000 (8119)
2010-06-16 18:56:54 KERN: heap_add_area: area 8120 added to small heap 0xccd0c000 - usable range 0xa0007000 - 0xa0400000
2010-06-16 18:56:55 KERN: slab memory manager: created area 0xa0800000 (8121)
2010-06-16 18:56:57 KERN: slab memory manager: created area 0xa1000000 (8122)
2010-06-16 18:56:57 KERN: heap_add_area: area 8123 added to small heap 0xccd0c000 - usable range 0xa0407000 - 0xa0800000
2010-06-16 18:56:58 KERN: slab memory manager: created area 0xa1800000 (8124)
2010-06-16 18:56:59 KERN: slab memory manager: created area 0xa2000000 (8125)
2010-06-16 18:57:00 KERN: heap_add_area: area 8126 added to small heap 0xccd0c000 - usable range 0xa2807000 - 0xa2c00000
2010-06-16 18:57:01 KERN: slab memory manager: created area 0xa3000000 (8127)
2010-06-16 18:57:02 KERN: slab memory manager: created area 0xa3800000 (8128)
2010-06-16 18:57:04 KERN: slab memory manager: created area 0xa4000000 (8129)
2010-06-16 18:57:04 KERN: heap_add_area: area 8130 added to small heap 0xccd0c000 - usable range 0xa2c07000 - 0xa3000000
2010-06-16 18:57:05 KERN: slab memory manager: created area 0xa4800000 (8131)
2010-06-16 18:57:06 KERN: slab memory manager: created area 0xa5000000 (8132)
2010-06-16 18:57:07 KERN: heap_add_area: area 8133 added to small heap 0xccd0c000 - usable range 0xa5807000 - 0xa5c00000
2010-06-16 18:57:08 KERN: slab memory manager: created area 0xa6000000 (8134)
2010-06-16 18:57:09 KERN: slab memory manager: created area 0xa6800000 (8135)
2010-06-16 18:57:09 KERN: heap_add_area: area 8136 added to small heap 0xccd0c000 - usable range 0xa5c07000 - 0xa6000000
2010-06-16 18:57:11 KERN: slab memory manager: created area 0xa7000000 (8137)
2010-06-16 18:57:12 KERN: slab memory manager: created area 0xa7800000 (8138)
2010-06-16 18:57:12 KERN: heap_add_area: area 8139 added to small heap 0xccd0c000 - usable range 0xa8007000 - 0xa8400000
2010-06-16 18:57:14 KERN: slab memory manager: created area 0xa8800000 (8140)
2010-06-16 18:57:15 KERN: slab memory manager: created area 0xa9000000 (8141)
2010-06-16 18:57:16 KERN: heap_add_area: area 8142 added to small heap 0xccd0c000 - usable range 0xa8407000 - 0xa8800000
2010-06-16 18:57:17 KERN: slab memory manager: created area 0xa9800000 (8143)
2010-06-16 18:57:18 KERN: slab memory manager: created area 0xaa000000 (8144)
2010-06-16 18:57:18 KERN: heap_add_area: area 8145 added to small heap 0xccd0c000 - usable range 0xaa807000 - 0xaac00000
2010-06-16 18:57:19 KERN: slab memory manager: created area 0xab000000 (8146)
2010-06-16 18:57:21 KERN: slab memory manager: created area 0xab800000 (8147)
2010-06-16 18:57:23 KERN: slab memory manager: created area 0xac000000 (8148)
2010-06-16 18:57:24 KERN: heap_add_area: area 8149 added to small heap 0xccd0c000 - usable range 0xaac07000 - 0xab000000
2010-06-16 18:57:24 KERN: slab memory manager: created area 0xac800000 (8150)
2010-06-16 18:57:26 KERN: slab memory manager: created area 0xad000000 (8151)
2010-06-16 18:57:27 KERN: slab memory manager: created area 0xad800000 (8152)
2010-06-16 18:57:28 KERN: heap_add_area: area 8153 added to small heap 0xccd0c000 - usable range 0xae007000 - 0xae400000
2010-06-16 18:57:29 KERN: slab memory manager: created area 0xae800000 (8154)
2010-06-16 18:57:30 KERN: slab memory manager: created area 0xaf000000 (8155)
2010-06-16 18:57:31 KERN: heap_add_area: area 8156 added to small heap 0xccd0c000 - usable range 0xae407000 - 0xae800000
2010-06-16 18:57:31 KERN: slab memory manager: created area 0xaf800000 (8157)
2010-06-16 18:57:33 KERN: slab memory manager: created area 0xb0000000 (8158)
2010-06-16 18:57:35 KERN: slab memory manager: created area 0xb0800000 (8159)
2010-06-16 18:57:35 KERN: heap_add_area: area 8160 added to small heap 0xccd0c000 - usable range 0xb1007000 - 0xb1400000
2010-06-16 18:57:36 KERN: slab memory manager: created area 0xb1800000 (8161)
2010-06-16 18:57:39 KERN: slab memory manager: created area 0xb2000000 (8162)
2010-06-16 18:57:41 KERN: heap_add_area: area 8163 added to small heap 0xccd0c000 - usable range 0xb1407000 - 0xb1800000
2010-06-16 18:57:41 KERN: slab memory manager: created area 0xb2800000 (8164)
2010-06-16 18:57:42 KERN: slab memory manager: created area 0xb3000000 (8165)
2010-06-16 18:57:44 KERN: slab memory manager: created area 0xb3800000 (8166)
2010-06-16 18:57:46 KERN: heap_add_area: area 8167 added to small heap 0xccd0c000 - usable range 0xb4007000 - 0xb4400000
2010-06-16 18:57:47 KERN: slab memory manager: created area 0xb4800000 (8168)
2010-06-16 18:57:51 KERN: slab memory manager: created area 0xb5000000 (8169)
2010-06-16 18:57:55 KERN: slab memory manager: created area 0xb5800000 (8170)
2010-06-16 18:57:58 KERN: heap_add_area: area 8171 added to small heap 0xccd0c000 - usable range 0xb4407000 - 0xb4800000
2010-06-16 18:57:59 KERN: slab memory manager: created area 0xb6000000 (8172)
2010-06-16 18:58:00 KERN: slab memory manager: created area 0xb6800000 (8173)
2010-06-16 18:58:01 KERN: slab memory manager: created area 0xb7000000 (8174)
2010-06-16 18:58:02 KERN: heap_add_area: area 8175 added to small heap 0xccd0c000 - usable range 0xb7807000 - 0xb7c00000
2010-06-16 18:58:03 KERN: slab memory manager: created area 0xb8000000 (8176)
2010-06-16 18:58:04 KERN: slab memory manager: created area 0xb9000000 (8178)
2010-06-16 18:58:06 KERN: slab memory manager: created area 0xb9800000 (8179)
2010-06-16 18:58:06 KERN: heap_add_area: area 8180 added to small heap 0xccd0c000 - usable range 0xb7c07000 - 0xb8000000
2010-06-16 18:58:07 KERN: slab memory manager: created area 0xba000000 (8181)
2010-06-16 18:58:08 KERN: slab memory manager: created area 0xba800000 (8182)
2010-06-16 18:58:09 KERN: heap_add_area: area 8183 added to small heap 0xccd0c000 - usable range 0x8f007000 - 0x8f400000
2010-06-16 18:58:10 KERN: slab memory manager: created area 0xbb000000 (8184)
2010-06-16 18:58:12 KERN: slab memory manager: created area 0xbb800000 (8185)
2010-06-16 18:58:14 KERN: slab memory manager: created area 0xbc000000 (8186)
2010-06-16 18:58:17 KERN: slab memory manager: created area 0xbc800000 (8187)
2010-06-16 18:58:18 KERN: heap_add_area: area 8188 added to small heap 0xccd0c000 - usable range 0xbd007000 - 0xbd400000
2010-06-16 18:58:19 KERN: slab memory manager: created area 0xbd800000 (8189)
2010-06-16 18:58:21 KERN: slab memory manager: created area 0xbe000000 (8190)
2010-06-16 18:58:22 KERN: slab memory manager: created area 0xbe800000 (8191)
2010-06-16 18:58:22 KERN: heap_add_area: area 8192 added to small heap 0xccd0c000 - usable range 0xbd407000 - 0xbd800000
2010-06-16 18:58:23 KERN: slab memory manager: created area 0xbf000000 (8193)
2010-06-16 18:58:25 KERN: slab memory manager: created area 0xbf800000 (8194)
2010-06-16 18:58:27 KERN: slab memory manager: created area 0xc0000000 (8195)
2010-06-16 18:58:27 KERN: heap_add_area: area 8196 added to small heap 0xccd0c000 - usable range 0xc0807000 - 0xc0c00000
2010-06-16 18:58:28 KERN: slab memory manager: created area 0xc1000000 (8197)
2010-06-16 18:58:30 KERN: slab memory manager: created area 0xc1800000 (8198)
2010-06-16 18:58:30 KERN: heap_add_area: area 8199 added to small heap 0xccd0c000 - usable range 0xc0c07000 - 0xc1000000
2010-06-16 18:58:31 KERN: slab memory manager: created area 0xc2000000 (8200)
2010-06-16 18:58:32 KERN: slab memory manager: created area 0xc2800000 (8201)
2010-06-16 18:58:33 KERN: heap_add_area: area 8202 added to small heap 0xccd0c000 - usable range 0xc3007000 - 0xc3400000
2010-06-16 18:58:33 KERN: slab memory manager: created area 0xc3800000 (8203)
2010-06-16 18:58:34 KERN: slab memory manager: created area 0xc4000000 (8204)
2010-06-16 18:58:35 KERN: heap_add_area: area 8205 added to small heap 0xccd0c000 - usable range 0xc3407000 - 0xc3800000
2010-06-16 18:58:35 KERN: slab memory manager: created area 0xc4800000 (8206)
2010-06-16 18:58:36 KERN: slab memory manager: created area 0xc5000000 (8207)
2010-06-16 18:58:38 KERN: slab memory manager: created area 0xc5800000 (8208)
2010-06-16 18:58:39 KERN: heap_add_area: area 8209 added to small heap 0xccd0c000 - usable range 0xc6007000 - 0xc6400000
2010-06-16 18:58:40 KERN: slab memory manager: created area 0xc6800000 (8210)
2010-06-16 18:58:44 KERN: heap_add_area: area 8211 added to small heap 0xccd0c000 - usable range 0xc6407000 - 0xc6800000
2010-06-16 18:58:47 KERN: heap_add_area: area 8212 added to small heap 0xccd0c000 - usable range 0xc7007000 - 0xc7400000
2010-06-16 18:58:49 KERN: heap_add_area: area 8213 added to small heap 0xccd0c000 - usable range 0xc7407000 - 0xc7800000
2010-06-16 18:58:54 KERN: heap_add_area: area 8214 added to small heap 0xccd0c000 - usable range 0xc7807000 - 0xc7c00000
2010-06-16 18:58:57 KERN: heap_add_area: area 8215 added to small heap 0xccd0c000 - usable range 0xc7c07000 - 0xc8000000
2010-06-16 18:59:00 KERN: heap_add_area: area 8216 added to small heap 0xccd0c000 - usable range 0xc8007000 - 0xc8400000
2010-06-16 18:59:04 KERN: heap_add_area: area 8217 added to small heap 0xccd0c000 - usable range 0xc8407000 - 0xc8800000
2010-06-16 18:59:06 KERN: heap_add_area: area 8218 added to small heap 0xccd0c000 - usable range 0xc8807000 - 0xc8c00000
2010-06-16 18:59:09 KERN: heap_add_area: area 8219 added to small heap 0xccd0c000 - usable range 0xc8c07000 - 0xc9000000
2010-06-16 18:59:11 KERN: heap_add_area: area 8220 added to small heap 0x82800000 - usable range 0xc9007000 - 0xc9400000
2010-06-16 18:59:12 KERN: heap_add_area: area 8221 added to small heap 0xccd0c000 - usable range 0xc9407000 - 0xc9800000
2010-06-16 18:59:14 KERN: heap_add_area: area 8222 added to small heap 0xccd0c000 - usable range 0xc9807000 - 0xc9c00000
2010-06-16 18:59:16 KERN: heap_add_area: area 8223 added to small heap 0xccd0c000 - usable range 0xc9c07000 - 0xca000000
2010-06-16 18:59:19 KERN: heap_add_area: area 8224 added to small heap 0xccd0c000 - usable range 0xca007000 - 0xca400000
2010-06-16 18:59:21 KERN: heap_add_area: area 8225 added to small heap 0xccd0c000 - usable range 0xca407000 - 0xca800000
2010-06-16 18:59:26 KERN: heap_add_area: area 8226 added to small heap 0xccd0c000 - usable range 0xca807000 - 0xcac00000
2010-06-16 18:59:30 KERN: heap_add_area: area 8227 added to small heap 0xccd0c000 - usable range 0xcac07000 - 0xcb000000
2010-06-16 18:59:35 KERN: heap_add_area: area 8228 added to small heap 0xccd0c000 - usable range 0xcb007000 - 0xcb400000
2010-06-16 18:59:41 KERN: heap_add_area: area 8229 added to small heap 0xccd0c000 - usable range 0xcb407000 - 0xcb800000
2010-06-16 18:59:44 KERN: heap_add_area: area 8230 added to small heap 0xccd0c000 - usable range 0xcb807000 - 0xcbc00000
2010-06-16 18:59:47 KERN: heap_add_area: area 8231 added to small heap 0xccd0c000 - usable range 0xcbc07000 - 0xcc000000
2010-06-16 18:59:53 KERN: heap_add_area: area 8232 added to small heap 0xccd0c000 - usable range 0xcc007000 - 0xcc400000
2010-06-16 18:59:59 KERN: heap_add_area: area 8233 added to small heap 0xccd0c000 - usable range 0xcc407000 - 0xcc800000
2010-06-16 19:00:02 KERN: heap_add_area: area 8234 added to small heap 0xccd0c000 - usable range 0xcc807000 - 0xccc00000
2010-06-16 19:00:05 KERN: heap_grower: failed to create new heap area
2010-06-16 19:01:17 KERN: Last message repeated 26330 times.
2010-06-16 19:01:17 KERN: PANIC: heap: kernel heap has run out of memory
2010-06-16 19:01:17 KERN: 
2010-06-16 19:01:17 KERN: Welcome to Kernel Debugging Land...
2010-06-16 19:01:17 KERN: Thread 1659 "checkfs" running on CPU 1
2010-06-16 19:01:17 KERN: stack trace for thread 1659 "checkfs"
2010-06-16 19:01:17 KERN:     kernel stack: 0x810ff000 to 0x81103000
2010-06-16 19:01:17 KERN:       user stack: 0x7efef000 to 0x7ffef000
2010-06-16 19:01:17 KERN: frame               caller     <image>:function + offset
2010-06-16 19:01:17 KERN:  0 81102948 (+  32) 800e780b   <kernel_x86>:arch_debug_stack_trace + 0x000f
2010-06-16 19:01:17 KERN:  1 81102968 (+  16) 80071c5a   <kernel_x86> stack_trace_trampoline(void*: NULL) + 0x000b
2010-06-16 19:01:17 KERN:  2 81102978 (+  12) 800ecfca   <kernel_x86>:arch_debug_call_with_fault_handler + 0x001b
2010-06-16 19:01:17 KERN:  3 81102984 (+  48) 80072150   <kernel_x86>:debug_call_with_fault_handler + 0x0053
2010-06-16 19:01:17 KERN:  4 811029b4 (+  64) 8007310b   <kernel_x86> kernel_debugger_loop(char const*: [34m0x1[0m [31m"<???>"[0m, char const*: [34m0x8012b8e8[0m [36m"heap: kernel heap has run out of memory
2010-06-16 19:01:17 KERN: "[0m, char*: [34m0x81102a24[0m, int32: [34m-2147011705[0m) + 0x0220
2010-06-16 19:01:17 KERN:  5 811029f4 (+  48) 800733a5   <kernel_x86> kernel_debugger_internal(char const*: [34m0x1[0m [31m"<???>"[0m, char const*: [34m0x0[0m [31m"<NULL>"[0m, char*: [34m0x81102a44[0m, int32: [34m-2147011211[0m) + 0x0110
2010-06-16 19:01:17 KERN:  6 81102a24 (+  32) 80073588   <kernel_x86>:panic + 0x0023
2010-06-16 19:01:17 KERN:  7 81102a44 (+  80) 8004d4c5   <kernel_x86>:memalign + 0x02bd
2010-06-16 19:01:17 KERN:  8 81102a94 (+  32) 8004d51d   <kernel_x86>:malloc + 0x0010
2010-06-16 19:01:17 KERN:  9 81102ab4 (+  64) 800afc3f   <kernel_x86> create_new_vnode_and_lock(int32: [34m-2129646788[0m, int64: [34m-9146763201246909629[0m, vnode*&: [34m0x800b120e[0m, bool&: [34m0x80150200[0m) + 0x0018
2010-06-16 19:01:17 KERN: 10 81102af4 (+  96) 800b1393   <kernel_x86> get_vnode(int32: [34m-2129646716[0m, int64: [34m4294967297[0m, vnode**: [34m0x588f36[0m, [34mfalse[0m, int32: [34m5803520[0m) + 0x01d6
2010-06-16 19:01:17 KERN: 11 81102b54 (+  64) 800b2ee8   <kernel_x86>:get_vnode + 0x002d
2010-06-16 19:01:17 KERN: 12 81102b94 (+ 464) 81402b0c   <bfs> BlockAllocator<[32m0x830457d8[0m>::CheckNextNode(check_control*: [34m0x81102d84[0m) + 0x0a8e
2010-06-16 19:01:17 KERN: 13 81102d64 (+ 352) 8141bc3c   <bfs> bfs_ioctl(fs_volume*: [34m0x828afa78[0m, fs_vnode*: [34m0xcce48ae4[0m, void*: [34m0x828ab5a0[0m, uint32: [34m0x377b[0m ([34m14203[0m), void*: [34m0x7ffeeca8[0m, uint32: [34m0x134[0m ([34m308[0m)) + 0x0113
2010-06-16 19:01:17 KERN: 14 81102ec4 (+  48) 800ac731   <kernel_x86> common_ioctl(file_descriptor*: [34m0x828af258[0m, uint32: [34m0x377b[0m ([34m14203[0m), void*: [34m0x7ffeeca8[0m, uint32: [34m0x134[0m ([34m308[0m)) + 0x0034
2010-06-16 19:01:17 KERN: 15 81102ef4 (+  48) 800a3bd8   <kernel_x86> fd_ioctl([34mtrue[0m, int32: [34m308[0m, uint32: [34m0x1[0m ([34m1[0m), void*: NULL, uint32: [34m0x1[0m ([34m1[0m)) + 0x005e
2010-06-16 19:01:17 KERN: 16 81102f24 (+  32) 800a406e   <kernel_x86>:_user_ioctl + 0x0034
2010-06-16 19:01:17 KERN: 17 81102f44 (+ 100) 800ee1b2   <kernel_x86>:handle_syscall + 0x00af
2010-06-16 19:01:17 KERN: user iframe at 0x81102fa8 (end = 0x81103000)
2010-06-16 19:01:17 KERN:  eax 0x8b           ebx 0x513a28        ecx 0x7ffeec2c   edx 0xffff0114
2010-06-16 19:01:17 KERN:  esi 0x8c9cf        edi 0x0             ebp 0x7ffeec48   esp 0x81102fdc
2010-06-16 19:01:17 KERN:  eip 0xffff0114  eflags 0x216      user esp 0x7ffeec2c
2010-06-16 19:01:17 KERN:  vector: 0x63, error code: 0x0
2010-06-16 19:01:17 KERN: 18 81102fa8 (+   0) ffff0114   <commpage>:commpage_syscall + 0x0004
2010-06-16 19:01:17 KERN: 19 7ffeec48 (+ 576) 005d3981   </boot/system/add-ons/disk_systems/bfs@0x005d0000>:unknown + 0x3981
2010-06-16 19:01:17 KERN: 20 7ffeee88 (+  32) 003a96b7   <libbe.so> BPartition::Delegate<[32m0x1801a820[0m>::Repair([34mfalse[0m) + 0x002b
2010-06-16 19:01:17 KERN: 21 7ffeeea8 (+  32) 003a8174   <libbe.so> BPartition<[32m0x1801f0e0[0m>::Repair const([34mfalse[0m) + 0x0036
2010-06-16 19:01:17 KERN: 22 7ffeeec8 (+ 160) 00201222   <_APP_>:main + 0x02e7
2010-06-16 19:01:17 KERN: 23 7ffeef68 (+  52) 00200dcd   <_APP_>:_start + 0x0051
2010-06-16 19:01:17 KERN: 24 7ffeef9c (+  64) 0010534f   </boot/system/runtime_loader@0x00100000>:unknown + 0x534f
2010-06-16 19:01:17 KERN: 25 7ffeefdc (+   0) 7ffeefec   7960:checkfs_main_stack@0x7efef000 + 0xffffec
2010-06-16 19:01:17 KERN: kdebug> avail Available memory: 1363910656/3214311424 bytes
2010-06-16 19:01:17 KERN: kdebug> page?s[1D[1D[1D[1D)[1D[1D_stats page stats:
2010-06-16 19:01:17 KERN: total: 784838
2010-06-16 19:01:17 KERN: active: 745 (busy: 0)
2010-06-16 19:01:17 KERN: inactive: 13207 (busy: 0)
2010-06-16 19:01:17 KERN: cached: 17096 (busy: 0)
2010-06-16 19:01:17 KERN: unused: 142 (busy: 0)
2010-06-16 19:01:17 KERN: wired: 435520 (busy: 0)
2010-06-16 19:01:17 KERN: modified: 0 (busy: 0)
2010-06-16 19:01:17 KERN: free: 300643
2010-06-16 19:01:17 KERN: clear: 17485
2010-06-16 19:01:17 KERN: unreserved free pages: 318128
2010-06-16 19:01:17 KERN: unsatisfied page reservations: 0
2010-06-16 19:01:17 KERN: mapped pages: 439178
2010-06-16 19:01:17 KERN: longest free pages run: 301116 pages (at 483429)
2010-06-16 19:01:17 KERN: longest free/cached pages run: 301117 pages (at 483428)
2010-06-16 19:01:17 KERN: waiting threads:
2010-06-16 19:01:17 KERN: 
2010-06-16 19:01:17 KERN: free queue: 0x8015b4b0, count = 300643
2010-06-16 19:01:17 KERN: clear queue: 0x8015b4c4, count = 17485
2010-06-16 19:01:17 KERN: modified queue: 0x8015b488, count = 0 (0 temporary, 0 swappable, inactive: 0)
2010-06-16 19:01:17 KERN: active queue: 0x8015b460, count = 745
2010-06-16 19:01:17 KERN: inactive queue: 0x8015b474, count = 13207
2010-06-16 19:01:17 KERN: cached queue: 0x8015b49c, count = 17096
2010-06-16 19:01:17 KERN: kdebug> d[1D[1Dswapswap files:
2010-06-16 19:01:17 KERN: 
2010-06-16 19:01:17 KERN: swap space in pages:
2010-06-16 19:01:17 KERN: total:             0
2010-06-16 19:01:17 KERN: available:         0
2010-06-16 19:01:17 KERN: reserved:          0
2010-06-16 19:01:17 KERN: used:              0
2010-06-16 19:01:17 KERN: free:              0

comment:3 Changed 9 years ago by bonefish

Owner: changed from axeld to bonefish
Status: newassigned

I've also run into this during the last few days while testing PAE on a box with 4 GB RAM. After a dd if=/dev/zero of=tt bs=1M count=3500 I started copying two haiku trees (with objects, weighing in at about 3.6 GB) from one partition to another. The copy process started relatively slow -- 2 to 3 MB/s -- and after a while dropped to abysmally slow -- < 100 KB/s. I didn't always let the copy process run on, but twice I let it run until it hit the out of memory panic. The syslog said pretty much the same as in the previous comment: The slab and heap would grow until the heap grower failed to allocate new areas. The reason being lack of address space in my case and apparently in the above one, too.

As can be seen in the output, the slab allocations stop at some point -- probably when the address space low resource state reaches "note" and the block cache stops allocated new blocks -- and only the heap continues to grow -- probably for vnodes and related structures. That is suboptimal already -- currently the kernel favors vnodes over block cache over file cache. Still it doesn't explain why slab areas aren't freed at some point. At low resource state "critical" the block cache should go all out and throw away lots of blocks. Also, that doesn't explain the amazingly poor copying performance.

Will try to analyze a bit.

comment:4 Changed 9 years ago by bonefish

Blocking: 4963 added

(In #4963) Duplicate of #5816, which has more info attached.

comment:5 Changed 9 years ago by bonefish

Just finished running the test again with some more debug output. The low resource state transitions are logged and block_cache::_LowMemoryHandler() prints how the RemoveUnusedBlocks() changed the number of unused blocks. The test started again at about 2-3 MB/s and -- I hadn't noticed this before -- after a few seconds reaches even > 10 MB/s for a short time. Afterwards it varies from 1 to 10 MB/s until the address space "note" low resource state is reached. Then the throughput drops immediately to < 1 MB, varying from 10 KB/s to a few 100 KB/s, only from time to time recovering to 1 or 2 MB/s (sometimes even more) for a few seconds.

Here's the output from around the point where the low resource limit is hit:

heap_add_area: area 3649 added to small heap 0xd090c000 - usable range 0xc2c07000 - 0xc3000000
slab memory manager: created area 0xc5800000 (3650)
slab memory manager: created area 0xc6000000 (3651)
low resource address space: normal -> note
block_cache::_LowMemoryHandler(): 0xcd92a198: unused: 25997 -> 25997
block_cache::_LowMemoryHandler(): 0xcd92a7f8: unused: 3 -> 0
block_cache::_LowMemoryHandler(): 0xccd27b28: unused: 313434 -> 313384
block_cache::_LowMemoryHandler(): 0xcfd1f660: unused: 13 -> 0
block_cache::_LowMemoryHandler(): 0xcd92a198: unused: 2598 -> 2598
block_cache::_LowMemoryHandler(): 0xcd92a7f8: unused: 0 -> 0
block_cache::_LowMemoryHandler(): 0xccd27b28: unused: 311848 -> 311798
block_cache::_LowMemoryHandler(): 0xcfd1f660: unused: 0 -> 0
block_cache::_LowMemoryHandler(): 0xcd92a198: unused: 22 -> 22
block_cache::_LowMemoryHandler(): 0xcd92a7f8: unused: 0 -> 0
block_cache::_LowMemoryHandler(): 0xccd27b28: unused: 311378 -> 311328
block_cache::_LowMemoryHandler(): 0xcfd1f660: unused: 0 -> 0
block_cache::_LowMemoryHandler(): 0xcd92a198: unused: 0 -> 0
block_cache::_LowMemoryHandler(): 0xcd92a7f8: unused: 0 -> 0
block_cache::_LowMemoryHandler(): 0xccd27b28: unused: 311244 -> 311194
block_cache::_LowMemoryHandler(): 0xcfd1f660: unused: 0 -> 0

The first block cache (0xcd92a198) is the one of the volume I'm copying, the third one (0xccd27b28) from the target volume (both on the same disk). As can be seen Tracker's pre-flight phase has let the number of unused blocks of the source volume grow to > 300000. The 25997 blocks of the target volume are probably mostly the ones that have already been written.

After only three seconds the target cache has been drained completely. RemoveUnusedBlocks() didn't remove a single block, though, since at level "note" only blocks older than two minutes are removed. The reason that all blocks are removed nonetheless is block_cache::Allocate(), which removes 2 unused blocks each time it is called. And since the target volume is quite active -- lots of writing and transactions going on -- that happens quite often. This would also explain the abysmal performance from that point on, since all blocks that become unused even for a short time are thrown away and have to be re-read from disk when they are needed again. The low resource behavior definitely needs to be rethought.

The source volume cache lost at max a few hundred blocks per second. It took about 1:45 h to drop to 0. Interestingly the whole time not a single slab are was released -- which must be a bug/leak somewhere -- and the heap kept growing -- likely for vnodes, file caches and BFS data structures, which don't care about the address space state. After about 2:20 h (80 % copied) the heap grower failed to allocate a heap area.

kdebug> low_resource
current state: ---a
  pages:  normal (723271)
  memory: normal (3025899520)
  sems:   normal (64352)
  aspace: note (28450816)

function    data         resources  prio  function-name
0x800df218  0x00000000   p m   a         5  object_cache_low_memory__FPvUll
0x800bad8c  0x00000000   p m             0  vnode_low_resource_handler__FPvUll
0x8003d044  0xcd92a198   p m   a         0  _LowMemoryHandler__11block_cachePvUll
0x8003d044  0xcd92a7f8   p m   a         0  _LowMemoryHandler__11block_cachePvUll
0x8003d044  0xccd27b28   p m   a         0  _LowMemoryHandler__11block_cachePvUll
0x8003d044  0xcfd1f660   p m   a         0  _LowMemoryHandler__11block_cachePvUll

Since there are still 28 MB address space left (probably fragmented or partially reserved), the address space low resource state is still "note". "Warning" state is currently reached at 8 MB, "critical" at utterly useless 32 KB. This is obviously something that needs to be changed.

As for the block caches:

kdebug> block_cache 0xcd92a198
BLOCK CACHE: 0xcd92a198
 fd:           0
 max_blocks:   26214048
 block_size:   2048
 next_transaction_id: 198
 buffer_cache: 0x828a1dc0
 busy_reading: 0, no waiters
 busy_writing: 0, no waiters
 797 blocks total, 311 dirty, 0 discarded, 0 referenced, 0 busy, 486 in unused.
kdebug> block_cache 0xccd27b28
BLOCK CACHE: 0xccd27b28
 fd:           7
 max_blocks:   26214048
 block_size:   2048
 next_transaction_id: 4
 buffer_cache: 0x828a6a00
 busy_reading: 0, no waiters
 busy_writing: 0, no waiters
 4 blocks total, 3 dirty, 0 discarded, 0 referenced, 0 busy, 1 in unused.

They don't know a lot of blocks.

The slabs:

kdebug> slabs
   address                   name  objsize    usage  empty  usedobj    total    flags
[...]
0x82808d00          cached blocks       64 12124160    178      814   189255 20000000
0x828a1dc0    block cache buffers     2048 122683392    192     1622    59904 20000000
0x828a5000    block cache buffers     4096   524288      0        2      128 20000000
0x828a6a00    block cache buffers     2048 642777088   1225       11   313856 20000000
0x828a6280    block cache buffers     2048   524288      1        0      256 20000000
kdebug> slab_cache 0x82808d00
name:              cached blocks
lock:              0x82808d28
object_size:       64
cache_color_cycle: 0
total_objects:     189255
used_count:        814
empty_count:       178
pressure:          354
slab_size:         65536
usage:             12124160
maximum:           0
flags:             0x20000000
cookie:            0x00000000
resize entry don't wait: 0x00000000
resize entry can wait:   0x00000000
  slab        chunk         size   used offset  free
empty:
  0xe902ffe4  0xe9020000    1023   1023      0  0xe90217fc
  0xe804ffe4  0xe8040000    1023   1023      0  0xe8044afc
  0x82efffe4  0x82ef0000    1023   1023      0  0x82efbebc
[...]
partial:
  0xda05ffe4  0xda050000    1023    491      0  0xda057cbc
  0xe085ffe4  0xe0850000    1023   1011      0  0xe085733c
  0x8291ffe4  0x82910000    1023   1010      0  0x8291927c
  0xdf04ffe4  0xdf040000    1023   1018      0  0xdf047ebc
  0xdd81ffe4  0xdd810000    1023    926      0  0xdd8151bc
  0xd905ffe4  0xd9050000    1023    944      0  0xd905eafc
  0xe105ffe4  0xe1050000    1023    947      0  0xe105773c
depot:
  full:     0x00000000, count 16
  empty:    0x00000000, count 0
  max full: 16
  capacity: 32
  stores:
  [0] loaded:   0x00000000
      previous: 0x00000000
  [1] loaded:   0x00000000
      previous: 0x00000000
  [2] loaded:   0x828ce140
      previous: 0x00000000
  [3] loaded:   0x00000000
      previous: 0x00000000
  [4] loaded:   0x828ce780
      previous: 0x00000000
  [5] loaded:   0x828ceaa0
      previous: 0x00000000
  [6] loaded:   0x00000000
      previous: 0x00000000
  [7] loaded:   0x828ce8c0
      previous: 0x00000000
kdebug> slab_cache 0x828a6a00
name:              block cache buffers
lock:              0x828a6a28
object_size:       2048
cache_color_cycle: 0
total_objects:     313856
used_count:        11
empty_count:       1225
pressure:          2449
slab_size:         524288
usage:             642777088
maximum:           0
flags:             0x20000000
cookie:            0x00000000
resize entry don't wait: 0x00000000
resize entry can wait:   0x00000000
  slab        chunk         size   used offset  free
empty:
  0x828af300  0xde200000     256    256      0  0xde27e7fc
  0x828af240  0xde500000     256    256      0  0xde559ffc
  0x828af1c0  0xde700000     256    256      0  0xde730ffc
[...]
partial:
  0x828059e0  0x82d00000     256    245      0  0x82d16ffc
depot:
  full:     0x00000000, count 4
  empty:    0x00000000, count 0
  max full: 4
  capacity: 8
  stores:
  [0] loaded:   0x00000000
      previous: 0x00000000
  [1] loaded:   0x00000000
      previous: 0x00000000
  [2] loaded:   0x00000000
      previous: 0x00000000
  [3] loaded:   0x00000000
      previous: 0x00000000
  [4] loaded:   0x00000000
      previous: 0x00000000
  [5] loaded:   0x00000000
      previous: 0x00000000
  [6] loaded:   0x00000000
      previous: 0x00000000
  [7] loaded:   0x00000000
      previous: 0x00000000

Truncated the free slabs lists -- there are loads of them. Apparently something goes wrong in the slab allocator's low resource handler. Will look into this tomorrow.

comment:6 Changed 9 years ago by axeld

The throwing away of blocks in block_cache::Allocate() is more of a work-around to this problem than a solution.

You should be able to reproduce the problem with a checkfs run much faster, though :-)

comment:7 in reply to:  6 Changed 9 years ago by bonefish

Replying to axeld:

The throwing away of blocks in block_cache::Allocate() is more of a work-around to this problem than a solution.

Given that it is probably responsible for the poor performance, it is definitely not a solution. :-)

You should be able to reproduce the problem with a checkfs run much faster, though :-)

I've never seen this with checkfs. There's perhaps not enough data on the partition (a bit less than 300000 nodes IIRC).

comment:8 Changed 9 years ago by bonefish

Some additional output in object_cache_low_memory() makes it clear what is happening:

low resource address space: normal -> note
object_cache_low_memory(): cache 0x82808d00: pressure: 356, empty: 0
object_cache_low_memory(): cache 0x828a1dc0: pressure: 397, empty: 2
object_cache_low_memory(): cache 0x828a6140: pressure: 2449, empty: 0
block_cache::_LowMemoryHandler(): 0xd1526000: unused: 10 -> 10
block_cache::_LowMemoryHandler(): 0x83023594: unused: 3 -> 0
block_cache::_LowMemoryHandler(): 0xd1526594: unused: 313434 -> 313384
block_cache::_LowMemoryHandler(): 0xd0921198: unused: 13 -> 0
object_cache_low_memory(): cache 0x82808d00: pressure: 356, empty: 0
object_cache_low_memory(): cache 0x828a1dc0: pressure: 397, empty: 9
object_cache_low_memory(): cache 0x828a6140: pressure: 2449, empty: 0
block_cache::_LowMemoryHandler(): 0xd1526000: unused: 3156 -> 3156
block_cache::_LowMemoryHandler(): 0x83023594: unused: 0 -> 0
block_cache::_LowMemoryHandler(): 0xd1526594: unused: 313054 -> 313004
block_cache::_LowMemoryHandler(): 0xd0921198: unused: 0 -> 0
object_cache_low_memory(): cache 0x82808d00: pressure: 356, empty: 3
object_cache_low_memory(): cache 0x828a1dc0: pressure: 397, empty: 141
object_cache_low_memory(): cache 0x828a6140: pressure: 2449, empty: 0
block_cache::_LowMemoryHandler(): 0xd1526000: unused: 0 -> 0
block_cache::_LowMemoryHandler(): 0x83023594: unused: 0 -> 0
block_cache::_LowMemoryHandler(): 0xd1526594: unused: 312768 -> 312718
block_cache::_LowMemoryHandler(): 0xd0921198: unused: 0 -> 0

ObjectCache::pressure is only decreased when the low resource level reaches at least "warning" (which never happens during the test), so the empty slab counts have a fixed upper limit (e.g. 1225 for the source volume block cache).

Reviewing the figures from the previous run, the slabs use less than 800 MB, BTW. Fragmentation of the slab areas might translate that to quite a bit more, but I guess the real problem is that the heap is continuously growing -- i.e. even if the slabs were drained completely, the address space limit would be hit eventually. It also doesn't help that heap and slab have different allocation backends (IOW that the slab allocator is not used as the heap).

comment:9 Changed 9 years ago by bonefish

Resolution: fixed
Status: assignedclosed

Fixed:

  • the way too low "warning" and "critical" low resource state address space limits in hrev37168,
  • the vnode cache ignoring the address space low resource state in hrev37169,
  • extremely quick draining of actively written to block caches (and thus the performance issue) when in "note" low resource state in hrev37170,
  • the rather slow reduction of unused blocks of huge block caches when reaching a low resource state also in hrev37170,
  • the slab object caches keeping a high number of empty slabs in "note" low resource state in hrev37171.

Not fixed yet is that vnode and block caches are favored over file cache and IOCache. I haven't tested fscheck -- as mentioned earlier, I couldn't reproduce the problem with it -- but at least my copy test runs smoothly, now. Please reopen, if there are still memory related issues with fscheck.

Note: See TracTickets for help on using tickets.