Opened 14 years ago
Closed 14 years ago
#5816 closed bug (fixed)
Kernel out of memory
| Reported by: | axeld | Owned by: | bonefish |
|---|---|---|---|
| Priority: | normal | Milestone: | R1/alpha3 |
| Component: | System/Kernel | Version: | R1/Development |
| Keywords: | Cc: | ||
| Blocked By: | Blocking: | #4963 | |
| Platform: | All |
Description
When running checkfs on a volume with many files, or when copying these files to another volume, the slabs keep an insane amount of memory back.
This works for a while, but ultimately, the kernel runs out of memory during the above actions. The test case was a 2 GB machine with around 400,000 files on it.
I'm currently trying to reproduce the problem in VMware with a lower amount of memory (and files).
Change History (9)
comment:1 by , 14 years ago
comment:2 by , 14 years ago
Still with us on hrev37154 gcc4h. (Btw, #4963 looks like a dup, respectively vice versa.)
checkfs run with additional avail, page_stats, swap:
2010-06-16 18:52:11 KERN: slab memory manager: created area 0xdf800000 (7983) 2010-06-16 18:52:13 KERN: slab memory manager: created area 0xe0000000 (7984) 2010-06-16 18:52:16 KERN: slab memory manager: created area 0xe0800000 (7985) 2010-06-16 18:52:17 KERN: heap_add_area: area 7986 added to small heap 0xccd0c000 - usable range 0xdec07000 - 0xdf000000 2010-06-16 18:52:20 KERN: slab memory manager: created area 0xe1000000 (7987) 2010-06-16 18:52:21 KERN: slab memory manager: created area 0xe1800000 (7988) 2010-06-16 18:52:22 KERN: heap_add_area: area 7989 added to small heap 0xccd0c000 - usable range 0xe2007000 - 0xe2400000 2010-06-16 18:52:22 KERN: slab memory manager: created area 0xe2800000 (7990) 2010-06-16 18:52:24 KERN: slab memory manager: created area 0xe3000000 (7991) 2010-06-16 18:52:24 KERN: heap_add_area: area 7992 added to small heap 0xccd0c000 - usable range 0xe2407000 - 0xe2800000 2010-06-16 18:52:25 KERN: slab memory manager: created area 0xe3800000 (7993) 2010-06-16 18:52:28 KERN: slab memory manager: created area 0xe4000000 (7994) 2010-06-16 18:52:31 KERN: heap_add_area: area 7995 added to small heap 0xccd0c000 - usable range 0xe4807000 - 0xe4c00000 2010-06-16 18:52:31 KERN: slab memory manager: created area 0xe5000000 (7996) 2010-06-16 18:52:34 KERN: slab memory manager: created area 0xe5800000 (7997) 2010-06-16 18:52:37 KERN: slab memory manager: created area 0xe6000000 (7998) 2010-06-16 18:52:38 KERN: heap_add_area: area 7999 added to small heap 0xccd0c000 - usable range 0xe4c07000 - 0xe5000000 2010-06-16 18:52:39 KERN: slab memory manager: created area 0xe6800000 (8000) 2010-06-16 18:52:41 KERN: slab memory manager: created area 0xe7000000 (8001) 2010-06-16 18:52:43 KERN: heap_add_area: area 8002 added to small heap 0xccd0c000 - usable range 0xe7807000 - 0xe7c00000 2010-06-16 18:52:44 KERN: slab memory manager: created area 0xe8000000 (8003) 2010-06-16 18:52:46 KERN: slab memory manager: created area 0xe8800000 (8004) 2010-06-16 18:52:47 KERN: heap_add_area: area 8005 added to small heap 0xccd0c000 - usable range 0xe7c07000 - 0xe8000000 2010-06-16 18:52:47 KERN: slab memory manager: created area 0xe9000000 (8006) 2010-06-16 18:52:48 KERN: slab memory manager: created area 0xe9800000 (8007) 2010-06-16 18:52:49 KERN: heap_add_area: area 8008 added to small heap 0xccd0c000 - usable range 0xea007000 - 0xea400000 2010-06-16 18:52:49 KERN: slab memory manager: created area 0xea800000 (8009) 2010-06-16 18:52:51 KERN: slab memory manager: created area 0xeb000000 (8010) 2010-06-16 18:52:51 KERN: heap_add_area: area 8011 added to small heap 0xccd0c000 - usable range 0xea407000 - 0xea800000 2010-06-16 18:52:52 KERN: slab memory manager: created area 0xeb800000 (8012) 2010-06-16 18:52:53 KERN: slab memory manager: created area 0xec000000 (8013) 2010-06-16 18:52:54 KERN: heap_add_area: area 8014 added to small heap 0xccd0c000 - usable range 0xec807000 - 0xecc00000 2010-06-16 18:52:54 KERN: slab memory manager: created area 0xed000000 (8015) 2010-06-16 18:52:59 KERN: slab memory manager: created area 0xed800000 (8016) 2010-06-16 18:53:06 KERN: slab memory manager: created area 0xee000000 (8017) 2010-06-16 18:53:06 KERN: heap_add_area: area 8018 added to small heap 0xccd0c000 - usable range 0xecc07000 - 0xed000000 2010-06-16 18:53:11 KERN: slab memory manager: created area 0xee800000 (8019) 2010-06-16 18:53:12 KERN: slab memory manager: created area 0xef000000 (8020) 2010-06-16 18:53:13 KERN: heap_add_area: area 8021 added to small heap 0xccd0c000 - usable range 0xef807000 - 0xefc00000 2010-06-16 18:53:14 KERN: slab memory manager: created area 0xf0000000 (8022) 2010-06-16 18:53:16 KERN: slab memory manager: created area 0xf0800000 (8023) 2010-06-16 18:53:20 KERN: heap_add_area: area 8024 added to small heap 0xccd0c000 - usable range 0xefc07000 - 0xf0000000 2010-06-16 18:53:21 KERN: slab memory manager: created area 0xf1000000 (8025) 2010-06-16 18:53:23 KERN: slab memory manager: created area 0xf1800000 (8026) 2010-06-16 18:53:25 KERN: heap_add_area: area 8027 added to small heap 0xccd0c000 - usable range 0xf2007000 - 0xf2400000 2010-06-16 18:53:26 KERN: slab memory manager: created area 0xf2800000 (8028) 2010-06-16 18:53:31 KERN: slab memory manager: created area 0xf3000000 (8029) 2010-06-16 18:53:35 KERN: slab memory manager: created area 0xf3800000 (8030) 2010-06-16 18:53:37 KERN: heap_add_area: area 8031 added to small heap 0xccd0c000 - usable range 0xf2407000 - 0xf2800000 2010-06-16 18:53:39 KERN: slab memory manager: created area 0xf4000000 (8032) 2010-06-16 18:53:43 KERN: slab memory manager: created area 0xf4800000 (8033) 2010-06-16 18:53:48 KERN: heap_add_area: area 8034 added to small heap 0xccd0c000 - usable range 0xf5007000 - 0xf5400000 2010-06-16 18:53:48 KERN: slab memory manager: created area 0xf5800000 (8035) 2010-06-16 18:53:53 KERN: slab memory manager: created area 0xf6000000 (8036) 2010-06-16 18:53:57 KERN: slab memory manager: created area 0xf6800000 (8037) 2010-06-16 18:54:00 KERN: heap_add_area: area 8038 added to small heap 0xccd0c000 - usable range 0xf5407000 - 0xf5800000 2010-06-16 18:54:01 KERN: slab memory manager: created area 0xf7000000 (8039) 2010-06-16 18:54:04 KERN: slab memory manager: created area 0xf7800000 (8040) 2010-06-16 18:54:06 KERN: slab memory manager: created area 0xf8000000 (8041) 2010-06-16 18:54:09 KERN: heap_add_area: area 8042 added to small heap 0xccd0c000 - usable range 0xf8807000 - 0xf8c00000 2010-06-16 18:54:10 KERN: slab memory manager: created area 0xf9000000 (8043) 2010-06-16 18:54:14 KERN: slab memory manager: created area 0xf9800000 (8044) 2010-06-16 18:54:19 KERN: heap_add_area: area 8045 added to small heap 0xccd0c000 - usable range 0xf8c07000 - 0xf9000000 2010-06-16 18:54:20 KERN: slab memory manager: created area 0xfa000000 (8046) 2010-06-16 18:54:25 KERN: slab memory manager: created area 0xfa800000 (8047) 2010-06-16 18:54:28 KERN: slab memory manager: created area 0xfb000000 (8048) 2010-06-16 18:54:28 KERN: heap_add_area: area 8049 added to small heap 0xccd0c000 - usable range 0xfb807000 - 0xfbc00000 2010-06-16 18:54:32 KERN: slab memory manager: created area 0xfc000000 (8050) 2010-06-16 18:54:34 KERN: slab memory manager: created area 0xfc800000 (8051) 2010-06-16 18:54:35 KERN: heap_add_area: area 8052 added to small heap 0xccd0c000 - usable range 0xfbc07000 - 0xfc000000 2010-06-16 18:54:37 KERN: slab memory manager: created area 0xfd000000 (8053) 2010-06-16 18:54:39 KERN: slab memory manager: created area 0xfd800000 (8054) 2010-06-16 18:54:41 KERN: heap_add_area: area 8055 added to small heap 0xccd0c000 - usable range 0xfe007000 - 0xfe400000 2010-06-16 18:54:44 KERN: slab memory manager: created area 0xfe800000 (8056) 2010-06-16 18:54:48 KERN: slab memory manager: created area 0xff000000 (8057) 2010-06-16 18:54:49 KERN: heap_add_area: area 8058 added to small heap 0xccd0c000 - usable range 0xfe407000 - 0xfe800000 2010-06-16 18:54:52 KERN: slab memory manager: created area 0x86800000 (8059) 2010-06-16 18:54:55 KERN: slab memory manager: created area 0x87000000 (8060) 2010-06-16 18:54:59 KERN: heap_add_area: area 8061 added to small heap 0xccd0c000 - usable range 0xff807000 - 0xffc00000 2010-06-16 18:54:59 KERN: slab memory manager: created area 0x87800000 (8062) 2010-06-16 18:55:00 KERN: slab memory manager: created area 0x88000000 (8063) 2010-06-16 18:55:02 KERN: heap_add_area: area 8064 added to small heap 0xccd0c000 - usable range 0x86407000 - 0x86800000 2010-06-16 18:55:02 KERN: slab memory manager: created area 0x88800000 (8065) 2010-06-16 18:55:05 KERN: slab memory manager: created area 0x89000000 (8066) 2010-06-16 18:55:09 KERN: heap_add_area: area 8067 added to small heap 0xccd0c000 - usable range 0x89807000 - 0x89c00000 2010-06-16 18:55:10 KERN: slab memory manager: created area 0x8a000000 (8068) 2010-06-16 18:55:15 KERN: slab memory manager: created area 0x8a800000 (8069) 2010-06-16 18:55:20 KERN: slab memory manager: created area 0x8b000000 (8070) 2010-06-16 18:55:21 KERN: heap_add_area: area 8071 added to small heap 0xccd0c000 - usable range 0x89c07000 - 0x8a000000 2010-06-16 18:55:23 KERN: slab memory manager: created area 0x8b800000 (8072) 2010-06-16 18:55:27 KERN: slab memory manager: created area 0x8c000000 (8073) 2010-06-16 18:55:30 KERN: heap_add_area: area 8074 added to small heap 0xccd0c000 - usable range 0x8c807000 - 0x8cc00000 2010-06-16 18:55:30 KERN: slab memory manager: created area 0x8d000000 (8075) 2010-06-16 18:55:34 KERN: slab memory manager: created area 0x8d800000 (8076) 2010-06-16 18:55:38 KERN: slab memory manager: created area 0x8e000000 (8077) 2010-06-16 18:55:40 KERN: heap_add_area: area 8078 added to small heap 0xccd0c000 - usable range 0x8cc07000 - 0x8d000000 2010-06-16 18:55:41 KERN: slab memory manager: created area 0x8e800000 (8079) 2010-06-16 18:55:44 KERN: slab memory manager: created area 0x8f800000 (8081) 2010-06-16 18:55:48 KERN: slab memory manager: created area 0x90000000 (8082) 2010-06-16 18:55:51 KERN: heap_add_area: area 8083 added to small heap 0xccd0c000 - usable range 0x8f407000 - 0x8f800000 2010-06-16 18:55:53 KERN: slab memory manager: created area 0x90800000 (8084) 2010-06-16 18:55:58 KERN: slab memory manager: created area 0x91000000 (8085) 2010-06-16 18:56:03 KERN: slab memory manager: created area 0x91800000 (8086) 2010-06-16 18:56:04 KERN: heap_add_area: area 8087 added to small heap 0xccd0c000 - usable range 0x92007000 - 0x92400000 2010-06-16 18:56:05 KERN: slab memory manager: created area 0x92800000 (8088) 2010-06-16 18:56:07 KERN: slab memory manager: created area 0x93000000 (8089) 2010-06-16 18:56:07 KERN: heap_add_area: area 8090 added to small heap 0xccd0c000 - usable range 0x92407000 - 0x92800000 2010-06-16 18:56:08 KERN: slab memory manager: created area 0x93800000 (8091) 2010-06-16 18:56:09 KERN: slab memory manager: created area 0x94000000 (8092) 2010-06-16 18:56:10 KERN: heap_add_area: area 8093 added to small heap 0xccd0c000 - usable range 0x94807000 - 0x94c00000 2010-06-16 18:56:11 KERN: slab memory manager: created area 0x95000000 (8094) 2010-06-16 18:56:14 KERN: slab memory manager: created area 0x95800000 (8095) 2010-06-16 18:56:15 KERN: heap_add_area: area 8096 added to small heap 0xccd0c000 - usable range 0x94c07000 - 0x95000000 2010-06-16 18:56:18 KERN: slab memory manager: created area 0x96000000 (8097) 2010-06-16 18:56:21 KERN: slab memory manager: created area 0x96800000 (8098) 2010-06-16 18:56:25 KERN: heap_add_area: area 8099 added to small heap 0xccd0c000 - usable range 0x97007000 - 0x97400000 2010-06-16 18:56:25 KERN: slab memory manager: created area 0x97800000 (8100) 2010-06-16 18:56:31 KERN: slab memory manager: created area 0x98000000 (8101) 2010-06-16 18:56:35 KERN: slab memory manager: created area 0x98800000 (8102) 2010-06-16 18:56:36 KERN: heap_add_area: area 8103 added to small heap 0xccd0c000 - usable range 0x97407000 - 0x97800000 2010-06-16 18:56:37 KERN: slab memory manager: created area 0x99000000 (8104) 2010-06-16 18:56:38 KERN: slab memory manager: created area 0x99800000 (8105) 2010-06-16 18:56:40 KERN: slab memory manager: created area 0x9a000000 (8106) 2010-06-16 18:56:41 KERN: heap_add_area: area 8107 added to small heap 0xccd0c000 - usable range 0x9a807000 - 0x9ac00000 2010-06-16 18:56:41 KERN: slab memory manager: created area 0x9b000000 (8108) 2010-06-16 18:56:43 KERN: slab memory manager: created area 0x9b800000 (8109) 2010-06-16 18:56:43 KERN: heap_add_area: area 8110 added to small heap 0xccd0c000 - usable range 0x9ac07000 - 0x9b000000 2010-06-16 18:56:44 KERN: slab memory manager: created area 0x9c000000 (8111) 2010-06-16 18:56:45 KERN: slab memory manager: created area 0x9c800000 (8112) 2010-06-16 18:56:47 KERN: slab memory manager: created area 0x9d000000 (8113) 2010-06-16 18:56:47 KERN: heap_add_area: area 8114 added to small heap 0xccd0c000 - usable range 0x9d807000 - 0x9dc00000 2010-06-16 18:56:48 KERN: slab memory manager: created area 0x9e000000 (8115) 2010-06-16 18:56:50 KERN: slab memory manager: created area 0x9e800000 (8116) 2010-06-16 18:56:50 KERN: heap_add_area: area 8117 added to small heap 0xccd0c000 - usable range 0x9dc07000 - 0x9e000000 2010-06-16 18:56:51 KERN: slab memory manager: created area 0x9f000000 (8118) 2010-06-16 18:56:53 KERN: slab memory manager: created area 0x9f800000 (8119) 2010-06-16 18:56:54 KERN: heap_add_area: area 8120 added to small heap 0xccd0c000 - usable range 0xa0007000 - 0xa0400000 2010-06-16 18:56:55 KERN: slab memory manager: created area 0xa0800000 (8121) 2010-06-16 18:56:57 KERN: slab memory manager: created area 0xa1000000 (8122) 2010-06-16 18:56:57 KERN: heap_add_area: area 8123 added to small heap 0xccd0c000 - usable range 0xa0407000 - 0xa0800000 2010-06-16 18:56:58 KERN: slab memory manager: created area 0xa1800000 (8124) 2010-06-16 18:56:59 KERN: slab memory manager: created area 0xa2000000 (8125) 2010-06-16 18:57:00 KERN: heap_add_area: area 8126 added to small heap 0xccd0c000 - usable range 0xa2807000 - 0xa2c00000 2010-06-16 18:57:01 KERN: slab memory manager: created area 0xa3000000 (8127) 2010-06-16 18:57:02 KERN: slab memory manager: created area 0xa3800000 (8128) 2010-06-16 18:57:04 KERN: slab memory manager: created area 0xa4000000 (8129) 2010-06-16 18:57:04 KERN: heap_add_area: area 8130 added to small heap 0xccd0c000 - usable range 0xa2c07000 - 0xa3000000 2010-06-16 18:57:05 KERN: slab memory manager: created area 0xa4800000 (8131) 2010-06-16 18:57:06 KERN: slab memory manager: created area 0xa5000000 (8132) 2010-06-16 18:57:07 KERN: heap_add_area: area 8133 added to small heap 0xccd0c000 - usable range 0xa5807000 - 0xa5c00000 2010-06-16 18:57:08 KERN: slab memory manager: created area 0xa6000000 (8134) 2010-06-16 18:57:09 KERN: slab memory manager: created area 0xa6800000 (8135) 2010-06-16 18:57:09 KERN: heap_add_area: area 8136 added to small heap 0xccd0c000 - usable range 0xa5c07000 - 0xa6000000 2010-06-16 18:57:11 KERN: slab memory manager: created area 0xa7000000 (8137) 2010-06-16 18:57:12 KERN: slab memory manager: created area 0xa7800000 (8138) 2010-06-16 18:57:12 KERN: heap_add_area: area 8139 added to small heap 0xccd0c000 - usable range 0xa8007000 - 0xa8400000 2010-06-16 18:57:14 KERN: slab memory manager: created area 0xa8800000 (8140) 2010-06-16 18:57:15 KERN: slab memory manager: created area 0xa9000000 (8141) 2010-06-16 18:57:16 KERN: heap_add_area: area 8142 added to small heap 0xccd0c000 - usable range 0xa8407000 - 0xa8800000 2010-06-16 18:57:17 KERN: slab memory manager: created area 0xa9800000 (8143) 2010-06-16 18:57:18 KERN: slab memory manager: created area 0xaa000000 (8144) 2010-06-16 18:57:18 KERN: heap_add_area: area 8145 added to small heap 0xccd0c000 - usable range 0xaa807000 - 0xaac00000 2010-06-16 18:57:19 KERN: slab memory manager: created area 0xab000000 (8146) 2010-06-16 18:57:21 KERN: slab memory manager: created area 0xab800000 (8147) 2010-06-16 18:57:23 KERN: slab memory manager: created area 0xac000000 (8148) 2010-06-16 18:57:24 KERN: heap_add_area: area 8149 added to small heap 0xccd0c000 - usable range 0xaac07000 - 0xab000000 2010-06-16 18:57:24 KERN: slab memory manager: created area 0xac800000 (8150) 2010-06-16 18:57:26 KERN: slab memory manager: created area 0xad000000 (8151) 2010-06-16 18:57:27 KERN: slab memory manager: created area 0xad800000 (8152) 2010-06-16 18:57:28 KERN: heap_add_area: area 8153 added to small heap 0xccd0c000 - usable range 0xae007000 - 0xae400000 2010-06-16 18:57:29 KERN: slab memory manager: created area 0xae800000 (8154) 2010-06-16 18:57:30 KERN: slab memory manager: created area 0xaf000000 (8155) 2010-06-16 18:57:31 KERN: heap_add_area: area 8156 added to small heap 0xccd0c000 - usable range 0xae407000 - 0xae800000 2010-06-16 18:57:31 KERN: slab memory manager: created area 0xaf800000 (8157) 2010-06-16 18:57:33 KERN: slab memory manager: created area 0xb0000000 (8158) 2010-06-16 18:57:35 KERN: slab memory manager: created area 0xb0800000 (8159) 2010-06-16 18:57:35 KERN: heap_add_area: area 8160 added to small heap 0xccd0c000 - usable range 0xb1007000 - 0xb1400000 2010-06-16 18:57:36 KERN: slab memory manager: created area 0xb1800000 (8161) 2010-06-16 18:57:39 KERN: slab memory manager: created area 0xb2000000 (8162) 2010-06-16 18:57:41 KERN: heap_add_area: area 8163 added to small heap 0xccd0c000 - usable range 0xb1407000 - 0xb1800000 2010-06-16 18:57:41 KERN: slab memory manager: created area 0xb2800000 (8164) 2010-06-16 18:57:42 KERN: slab memory manager: created area 0xb3000000 (8165) 2010-06-16 18:57:44 KERN: slab memory manager: created area 0xb3800000 (8166) 2010-06-16 18:57:46 KERN: heap_add_area: area 8167 added to small heap 0xccd0c000 - usable range 0xb4007000 - 0xb4400000 2010-06-16 18:57:47 KERN: slab memory manager: created area 0xb4800000 (8168) 2010-06-16 18:57:51 KERN: slab memory manager: created area 0xb5000000 (8169) 2010-06-16 18:57:55 KERN: slab memory manager: created area 0xb5800000 (8170) 2010-06-16 18:57:58 KERN: heap_add_area: area 8171 added to small heap 0xccd0c000 - usable range 0xb4407000 - 0xb4800000 2010-06-16 18:57:59 KERN: slab memory manager: created area 0xb6000000 (8172) 2010-06-16 18:58:00 KERN: slab memory manager: created area 0xb6800000 (8173) 2010-06-16 18:58:01 KERN: slab memory manager: created area 0xb7000000 (8174) 2010-06-16 18:58:02 KERN: heap_add_area: area 8175 added to small heap 0xccd0c000 - usable range 0xb7807000 - 0xb7c00000 2010-06-16 18:58:03 KERN: slab memory manager: created area 0xb8000000 (8176) 2010-06-16 18:58:04 KERN: slab memory manager: created area 0xb9000000 (8178) 2010-06-16 18:58:06 KERN: slab memory manager: created area 0xb9800000 (8179) 2010-06-16 18:58:06 KERN: heap_add_area: area 8180 added to small heap 0xccd0c000 - usable range 0xb7c07000 - 0xb8000000 2010-06-16 18:58:07 KERN: slab memory manager: created area 0xba000000 (8181) 2010-06-16 18:58:08 KERN: slab memory manager: created area 0xba800000 (8182) 2010-06-16 18:58:09 KERN: heap_add_area: area 8183 added to small heap 0xccd0c000 - usable range 0x8f007000 - 0x8f400000 2010-06-16 18:58:10 KERN: slab memory manager: created area 0xbb000000 (8184) 2010-06-16 18:58:12 KERN: slab memory manager: created area 0xbb800000 (8185) 2010-06-16 18:58:14 KERN: slab memory manager: created area 0xbc000000 (8186) 2010-06-16 18:58:17 KERN: slab memory manager: created area 0xbc800000 (8187) 2010-06-16 18:58:18 KERN: heap_add_area: area 8188 added to small heap 0xccd0c000 - usable range 0xbd007000 - 0xbd400000 2010-06-16 18:58:19 KERN: slab memory manager: created area 0xbd800000 (8189) 2010-06-16 18:58:21 KERN: slab memory manager: created area 0xbe000000 (8190) 2010-06-16 18:58:22 KERN: slab memory manager: created area 0xbe800000 (8191) 2010-06-16 18:58:22 KERN: heap_add_area: area 8192 added to small heap 0xccd0c000 - usable range 0xbd407000 - 0xbd800000 2010-06-16 18:58:23 KERN: slab memory manager: created area 0xbf000000 (8193) 2010-06-16 18:58:25 KERN: slab memory manager: created area 0xbf800000 (8194) 2010-06-16 18:58:27 KERN: slab memory manager: created area 0xc0000000 (8195) 2010-06-16 18:58:27 KERN: heap_add_area: area 8196 added to small heap 0xccd0c000 - usable range 0xc0807000 - 0xc0c00000 2010-06-16 18:58:28 KERN: slab memory manager: created area 0xc1000000 (8197) 2010-06-16 18:58:30 KERN: slab memory manager: created area 0xc1800000 (8198) 2010-06-16 18:58:30 KERN: heap_add_area: area 8199 added to small heap 0xccd0c000 - usable range 0xc0c07000 - 0xc1000000 2010-06-16 18:58:31 KERN: slab memory manager: created area 0xc2000000 (8200) 2010-06-16 18:58:32 KERN: slab memory manager: created area 0xc2800000 (8201) 2010-06-16 18:58:33 KERN: heap_add_area: area 8202 added to small heap 0xccd0c000 - usable range 0xc3007000 - 0xc3400000 2010-06-16 18:58:33 KERN: slab memory manager: created area 0xc3800000 (8203) 2010-06-16 18:58:34 KERN: slab memory manager: created area 0xc4000000 (8204) 2010-06-16 18:58:35 KERN: heap_add_area: area 8205 added to small heap 0xccd0c000 - usable range 0xc3407000 - 0xc3800000 2010-06-16 18:58:35 KERN: slab memory manager: created area 0xc4800000 (8206) 2010-06-16 18:58:36 KERN: slab memory manager: created area 0xc5000000 (8207) 2010-06-16 18:58:38 KERN: slab memory manager: created area 0xc5800000 (8208) 2010-06-16 18:58:39 KERN: heap_add_area: area 8209 added to small heap 0xccd0c000 - usable range 0xc6007000 - 0xc6400000 2010-06-16 18:58:40 KERN: slab memory manager: created area 0xc6800000 (8210) 2010-06-16 18:58:44 KERN: heap_add_area: area 8211 added to small heap 0xccd0c000 - usable range 0xc6407000 - 0xc6800000 2010-06-16 18:58:47 KERN: heap_add_area: area 8212 added to small heap 0xccd0c000 - usable range 0xc7007000 - 0xc7400000 2010-06-16 18:58:49 KERN: heap_add_area: area 8213 added to small heap 0xccd0c000 - usable range 0xc7407000 - 0xc7800000 2010-06-16 18:58:54 KERN: heap_add_area: area 8214 added to small heap 0xccd0c000 - usable range 0xc7807000 - 0xc7c00000 2010-06-16 18:58:57 KERN: heap_add_area: area 8215 added to small heap 0xccd0c000 - usable range 0xc7c07000 - 0xc8000000 2010-06-16 18:59:00 KERN: heap_add_area: area 8216 added to small heap 0xccd0c000 - usable range 0xc8007000 - 0xc8400000 2010-06-16 18:59:04 KERN: heap_add_area: area 8217 added to small heap 0xccd0c000 - usable range 0xc8407000 - 0xc8800000 2010-06-16 18:59:06 KERN: heap_add_area: area 8218 added to small heap 0xccd0c000 - usable range 0xc8807000 - 0xc8c00000 2010-06-16 18:59:09 KERN: heap_add_area: area 8219 added to small heap 0xccd0c000 - usable range 0xc8c07000 - 0xc9000000 2010-06-16 18:59:11 KERN: heap_add_area: area 8220 added to small heap 0x82800000 - usable range 0xc9007000 - 0xc9400000 2010-06-16 18:59:12 KERN: heap_add_area: area 8221 added to small heap 0xccd0c000 - usable range 0xc9407000 - 0xc9800000 2010-06-16 18:59:14 KERN: heap_add_area: area 8222 added to small heap 0xccd0c000 - usable range 0xc9807000 - 0xc9c00000 2010-06-16 18:59:16 KERN: heap_add_area: area 8223 added to small heap 0xccd0c000 - usable range 0xc9c07000 - 0xca000000 2010-06-16 18:59:19 KERN: heap_add_area: area 8224 added to small heap 0xccd0c000 - usable range 0xca007000 - 0xca400000 2010-06-16 18:59:21 KERN: heap_add_area: area 8225 added to small heap 0xccd0c000 - usable range 0xca407000 - 0xca800000 2010-06-16 18:59:26 KERN: heap_add_area: area 8226 added to small heap 0xccd0c000 - usable range 0xca807000 - 0xcac00000 2010-06-16 18:59:30 KERN: heap_add_area: area 8227 added to small heap 0xccd0c000 - usable range 0xcac07000 - 0xcb000000 2010-06-16 18:59:35 KERN: heap_add_area: area 8228 added to small heap 0xccd0c000 - usable range 0xcb007000 - 0xcb400000 2010-06-16 18:59:41 KERN: heap_add_area: area 8229 added to small heap 0xccd0c000 - usable range 0xcb407000 - 0xcb800000 2010-06-16 18:59:44 KERN: heap_add_area: area 8230 added to small heap 0xccd0c000 - usable range 0xcb807000 - 0xcbc00000 2010-06-16 18:59:47 KERN: heap_add_area: area 8231 added to small heap 0xccd0c000 - usable range 0xcbc07000 - 0xcc000000 2010-06-16 18:59:53 KERN: heap_add_area: area 8232 added to small heap 0xccd0c000 - usable range 0xcc007000 - 0xcc400000 2010-06-16 18:59:59 KERN: heap_add_area: area 8233 added to small heap 0xccd0c000 - usable range 0xcc407000 - 0xcc800000 2010-06-16 19:00:02 KERN: heap_add_area: area 8234 added to small heap 0xccd0c000 - usable range 0xcc807000 - 0xccc00000 2010-06-16 19:00:05 KERN: heap_grower: failed to create new heap area 2010-06-16 19:01:17 KERN: Last message repeated 26330 times. 2010-06-16 19:01:17 KERN: PANIC: heap: kernel heap has run out of memory 2010-06-16 19:01:17 KERN: 2010-06-16 19:01:17 KERN: Welcome to Kernel Debugging Land... 2010-06-16 19:01:17 KERN: Thread 1659 "checkfs" running on CPU 1 2010-06-16 19:01:17 KERN: stack trace for thread 1659 "checkfs" 2010-06-16 19:01:17 KERN: kernel stack: 0x810ff000 to 0x81103000 2010-06-16 19:01:17 KERN: user stack: 0x7efef000 to 0x7ffef000 2010-06-16 19:01:17 KERN: frame caller <image>:function + offset 2010-06-16 19:01:17 KERN: 0 81102948 (+ 32) 800e780b <kernel_x86>:arch_debug_stack_trace + 0x000f 2010-06-16 19:01:17 KERN: 1 81102968 (+ 16) 80071c5a <kernel_x86> stack_trace_trampoline(void*: NULL) + 0x000b 2010-06-16 19:01:17 KERN: 2 81102978 (+ 12) 800ecfca <kernel_x86>:arch_debug_call_with_fault_handler + 0x001b 2010-06-16 19:01:17 KERN: 3 81102984 (+ 48) 80072150 <kernel_x86>:debug_call_with_fault_handler + 0x0053 2010-06-16 19:01:17 KERN: 4 811029b4 (+ 64) 8007310b <kernel_x86> kernel_debugger_loop(char const*: [34m0x1[0m [31m"<???>"[0m, char const*: [34m0x8012b8e8[0m [36m"heap: kernel heap has run out of memory 2010-06-16 19:01:17 KERN: "[0m, char*: [34m0x81102a24[0m, int32: [34m-2147011705[0m) + 0x0220 2010-06-16 19:01:17 KERN: 5 811029f4 (+ 48) 800733a5 <kernel_x86> kernel_debugger_internal(char const*: [34m0x1[0m [31m"<???>"[0m, char const*: [34m0x0[0m [31m"<NULL>"[0m, char*: [34m0x81102a44[0m, int32: [34m-2147011211[0m) + 0x0110 2010-06-16 19:01:17 KERN: 6 81102a24 (+ 32) 80073588 <kernel_x86>:panic + 0x0023 2010-06-16 19:01:17 KERN: 7 81102a44 (+ 80) 8004d4c5 <kernel_x86>:memalign + 0x02bd 2010-06-16 19:01:17 KERN: 8 81102a94 (+ 32) 8004d51d <kernel_x86>:malloc + 0x0010 2010-06-16 19:01:17 KERN: 9 81102ab4 (+ 64) 800afc3f <kernel_x86> create_new_vnode_and_lock(int32: [34m-2129646788[0m, int64: [34m-9146763201246909629[0m, vnode*&: [34m0x800b120e[0m, bool&: [34m0x80150200[0m) + 0x0018 2010-06-16 19:01:17 KERN: 10 81102af4 (+ 96) 800b1393 <kernel_x86> get_vnode(int32: [34m-2129646716[0m, int64: [34m4294967297[0m, vnode**: [34m0x588f36[0m, [34mfalse[0m, int32: [34m5803520[0m) + 0x01d6 2010-06-16 19:01:17 KERN: 11 81102b54 (+ 64) 800b2ee8 <kernel_x86>:get_vnode + 0x002d 2010-06-16 19:01:17 KERN: 12 81102b94 (+ 464) 81402b0c <bfs> BlockAllocator<[32m0x830457d8[0m>::CheckNextNode(check_control*: [34m0x81102d84[0m) + 0x0a8e 2010-06-16 19:01:17 KERN: 13 81102d64 (+ 352) 8141bc3c <bfs> bfs_ioctl(fs_volume*: [34m0x828afa78[0m, fs_vnode*: [34m0xcce48ae4[0m, void*: [34m0x828ab5a0[0m, uint32: [34m0x377b[0m ([34m14203[0m), void*: [34m0x7ffeeca8[0m, uint32: [34m0x134[0m ([34m308[0m)) + 0x0113 2010-06-16 19:01:17 KERN: 14 81102ec4 (+ 48) 800ac731 <kernel_x86> common_ioctl(file_descriptor*: [34m0x828af258[0m, uint32: [34m0x377b[0m ([34m14203[0m), void*: [34m0x7ffeeca8[0m, uint32: [34m0x134[0m ([34m308[0m)) + 0x0034 2010-06-16 19:01:17 KERN: 15 81102ef4 (+ 48) 800a3bd8 <kernel_x86> fd_ioctl([34mtrue[0m, int32: [34m308[0m, uint32: [34m0x1[0m ([34m1[0m), void*: NULL, uint32: [34m0x1[0m ([34m1[0m)) + 0x005e 2010-06-16 19:01:17 KERN: 16 81102f24 (+ 32) 800a406e <kernel_x86>:_user_ioctl + 0x0034 2010-06-16 19:01:17 KERN: 17 81102f44 (+ 100) 800ee1b2 <kernel_x86>:handle_syscall + 0x00af 2010-06-16 19:01:17 KERN: user iframe at 0x81102fa8 (end = 0x81103000) 2010-06-16 19:01:17 KERN: eax 0x8b ebx 0x513a28 ecx 0x7ffeec2c edx 0xffff0114 2010-06-16 19:01:17 KERN: esi 0x8c9cf edi 0x0 ebp 0x7ffeec48 esp 0x81102fdc 2010-06-16 19:01:17 KERN: eip 0xffff0114 eflags 0x216 user esp 0x7ffeec2c 2010-06-16 19:01:17 KERN: vector: 0x63, error code: 0x0 2010-06-16 19:01:17 KERN: 18 81102fa8 (+ 0) ffff0114 <commpage>:commpage_syscall + 0x0004 2010-06-16 19:01:17 KERN: 19 7ffeec48 (+ 576) 005d3981 </boot/system/add-ons/disk_systems/bfs@0x005d0000>:unknown + 0x3981 2010-06-16 19:01:17 KERN: 20 7ffeee88 (+ 32) 003a96b7 <libbe.so> BPartition::Delegate<[32m0x1801a820[0m>::Repair([34mfalse[0m) + 0x002b 2010-06-16 19:01:17 KERN: 21 7ffeeea8 (+ 32) 003a8174 <libbe.so> BPartition<[32m0x1801f0e0[0m>::Repair const([34mfalse[0m) + 0x0036 2010-06-16 19:01:17 KERN: 22 7ffeeec8 (+ 160) 00201222 <_APP_>:main + 0x02e7 2010-06-16 19:01:17 KERN: 23 7ffeef68 (+ 52) 00200dcd <_APP_>:_start + 0x0051 2010-06-16 19:01:17 KERN: 24 7ffeef9c (+ 64) 0010534f </boot/system/runtime_loader@0x00100000>:unknown + 0x534f 2010-06-16 19:01:17 KERN: 25 7ffeefdc (+ 0) 7ffeefec 7960:checkfs_main_stack@0x7efef000 + 0xffffec 2010-06-16 19:01:17 KERN: kdebug> avail Available memory: 1363910656/3214311424 bytes 2010-06-16 19:01:17 KERN: kdebug> page?s[1D[1D[1D[1D)[1D[1D_stats page stats: 2010-06-16 19:01:17 KERN: total: 784838 2010-06-16 19:01:17 KERN: active: 745 (busy: 0) 2010-06-16 19:01:17 KERN: inactive: 13207 (busy: 0) 2010-06-16 19:01:17 KERN: cached: 17096 (busy: 0) 2010-06-16 19:01:17 KERN: unused: 142 (busy: 0) 2010-06-16 19:01:17 KERN: wired: 435520 (busy: 0) 2010-06-16 19:01:17 KERN: modified: 0 (busy: 0) 2010-06-16 19:01:17 KERN: free: 300643 2010-06-16 19:01:17 KERN: clear: 17485 2010-06-16 19:01:17 KERN: unreserved free pages: 318128 2010-06-16 19:01:17 KERN: unsatisfied page reservations: 0 2010-06-16 19:01:17 KERN: mapped pages: 439178 2010-06-16 19:01:17 KERN: longest free pages run: 301116 pages (at 483429) 2010-06-16 19:01:17 KERN: longest free/cached pages run: 301117 pages (at 483428) 2010-06-16 19:01:17 KERN: waiting threads: 2010-06-16 19:01:17 KERN: 2010-06-16 19:01:17 KERN: free queue: 0x8015b4b0, count = 300643 2010-06-16 19:01:17 KERN: clear queue: 0x8015b4c4, count = 17485 2010-06-16 19:01:17 KERN: modified queue: 0x8015b488, count = 0 (0 temporary, 0 swappable, inactive: 0) 2010-06-16 19:01:17 KERN: active queue: 0x8015b460, count = 745 2010-06-16 19:01:17 KERN: inactive queue: 0x8015b474, count = 13207 2010-06-16 19:01:17 KERN: cached queue: 0x8015b49c, count = 17096 2010-06-16 19:01:17 KERN: kdebug> d[1D[1Dswapswap files: 2010-06-16 19:01:17 KERN: 2010-06-16 19:01:17 KERN: swap space in pages: 2010-06-16 19:01:17 KERN: total: 0 2010-06-16 19:01:17 KERN: available: 0 2010-06-16 19:01:17 KERN: reserved: 0 2010-06-16 19:01:17 KERN: used: 0 2010-06-16 19:01:17 KERN: free: 0
comment:3 by , 14 years ago
| Owner: | changed from to |
|---|---|
| Status: | new → assigned |
I've also run into this during the last few days while testing PAE on a box with 4 GB RAM. After a dd if=/dev/zero of=tt bs=1M count=3500 I started copying two haiku trees (with objects, weighing in at about 3.6 GB) from one partition to another. The copy process started relatively slow -- 2 to 3 MB/s -- and after a while dropped to abysmally slow -- < 100 KB/s. I didn't always let the copy process run on, but twice I let it run until it hit the out of memory panic. The syslog said pretty much the same as in the previous comment: The slab and heap would grow until the heap grower failed to allocate new areas. The reason being lack of address space in my case and apparently in the above one, too.
As can be seen in the output, the slab allocations stop at some point -- probably when the address space low resource state reaches "note" and the block cache stops allocated new blocks -- and only the heap continues to grow -- probably for vnodes and related structures. That is suboptimal already -- currently the kernel favors vnodes over block cache over file cache. Still it doesn't explain why slab areas aren't freed at some point. At low resource state "critical" the block cache should go all out and throw away lots of blocks. Also, that doesn't explain the amazingly poor copying performance.
Will try to analyze a bit.
comment:4 by , 14 years ago
| Blocking: | 4963 added |
|---|
comment:5 by , 14 years ago
Just finished running the test again with some more debug output. The low resource state transitions are logged and block_cache::_LowMemoryHandler() prints how the RemoveUnusedBlocks() changed the number of unused blocks. The test started again at about 2-3 MB/s and -- I hadn't noticed this before -- after a few seconds reaches even > 10 MB/s for a short time. Afterwards it varies from 1 to 10 MB/s until the address space "note" low resource state is reached. Then the throughput drops immediately to < 1 MB, varying from 10 KB/s to a few 100 KB/s, only from time to time recovering to 1 or 2 MB/s (sometimes even more) for a few seconds.
Here's the output from around the point where the low resource limit is hit:
heap_add_area: area 3649 added to small heap 0xd090c000 - usable range 0xc2c07000 - 0xc3000000 slab memory manager: created area 0xc5800000 (3650) slab memory manager: created area 0xc6000000 (3651) low resource address space: normal -> note block_cache::_LowMemoryHandler(): 0xcd92a198: unused: 25997 -> 25997 block_cache::_LowMemoryHandler(): 0xcd92a7f8: unused: 3 -> 0 block_cache::_LowMemoryHandler(): 0xccd27b28: unused: 313434 -> 313384 block_cache::_LowMemoryHandler(): 0xcfd1f660: unused: 13 -> 0 block_cache::_LowMemoryHandler(): 0xcd92a198: unused: 2598 -> 2598 block_cache::_LowMemoryHandler(): 0xcd92a7f8: unused: 0 -> 0 block_cache::_LowMemoryHandler(): 0xccd27b28: unused: 311848 -> 311798 block_cache::_LowMemoryHandler(): 0xcfd1f660: unused: 0 -> 0 block_cache::_LowMemoryHandler(): 0xcd92a198: unused: 22 -> 22 block_cache::_LowMemoryHandler(): 0xcd92a7f8: unused: 0 -> 0 block_cache::_LowMemoryHandler(): 0xccd27b28: unused: 311378 -> 311328 block_cache::_LowMemoryHandler(): 0xcfd1f660: unused: 0 -> 0 block_cache::_LowMemoryHandler(): 0xcd92a198: unused: 0 -> 0 block_cache::_LowMemoryHandler(): 0xcd92a7f8: unused: 0 -> 0 block_cache::_LowMemoryHandler(): 0xccd27b28: unused: 311244 -> 311194 block_cache::_LowMemoryHandler(): 0xcfd1f660: unused: 0 -> 0
The first block cache (0xcd92a198) is the one of the volume I'm copying, the third one (0xccd27b28) from the target volume (both on the same disk). As can be seen Tracker's pre-flight phase has let the number of unused blocks of the source volume grow to > 300000. The 25997 blocks of the target volume are probably mostly the ones that have already been written.
After only three seconds the target cache has been drained completely. RemoveUnusedBlocks() didn't remove a single block, though, since at level "note" only blocks older than two minutes are removed. The reason that all blocks are removed nonetheless is block_cache::Allocate(), which removes 2 unused blocks each time it is called. And since the target volume is quite active -- lots of writing and transactions going on -- that happens quite often. This would also explain the abysmal performance from that point on, since all blocks that become unused even for a short time are thrown away and have to be re-read from disk when they are needed again. The low resource behavior definitely needs to be rethought.
The source volume cache lost at max a few hundred blocks per second. It took about 1:45 h to drop to 0. Interestingly the whole time not a single slab are was released -- which must be a bug/leak somewhere -- and the heap kept growing -- likely for vnodes, file caches and BFS data structures, which don't care about the address space state. After about 2:20 h (80 % copied) the heap grower failed to allocate a heap area.
kdebug> low_resource current state: ---a pages: normal (723271) memory: normal (3025899520) sems: normal (64352) aspace: note (28450816) function data resources prio function-name 0x800df218 0x00000000 p m a 5 object_cache_low_memory__FPvUll 0x800bad8c 0x00000000 p m 0 vnode_low_resource_handler__FPvUll 0x8003d044 0xcd92a198 p m a 0 _LowMemoryHandler__11block_cachePvUll 0x8003d044 0xcd92a7f8 p m a 0 _LowMemoryHandler__11block_cachePvUll 0x8003d044 0xccd27b28 p m a 0 _LowMemoryHandler__11block_cachePvUll 0x8003d044 0xcfd1f660 p m a 0 _LowMemoryHandler__11block_cachePvUll
Since there are still 28 MB address space left (probably fragmented or partially reserved), the address space low resource state is still "note". "Warning" state is currently reached at 8 MB, "critical" at utterly useless 32 KB. This is obviously something that needs to be changed.
As for the block caches:
kdebug> block_cache 0xcd92a198 BLOCK CACHE: 0xcd92a198 fd: 0 max_blocks: 26214048 block_size: 2048 next_transaction_id: 198 buffer_cache: 0x828a1dc0 busy_reading: 0, no waiters busy_writing: 0, no waiters 797 blocks total, 311 dirty, 0 discarded, 0 referenced, 0 busy, 486 in unused. kdebug> block_cache 0xccd27b28 BLOCK CACHE: 0xccd27b28 fd: 7 max_blocks: 26214048 block_size: 2048 next_transaction_id: 4 buffer_cache: 0x828a6a00 busy_reading: 0, no waiters busy_writing: 0, no waiters 4 blocks total, 3 dirty, 0 discarded, 0 referenced, 0 busy, 1 in unused.
They don't know a lot of blocks.
The slabs:
kdebug> slabs
address name objsize usage empty usedobj total flags
[...]
0x82808d00 cached blocks 64 12124160 178 814 189255 20000000
0x828a1dc0 block cache buffers 2048 122683392 192 1622 59904 20000000
0x828a5000 block cache buffers 4096 524288 0 2 128 20000000
0x828a6a00 block cache buffers 2048 642777088 1225 11 313856 20000000
0x828a6280 block cache buffers 2048 524288 1 0 256 20000000
kdebug> slab_cache 0x82808d00
name: cached blocks
lock: 0x82808d28
object_size: 64
cache_color_cycle: 0
total_objects: 189255
used_count: 814
empty_count: 178
pressure: 354
slab_size: 65536
usage: 12124160
maximum: 0
flags: 0x20000000
cookie: 0x00000000
resize entry don't wait: 0x00000000
resize entry can wait: 0x00000000
slab chunk size used offset free
empty:
0xe902ffe4 0xe9020000 1023 1023 0 0xe90217fc
0xe804ffe4 0xe8040000 1023 1023 0 0xe8044afc
0x82efffe4 0x82ef0000 1023 1023 0 0x82efbebc
[...]
partial:
0xda05ffe4 0xda050000 1023 491 0 0xda057cbc
0xe085ffe4 0xe0850000 1023 1011 0 0xe085733c
0x8291ffe4 0x82910000 1023 1010 0 0x8291927c
0xdf04ffe4 0xdf040000 1023 1018 0 0xdf047ebc
0xdd81ffe4 0xdd810000 1023 926 0 0xdd8151bc
0xd905ffe4 0xd9050000 1023 944 0 0xd905eafc
0xe105ffe4 0xe1050000 1023 947 0 0xe105773c
depot:
full: 0x00000000, count 16
empty: 0x00000000, count 0
max full: 16
capacity: 32
stores:
[0] loaded: 0x00000000
previous: 0x00000000
[1] loaded: 0x00000000
previous: 0x00000000
[2] loaded: 0x828ce140
previous: 0x00000000
[3] loaded: 0x00000000
previous: 0x00000000
[4] loaded: 0x828ce780
previous: 0x00000000
[5] loaded: 0x828ceaa0
previous: 0x00000000
[6] loaded: 0x00000000
previous: 0x00000000
[7] loaded: 0x828ce8c0
previous: 0x00000000
kdebug> slab_cache 0x828a6a00
name: block cache buffers
lock: 0x828a6a28
object_size: 2048
cache_color_cycle: 0
total_objects: 313856
used_count: 11
empty_count: 1225
pressure: 2449
slab_size: 524288
usage: 642777088
maximum: 0
flags: 0x20000000
cookie: 0x00000000
resize entry don't wait: 0x00000000
resize entry can wait: 0x00000000
slab chunk size used offset free
empty:
0x828af300 0xde200000 256 256 0 0xde27e7fc
0x828af240 0xde500000 256 256 0 0xde559ffc
0x828af1c0 0xde700000 256 256 0 0xde730ffc
[...]
partial:
0x828059e0 0x82d00000 256 245 0 0x82d16ffc
depot:
full: 0x00000000, count 4
empty: 0x00000000, count 0
max full: 4
capacity: 8
stores:
[0] loaded: 0x00000000
previous: 0x00000000
[1] loaded: 0x00000000
previous: 0x00000000
[2] loaded: 0x00000000
previous: 0x00000000
[3] loaded: 0x00000000
previous: 0x00000000
[4] loaded: 0x00000000
previous: 0x00000000
[5] loaded: 0x00000000
previous: 0x00000000
[6] loaded: 0x00000000
previous: 0x00000000
[7] loaded: 0x00000000
previous: 0x00000000
Truncated the free slabs lists -- there are loads of them. Apparently something goes wrong in the slab allocator's low resource handler. Will look into this tomorrow.
follow-up: 7 comment:6 by , 14 years ago
The throwing away of blocks in block_cache::Allocate() is more of a work-around to this problem than a solution.
You should be able to reproduce the problem with a checkfs run much faster, though :-)
comment:7 by , 14 years ago
Replying to axeld:
The throwing away of blocks in block_cache::Allocate() is more of a work-around to this problem than a solution.
Given that it is probably responsible for the poor performance, it is definitely not a solution. :-)
You should be able to reproduce the problem with a checkfs run much faster, though :-)
I've never seen this with checkfs. There's perhaps not enough data on the partition (a bit less than 300000 nodes IIRC).
comment:8 by , 14 years ago
Some additional output in object_cache_low_memory() makes it clear what is happening:
low resource address space: normal -> note object_cache_low_memory(): cache 0x82808d00: pressure: 356, empty: 0 object_cache_low_memory(): cache 0x828a1dc0: pressure: 397, empty: 2 object_cache_low_memory(): cache 0x828a6140: pressure: 2449, empty: 0 block_cache::_LowMemoryHandler(): 0xd1526000: unused: 10 -> 10 block_cache::_LowMemoryHandler(): 0x83023594: unused: 3 -> 0 block_cache::_LowMemoryHandler(): 0xd1526594: unused: 313434 -> 313384 block_cache::_LowMemoryHandler(): 0xd0921198: unused: 13 -> 0 object_cache_low_memory(): cache 0x82808d00: pressure: 356, empty: 0 object_cache_low_memory(): cache 0x828a1dc0: pressure: 397, empty: 9 object_cache_low_memory(): cache 0x828a6140: pressure: 2449, empty: 0 block_cache::_LowMemoryHandler(): 0xd1526000: unused: 3156 -> 3156 block_cache::_LowMemoryHandler(): 0x83023594: unused: 0 -> 0 block_cache::_LowMemoryHandler(): 0xd1526594: unused: 313054 -> 313004 block_cache::_LowMemoryHandler(): 0xd0921198: unused: 0 -> 0 object_cache_low_memory(): cache 0x82808d00: pressure: 356, empty: 3 object_cache_low_memory(): cache 0x828a1dc0: pressure: 397, empty: 141 object_cache_low_memory(): cache 0x828a6140: pressure: 2449, empty: 0 block_cache::_LowMemoryHandler(): 0xd1526000: unused: 0 -> 0 block_cache::_LowMemoryHandler(): 0x83023594: unused: 0 -> 0 block_cache::_LowMemoryHandler(): 0xd1526594: unused: 312768 -> 312718 block_cache::_LowMemoryHandler(): 0xd0921198: unused: 0 -> 0
ObjectCache::pressure is only decreased when the low resource level reaches at least "warning" (which never happens during the test), so the empty slab counts have a fixed upper limit (e.g. 1225 for the source volume block cache).
Reviewing the figures from the previous run, the slabs use less than 800 MB, BTW. Fragmentation of the slab areas might translate that to quite a bit more, but I guess the real problem is that the heap is continuously growing -- i.e. even if the slabs were drained completely, the address space limit would be hit eventually. It also doesn't help that heap and slab have different allocation backends (IOW that the slab allocator is not used as the heap).
comment:9 by , 14 years ago
| Resolution: | → fixed |
|---|---|
| Status: | assigned → closed |
Fixed:
- the way too low "warning" and "critical" low resource state address space limits in hrev37168,
- the vnode cache ignoring the address space low resource state in hrev37169,
- extremely quick draining of actively written to block caches (and thus the performance issue) when in "note" low resource state in hrev37170,
- the rather slow reduction of unused blocks of huge block caches when reaching a low resource state also in hrev37170,
- the slab object caches keeping a high number of empty slabs in "note" low resource state in hrev37171.
Not fixed yet is that vnode and block caches are favored over file cache and IOCache. I haven't tested fscheck -- as mentioned earlier, I couldn't reproduce the problem with it -- but at least my copy test runs smoothly, now. Please reopen, if there are still memory related issues with fscheck.
Note:
See TracTickets
for help on using tickets.
Can confirm this on a volume with >500k files on it. Haven't just found the time filing the issue. (4GB (3GB) machine here)