RAM 1GB of 4GB reported used after boot

[michael.weirauch]

hrev37154 trunk gcc4h (ThinkPad T500 NK13AGE; 4GB RAM)

System reports more then 1GB RAM used in userland. (AboutSystem, sysinfo)

sysinfo:

Kernel name: kernel_x86 built on: Jun 16 2010 17:10:09 version 0x1
2 Intel Core 2, revision 4676 running at 2792MHz (ID: 0x00000000 0x00000000)

CPU #0: "Intel(R) Core(TM)2 Duo CPU     T9600  @ 2.80GHz"
	Type 0, family 6, model 23, stepping 6, features 0xbfebfbff
		FPU VME DE PSE TSC MSR PAE MCE CX8 APIC SEP MTRR PGE MCA CMOV PAT
		PSE36 CFLUSH DS ACPI MMX FXSTR SSE SSE2 SS HTT TM PBE
	Extended Intel: 0x0008e3fd
		SSE3 DTES64 MONITOR DS-CPL VMX SMX EST TM2 SSSE3 CX16 xTPR PDCM SSE4.1
	Extended AMD: type 0, family 0, model 0, stepping 0, features 0x20100000
		NX 64
	Power Management Features:

	L2 Data cache fully associative, 1 lines/tag, 64 bytes/line
	L2 cache: 6144 KB, 16-way set associative, 0 lines/tag, 64 bytes/line

	Inst TLB: 2M-bytes pages, 4-way set associative, 8 entries OR 4M, 4-way, 4 entries
	Inst TLB: 4K-bytes pages, 4-way set associative, 128 entries
	Data TLB: 4M-byte pages, 4-way set associative, 32 entries
	64-byte Prefetching
	L1 Data TLB: 4K-bytes pages, 4-way set associative, 16 entries
	L1 Data TLB: 4M-bytes pages, 4-way set associative, 16 entries
	L2 cache: 6144 KB, 24-way set associative, 64 bytes/line
	L1 inst cache: 32 KB, 8-way set associative, 64 bytes/line
	Data TLB: 4K-bytes pages, 4-way set associative, 256 entries
	L1 data cache: 32 KB, 8-way set associative, 64 bytes/line

CPU #1: "Intel(R) Core(TM)2 Duo CPU     T9600  @ 2.80GHz"
	Type 0, family 6, model 23, stepping 6, features 0xbfebfbff
		FPU VME DE PSE TSC MSR PAE MCE CX8 APIC SEP MTRR PGE MCA CMOV PAT
		PSE36 CFLUSH DS ACPI MMX FXSTR SSE SSE2 SS HTT TM PBE
	Extended Intel: 0x0008e3fd
		SSE3 DTES64 MONITOR DS-CPL VMX SMX EST TM2 SSSE3 CX16 xTPR PDCM SSE4.1
	Extended AMD: type 0, family 0, model 0, stepping 0, features 0x20100000
		NX 64
	Power Management Features:

	L2 Inst cache fully associative, 1 lines/tag, 64 bytes/line
	L2 cache: 6144 KB, 16-way set associative, 0 lines/tag, 64 bytes/line

	Inst TLB: 2M-bytes pages, 4-way set associative, 8 entries OR 4M, 4-way, 4 entries
	Inst TLB: 4K-bytes pages, 4-way set associative, 128 entries
	Data TLB: 4M-byte pages, 4-way set associative, 32 entries
	64-byte Prefetching
	L1 Data TLB: 4K-bytes pages, 4-way set associative, 16 entries
	L1 Data TLB: 4M-bytes pages, 4-way set associative, 16 entries
	L2 cache: 6144 KB, 24-way set associative, 64 bytes/line
	L1 inst cache: 32 KB, 8-way set associative, 64 bytes/line
	Data TLB: 4K-bytes pages, 4-way set associative, 256 entries
	L1 data cache: 32 KB, 8-way set associative, 64 bytes/line

3028275200 bytes free      (used/max 1193730048 / 4222005248)
                           (cached    166764544)
     61897 semaphores free (used/max       3639 /      65536)
      3738 ports free      (used/max        358 /       4096)
      3814 threads free    (used/max        282 /       4096)
      2025 teams free      (used/max         23 /       2048)

avail, page_stats, swap:

2010-06-16 19:21:58 KERN: kdebug> availAvailable memory: 2937016320/3214311424 bytes
2010-06-16 19:21:58 KERN: kdebug> page?[1D[1D_statspage stats:
2010-06-16 19:21:58 KERN: total: 784838
2010-06-16 19:21:58 KERN: active: 13517 (busy: 0)
2010-06-16 19:21:58 KERN: inactive: 32189 (busy: 0)
2010-06-16 19:21:58 KERN: cached: 13463 (busy: 0)
2010-06-16 19:21:58 KERN: unused: 142 (busy: 0)
2010-06-16 19:21:58 KERN: wired: 24105 (busy: 0)
2010-06-16 19:21:58 KERN: modified: 0 (busy: 0)
2010-06-16 19:21:58 KERN: free: 578350
2010-06-16 19:21:58 KERN: clear: 123072
2010-06-16 19:21:58 KERN: unreserved free pages: 701422
2010-06-16 19:21:58 KERN: unsatisfied page reservations: 0
2010-06-16 19:21:58 KERN: mapped pages: 43925
2010-06-16 19:21:58 KERN: longest free pages run: 579668 pages (at 204877)
2010-06-16 19:21:58 KERN: longest free/cached pages run: 579668 pages (at 204877)
2010-06-16 19:21:58 KERN: waiting threads:
2010-06-16 19:21:58 KERN: 
2010-06-16 19:21:58 KERN: free queue: 0x8015b4b0, count = 578350
2010-06-16 19:21:58 KERN: clear queue: 0x8015b4c4, count = 123072
2010-06-16 19:21:58 KERN: modified queue: 0x8015b488, count = 0 (0 temporary, 0 swappable, inactive: 0)
2010-06-16 19:21:58 KERN: active queue: 0x8015b460, count = 13517
2010-06-16 19:21:58 KERN: inactive queue: 0x8015b474, count = 32189
2010-06-16 19:21:58 KERN: cached queue: 0x8015b49c, count = 13463
2010-06-16 19:21:58 KERN: kdebug> swapswap files:
2010-06-16 19:21:58 KERN: 
2010-06-16 19:21:58 KERN: swap space in pages:
2010-06-16 19:21:58 KERN: total:             0
2010-06-16 19:21:58 KERN: available:         0
2010-06-16 19:21:58 KERN: reserved:          0
2010-06-16 19:21:58 KERN: used:              0
2010-06-16 19:21:58 KERN: free:              0

[bonefish]

That's actually a feature, not a bug. Without PAE (Physical Address Extension) enabled (which it isn't by default), the processor can only access physical addresses up to 4 GB. Since graphics memory and registers of devices are also mapped in the 4 GB range, some of your RAM is beyond that limit and cannot be used. Until recently (hrev37117) that memory would be ignored completely (i.e. AboutSystem and others would display 3.2 GB or something like that as total available memory). Now the total available memory is displayed correctly and the non-accessible memory is considered used. It could be considered as free, but that's even less correct, since it can't be used. Cf. ticket #6124.

[michael.weirauch]

Thx for the explanation.

[mmlr]

Replying to bonefish:

That's actually a feature, not a bug. Without PAE (Physical Address Extension) enabled (which it isn't by default), the processor can only access physical addresses up to 4 GB. Since graphics memory and registers of devices are also mapped in the 4 GB range, some of your RAM is beyond that limit and cannot be used. Until recently (hrev37117) that memory would be ignored completely (i.e. AboutSystem and others would display 3.2 GB or something like that as total available memory). Now the total available memory is displayed correctly and the non-accessible memory is considered used. It could be considered as free, but that's even less correct, since it can't be used.

I somehow find that this gives a rather inaccurate picture though and makes the used memory figure pretty much useless. Considering that the RAM that is not addressable isn't actually available to the OS the previous way of displaying makes far more sense to me (actually showing "available" memory). With these changes you simply put up the impression that the OS uses a lot of memory even though it really doesn't, hence you can't really tell how much memory is actually "used" by the system. You can tell how much memory is still available with either way of displaying (3.1GB out of 4GB vs. 3.1GB out of 3.2GB when 0.1GB is actually used) so there's no real disadvantage in the previous way to handle it IMO.

[Wim]

How about reporting inaccessible memory separately?

[axeld]

That would be indeed much better IMO.

[bonefish]

I pretty much don't care. Whoever wants to change it, ​vm_page_get_stats() is where it all happens. To report the ignored memory back to userland the system_info structure has to be extended. Adding a field at the end would help to do that in a (mostly) binary compatible way. ​_user_get_system_info() needs to be changed to copy no more than requested back to userland, though.

[Wim]

Well, I care :-)

Reopening, I will try to prepare a patch in the next couple of days.

[Wim]

Replying to bonefish:

​_user_get_system_info() needs to be changed to copy no more than requested back to userland, though.

I don't understand this. Within this function the size parameter is only used in the following line:

status_t status = _get_system_info(&info, size);

and this call returns B_BAD_VALUE anyway when size != sizeof(system_info). It seems to me that the size parameter is pretty useless.

I am attaching a patch that I think should be sufficient. I don't dare to make the commit myself in fear of the wrath of the Haiku kernel developers :D

After this patch is accepted I will work on the AboutSystem app to report inaccessible memory. sysinfo will revert to its old behavior, which is probably better.

[Wim]

Separatly report the ignored memory instead of adding it to the max_pages and used_pages

[bonefish]

Replying to Wim:

Replying to bonefish:

​_user_get_system_info() needs to be changed to copy no more than requested back to userland, though.

I don't understand this. Within this function the size parameter is only used in the following line:

status_t status = _get_system_info(&info, size);

and this call returns B_BAD_VALUE anyway when size != sizeof(system_info). It seems to me that the size parameter is pretty useless.

The size parameter was intended for future backward binary compatibility. I didn't check whether there was still padding in the structure. But once the padding is used up and the structure is extended, the size parameter would imply the version of the structure. The check in _get_system_info() would have to be changed. Since Haiku supports ELF symbol versioning, that can be done simply by implementing a compatibility version of the function instead.

I am attaching a patch that I think should be sufficient. I don't dare to make the commit myself in fear of the wrath of the Haiku kernel developers :D

The patch looks good. Please commit.

After this patch is accepted I will work on the AboutSystem app to report inaccessible memory. sysinfo will revert to its old behavior, which is probably better.

sysinfo could print the ignored value as well.

[Wim]

Thank you for reviewing the patch, bonefish. Applied in hrev37175.