Failure to boot with wrong reporting too much RAM

[eu]

Burned latest x86_64 nightly image rev 54815 on USB stick and tried to boot on two UEFI machines:

• AMD Ryzen 3970X with 128GB of RAM
• Intel NUC10i7FN with 64GB of RAM

On both machines booting stops with message:

* PANIC * Can't currently support more than 512GB of RAM!

Something must be wrong when installed memory is detected.

[waddlesplash]

This is probably related to a just-introduced feature to support machines with up to 512GB of RAM. Clearly, however, some part of it does not work :)

[eu]

Last time a successfully boot Haiku it was a few years ago on a non-UEFI machine.

After a few more tries the Intel NUC with 64 GB of RAM booting process goes past that message but stops on the Haiku screen without lighting any of the icons. The AMD doesn't go beyond the PANIC message. UEFI booting clearly has glitches.

[waddlesplash]

Plenty of users and developers (myself included) boot Haiku exclusively via EFI without issues, so this is almost certainly a recent regression related to the addition of LA57 paging support. If korli does not come up with a fix soon, we can revert the patches until he does.

In the meantime, can you try with R1/beta2, which does not have those patches, and thus should not have these problems?

[korli]

well, keep cool, hrev54815 predates the kernel and bootloader hrev54816 changes :)

[waddlesplash]

In that case, this is probably related to: ​https://github.com/haiku/haiku/blob/master/src/system/boot/platform/efi/mmu.cpp#L119

I wonder why EFI is returning such addresses, possibly they are memory-map regions or something like that?

[eu]

Tried R1/beta2 (rev 54154) and also the latest nightly rev 54821. Both of them behave the same way:

• On 128GB RAM machine it stops with * PANIC * Can't currently support more than 512GB of RAM.
• On 64 GB RAM machine it stops at Haiku screen without lighting any of the 7 icons.

It looks like the 512GB message is not new, being there for quite a while!

What quantity of RAM are other people successfully booting?

Is there a debug option or version that shows more messages on screen during boot so we can pinpoint where it stops?

[waddlesplash]

I run Haiku regularly on a machine with 32GB of RAM.

Based on the message in the EFI loader, it appears the problem is with addresses above the 512GB boundary, which may not mean more than 512GB itself. CC'ing jessicah, tqh, kallisti5 to see if any of them have an idea about what we are supposed to do here.

There are options for viewing the bootloader log or onscreen debug output in the loader menus (spam the spacebar after your BIOS screen to get into that, if you can.)

[jessicah]

It would be worthwhile updating the panic section to print out the address that UEFI returned, and the size that's being requested to allocate, as it's probably a call from somewhere else trying to allocate something weird.

[eu]

Spamming the spacebar is not working, neither keeping it pressed to get some debug output.

Wondering if instead of getting early into graphical more (the Haiku screen with icons) it would be doable to have an option (like a key pressed during boot) that keeps the process in text mode, spitting out all the messages on screen, similar lo Linux or FreeBSD.

[waddlesplash]

Yes, you can enable that via the boot options menu, which is accessible (under EFI) by spamming the spacebar while Haiku's bootloader starts (or when using legacy BIOS, by holding the Shift key.) However, this particular message may be occurring in EFI loader startup before the menu would be shown, but I did not look into whether or not that is the case.

[eu]

Maybe I should mention that the 128GB AMD machine has an abnormally high number of disks, some of them new NVME SSDs, a couple SATA SSDs and many SATA HDDs, one used as boot disk (with an EFI partition) and others used in ZFS pools.

The 64GB Intel machine is more down to earth, with a NVME SSD containing an EFI partition and a bunch of other partitions for Windows, Linux and FreeBSD, plus a SATA SSD for data.

Maybe this could also be a reason why the boot process gets stuck.

[waddlesplash]

An abnormally high number of disks could potentially mean more PCI space used, which could cause interesting things to happen to the memory map, indeed. More partitions on one disk should not cause such things, though, so it is indeed strange the Intel machine also has this same problem.

[tqh]

First of all, how big is the USB stick? A stick that is too small will fail in very weird ways.

How much GPU memory do you have?

As the linked code shows, we get an address from UEFI and add the requested size to check if it is less than 512GB. This suggests that size is either -1 or corrupted, or your your machines have weird fw returning very high addresses. (High addresses are unlikely, as that would not follow UEFI spec).

I recommend re-downloading, and using a different USB stick as a first check.

[eu]

I already had a hunch about the USB stick being the issue, so I tried 3 of them, one 8GB and two 16GB, all different brands. So, that's out.

GPU is an old NVIDIA GTX TITAN with 6GB.

If both machines return odd UEFI addresses and variables then how three other operating systems function perfectly fine? I don't have programming expertise, but the fact that Haiku UEFI booting works on some machines and fails on others kind of implies that Haiku's implementation may be lacking.

[eu]

tqh' insight about weird addresses combined with jessicah' suggestion of printing/displaying the variables involved may help finding the problem.

[tqh]

It is weird. Most sizes are from ELF, kernel and drivers so should not be big, and addresses being close to 512GB doesn't make sense either. So if it is not the stick, it may be the download or how you copy it to the stick.

I see we have two other places where the same message is used. One in BIOS booting, and one in an arch x86_64 specific memory check, and I think the arch one looks strange: ​https://github.com/haiku/haiku/blob/d0e006f5569f24e73a4d83f5c9766f1132160d7a/src/system/boot/platform/efi/arch/x86_64/arch_mmu.cpp#L194

[eu]

First thing I do for all download images is verify their checksum. Then I use dd if=file.iso of=/dev/sdx bs=1M to burn on the stick.

Looking at that line 119 I see a "panic" function that takes a string argument. If that string would be pre-formated to also contain the values of all those variables we may be able to find the issue. That panic message would only be seen on machines that have the problem, not on those that boot fine.

[tqh]

Yes, we need to get some more data, but with the info you've given we can rule out most problems with the usb-stick. Not sure if we have checksums on downloads so it can be verified, but I think the arch code is worth investigating further.

[eu]

I download latest nightly image from ​https://download.haiku-os.org/nightly-images/x86_64 Checksum is provided for every image.

[eu]

boot/platform/efi/mmu.cpp has the 512GB panic message in two places, lines 123 and 180, so an extra complication to find out which of the two is triggered.

[waddlesplash]

I closed #17586 as a duplicate of this ticket. Has anyone looked into this code in the EFI loader since, to see if we can handle addresses above 512GB? Or determined that something else is going on?

[eu]

Alternatively, maybe somebody could let me know if I could help testing somehow on my machines, but please keep in mind that I do not have programming experience.

[eu]

And also regarding waddlesplash last comment: it's not about handling addresses above 512GB, it's maybe about why that message is issued when it is not the case, for 64GB or 128GB.

[cocobean]

eu: attach full syslog to this ticket from 64GB Intel machine and 128GB AMD machine. Devs can tell you the best way to obtain the info. If you can get into the bootmenu, you can page through the syslog messages and other options.

[eu]

It does not get into the bootmenu, just stops at that message about 512GB right after I see those five or six icons.

I'm testing intermittently, every few weeks, the latest nightly anyboot image. I even built Haiku myself using your Building Haiku guide, both the image and direct installation on a disk partition. All of them of course stop at that message.

[eu]

In system/boot/platform/efi/mmu.cpp line 119:

if (addr + size > (512ull * 1024 * 1024 * 1024))

panic("Can't currently support more than 512GB of RAM!");

It's obvious that addr and/or size variables get wrong values.

[eu]

Same message is in system/boot/platform/efi/arch/x86_64/arch_mmu.cpp line 201:

if (maxAddress / 0x40000000 > 512)

panic("Can't currently support more than 512GB of RAM!");

maxAddress variable could get somehow a wrong variable here.

Obviously I can't know which of the 2 lines of code trigger the message in my case.

[cocobean]

1. 512 GiB RAM limiter(s) -> (maxAddress / 0x40000000 > 512)
2. Tap the Spacebar or Shift keys during initial boot up after BIOS screen (motherboard) - test it for Haiku boot menu entry. Cold start computer, if needed (unplug power from Power supply for 30 secs).

Note (theoretical):

1. 0x8000000000 / 0x40000000 = 512
2. 0x2000000000 / 0x40000000 = 128
3. 0x1000000000 / 0x40000000 = 64
4. 0x800000000 / 0x40000000 = 32

[eu]

Left Shift does nothing.

[eu]

Secure Boot is disabled on both machines.

[madmax]

If you are OK downloading modified unofficial software from some random guy on the internet, ​here is a version with modified messages. Or, as you've already built Haiku yourself, you may try to apply the changes from the diff file.

[eu]

I can access that link but when I try to download anything from there I get error 403: Forbidden.

[eu]

Actually, the access is forbidden only to the image. I downloaded and applied the changes.diff file, then built Haiku from scratch. The message I get now is much better:

Can't currently support more than 512GB of RAM! (arch_mmu) maxAddress 0x140000000

[eu]

So, maxAddress is 140000000. Is it normal to divide 140000000 / 0x40000000 and check if > 512? Is it ok to have Ox prefix in front of 40000000?

[cocobean]

eu: See: ​https://www.haiku-os.org/docs/userguide/en/bootloader.html You can try to modify the kernel settings file to limit memory, safe mode, graphics driver, and enable serial debugging if your USB keyboard is not letting you get into the boot loader GUI menu.

[eu]

Not having a programming expertise, I still think there may be a mess of simple arithmetic, divisions between numbers in decimal and hex formats. It may happen to work for 32GB of RAM but not for more.

Better give up on Haiku and wait a few years until developers get their hands on machines with more RAM. Then, if it won't work for them, they'll have to fix it!

[kallisti5]

Better give up on Haiku and wait a few years until developers get their hands on machines with more RAM. Then, if it won't work for them, they'll have to fix it!

I actually boot Haiku on my desktop (Ryzen 3200x) with 64GiB of ram all the time via EFI. I wonder what the difference is?

Is it normal to divide 140000000 / 0x40000000 and check if > 512? Is it ok to have Ox prefix in front of 40000000?

Yes. 0x means hexadecimal

[eu]

Maybe various UEFI implementations differ.

My AMD machine has a Ryzen Threadripper 3970 on an MSI Creator TRX40 motherboard. The Intel machine is a NUC 10th generation.

The fact that I'm using three other OSes successfully (with GRUB) on both devices makes me draw the logical conclusion that something is not working right in Haiku UEFI boot loader.

There may not be enough people using Haiku on machines with large quantities of RAM, so there is not enough testing and we have only my complaint.

[kallisti5]

It is weird. Most sizes are from ELF, kernel and drivers so should not be big, and addresses being close to 512GB doesn't make sense either. So if it is not the stick, it may be the download or how you copy it to the stick.

I see we have two other places where the same message is used. One in BIOS booting, and one in an arch x86_64 specific memory check, and I think the arch one looks strange:​​https://github.com/haiku/haiku/blob/d0e006f5569f24e73a4d83f5c9766f1132160d7a/src/system/boot/platform/efi/arch/x86_64/arch_mmu.cpp#L194

The math here seems fine.

• Larger of maxMemory or 0x100000000 == 4294967296 == 4GiB
• Round up to 0x40000000 (closest 1GiB) (ok. this seems a little weird, not sure why it needs to be 1GiB aligned)

	if (maxAddress / 0x40000000 > 512)
		panic("Can't currently support more than 512GB of RAM!");

• 0x40000000 = 1073741824
  • Bytes / 1073741824 == How many GiB of ram.

[kallisti5]

The fact that I'm using three other OSes successfully (with GRUB) on both devices makes me draw the logical conclusion that something is not working right in Haiku UEFI boot loader.

Not arguing against this lol, just trying to determine what could be going wrong. It definitely isn't purely based on amount of ram since I can boot Haiku as EFI with 64GiB.

We made some adjustments to memory mapping due to the recent arm + x86 (32-bit) work which could be at play, however this issue was reported 14 months ago which is *long* before the efi loader rework.

To confirm eu. You saw the same issue 14 months ago, and now with the latest image?

[eu]

I tried 3 or 4 different sticks, all with the same result, so it's not the sticks.

I download probably once a month the latest nightly image.

In the end I followed the "Building Haiku" guide to build haiku direct from git source, with two different results: anyboot iso image and direct installation on hard disk partition. The latest hrev I built is 55874.

Maybe somebody knowledgeable could produce a test image where all those variables involved are printed on screen during the boot process, so we can figure out where the error is. I mean before that dreaded message comes up!

[pulkomandy]

To me the problem seems quite simple: physical RAM address don't always start at 0.

So, yes, you have only 64GB of RAM, but your hardware decided to put it near the end of its address space, and it ends up past the 512GB address.

If the RAM is installed in slots on the motherboard, make sure to use the first slots and not the last ones. If that's not possible, well, we really need to allow RAM to be this far in the address space, and we just had not yet found a machine requiring that.

Supporting this is not very easy, as it requires more physocal address bits to be used (that's no problem), but also some changes to the way we program the MMU. I think korli already has done some preliminary work towards it?

[eu]

The AMD MB has 8 slots, all filled with 16GB each. The Intel NUC has 2 slots filled with 32GB each.

What do you mean by "address space"? Your reasoning seems to imply that it is something virtual of finite dimensions and some machines put the RAM at different locations in that space. It doesn't make a lot of sense because it would affect also FreeBSD, Linux and Windows.

[korli]

arch_mmu_generate_post_efi_page_tables() is only called when starting the kernel, so you should be able to drop in the bootloader menu before that happens (hitting shift or space keys)

[madmax]

Surely I'm missing something, but if maxAddress is 0x140000000, isn't that 5GB and there should be no panic?

Maybe somebody could produce a test image where all those variables involved are printed on screen during the boot process

The patch you applied does print the physical addresses and number of pages of memory descriptors. I've uploaded a new version that forces debug output to screen.

[kallisti5]

The AMD MB has 8 slots

Ah-ha. This is a server motherboard? You never mentioned that. What's the model / chipset?

The AMD motherboard is not a surprise here... however I think the Intel NUC is the surprise to me with only 64GIB of ram. eu, are you sure the all of the devices listed in the ticket description have the *exact* same issue?

[eu]

The AMD machine does not have server MB, it is desktop, see details at ​https://www.msi.com/Motherboard/creator-trx40/Specification

The NUC is this one: ​https://www.intel.com/content/www/us/en/products/sku/188811/intel-nuc-10-performance-kit-nuc10i7fnh/specifications.html

And yes, both devices end up with the same message.

There are some random issues though: sometimes, not always, anyboot images freeze at the icons screen, not showing the message about 512GB of RAM, regardless of the USB stick used. It's more prevalent on the NUC, very rare on the Ryzen.

When I build Haiku from git source with direct installation on hard disk partition (only on AMD, not on the NUC) the message is always shown.

[eu]

Again, I don't have programming expertise, but looking at those lines of code, for example "maxAddress / 0x40000000 > 512" makes me think: well numbers in hex format are divided and then compared (>) with 512 which is a number in decimal format. Shouldn't it be 0x200 instead?

I still think there is a possibility of flawed arithmetic between various format of numbers involved. Of course, I could easily be wrong.

[pulkomandy]

The way you write the numbers in the sourcecode makes no difference at all. The compiler converts everything to binary that the computer can work on. The different way to write numbers are here for readability for humans only.

[eu]

Oh, OK.

Just let me know what else can I do, like testing various options/alternatives.

[halamix2]

In my instance the panic message comes from haiku/src/system/boot/platform/efi/arch/x86_64/arch_mmu.cpp:226

• Haiku hrev56632
• CPU: AMD Ryzen 9 7950X
• Mobo: Asus Prime X670E-PRO WiFi
• RAM: 2x32GB DDR5

I've tried changing the panic message to panic("Can't currently support more than 521GB of RAM! %lu %lu %lu", maxAddress, memory_map_size, descriptor_size); and I got 1099511627776 2736 48, so 1TiB of reported RAM and 57 entries.

I was asked to assign korli to this issue, but I don't have permission to do so

[korli]

halamix2, can you attach a screenshot?

[korli]

I checked what would be mapped for such a processor, for instance, ​https://linux-hardware.org/?probe=64299c7b4a&log=dmesg

0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009ffff] usable
[    0.000000] BIOS-e820: [mem 0x00000000000a0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x0000000009e01fff] usable
[    0.000000] BIOS-e820: [mem 0x0000000009e02000-0x0000000009ffffff] reserved
[    0.000000] BIOS-e820: [mem 0x000000000a000000-0x000000000a1fffff] usable
[    0.000000] BIOS-e820: [mem 0x000000000a200000-0x000000000a20ffff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x000000000a210000-0x000000000affffff] usable
[    0.000000] BIOS-e820: [mem 0x000000000b000000-0x000000000b020fff] reserved
[    0.000000] BIOS-e820: [mem 0x000000000b021000-0x00000000a05bafff] usable
[    0.000000] BIOS-e820: [mem 0x00000000a05bb000-0x00000000a65bafff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000a65bb000-0x00000000a666afff] ACPI data
[    0.000000] BIOS-e820: [mem 0x00000000a666b000-0x00000000a866afff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x00000000a866b000-0x00000000a9ffdfff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000a9ffe000-0x00000000a9ffefff] usable
[    0.000000] BIOS-e820: [mem 0x00000000a9fff000-0x00000000a9ffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000aa000000-0x00000000abffcfff] usable
[    0.000000] BIOS-e820: [mem 0x00000000abffd000-0x00000000acffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000ad7f5000-0x00000000afffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000f0000000-0x00000000f7ffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fd000000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000747ffffff] usable
[    0.000000] BIOS-e820: [mem 0x0000000748000000-0x0000000847ffffff] reserved
[    0.000000] BIOS-e820: [mem 0x000000084eec0000-0x00000008701fffff] reserved
[    0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved

This means, the top reserved ranges should be ignored. At ​https://cgit.haiku-os.org/haiku/tree/src/system/boot/platform/efi/arch/x86_64/arch_mmu.cpp#n226

	for (size_t i = 0; i < memory_map_size / descriptor_size; ++i) {
		efi_memory_descriptor *entry
			= (efi_memory_descriptor *)((addr_t)memory_map + i * descriptor_size);
		switch (entry->Type) {
		case EfiLoaderCode:
		case EfiLoaderData:
		case EfiBootServicesCode:
		case EfiBootServicesData:
		case EfiConventionalMemory:
		case EfiRuntimeServicesCode:
		case EfiRuntimeServicesData:
		case EfiPersistentMemory:
		case EfiACPIReclaimMemory:
		case EfiACPIMemoryNVS:
			maxAddress = std::max(maxAddress,
				entry->PhysicalStart + entry->NumberOfPages * 4096);										 
			break;
		}
		default:
			break;
	}

I'll submit a patch, or you can try yourself.

[halamix2]

my results: ​https://linux-hardware.org/?probe=b7fa78df7a&log=dmesg

[    0.000000] BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009ffff] usable
[    0.000000] BIOS-e820: [mem 0x00000000000a0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x0000000009afefff] usable
[    0.000000] BIOS-e820: [mem 0x0000000009aff000-0x0000000009ffffff] reserved
[    0.000000] BIOS-e820: [mem 0x000000000a000000-0x000000000a1fffff] usable
[    0.000000] BIOS-e820: [mem 0x000000000a200000-0x000000000a20ffff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x000000000a210000-0x000000000affffff] usable
[    0.000000] BIOS-e820: [mem 0x000000000b000000-0x000000000b020fff] reserved
[    0.000000] BIOS-e820: [mem 0x000000000b021000-0x000000006fdf8fff] usable
[    0.000000] BIOS-e820: [mem 0x000000006fdf9000-0x0000000076644fff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000076645000-0x00000000767c3fff] ACPI data
[    0.000000] BIOS-e820: [mem 0x00000000767c4000-0x00000000787c3fff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x00000000787c4000-0x000000007a1fefff] reserved
[    0.000000] BIOS-e820: [mem 0x000000007a1ff000-0x000000007bffafff] usable
[    0.000000] BIOS-e820: [mem 0x000000007bffb000-0x000000007cffffff] reserved
[    0.000000] BIOS-e820: [mem 0x000000007d7f5000-0x000000007fffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000f0000000-0x00000000f7ffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fd000000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000001057ffffff] usable
[    0.000000] BIOS-e820: [mem 0x0000001058000000-0x0000001077ffffff] reserved
[    0.000000] BIOS-e820: [mem 0x000000107eec0000-0x00000010a01fffff] reserved
[    0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved

This patch didn't help

I also have a physical serial connection if there's a way to output something useful this early during boot

[korli]

how did you test this patch?

[halamix2]

I've replaced the loop at ​https://cgit.haiku-os.org/haiku/tree/src/system/boot/platform/efi/arch/x86_64/arch_mmu.cpp#n223 with your code, recompiled Haiku and flashed iso to a pendrive with Balena Etcher (which also validates the pendrive against the iso)

[halamix2]

I am so sorry, the patch does work, I forgot to save the cpp file the first time...

I was able to boot the system with serial console enabled, and got to the desktop.

Without the serial console the boot hangs up when the rocket icon is red, but that's probably another issue Edit: that another issue was the pendrive, it's partially broken it seems. Everything works as expected with another one

[waddlesplash]

Fix merged in hrev56643 +beta4.