Fix ARM bootstrap for hard + soft float binaries

[kallisti5]

Our EFI arm (not arm64) bootloader requires software floating point, while our kernel + OS will use hardware floating point.

Initial work has been completed, and some changes to our bootstrap gcc have taken place to generate bootloader-specific libgcc and friends with software floating point. (​https://github.com/haikuports/haikuports.cross/commit/b98463ad4a26f6f9a6661d60dda3cc1777ce8c92)

The soft-fp bootstrap fails however with the following error:

/home/kallisti5/Code/haiku/generated.arm/cross-tools-arm/lib/gcc/arm-unknown-haiku/8.3.0/../../../../arm-unknown-haiku/bin/ld: error: /home/kallisti5/Code/haiku/generated.arm/objects/haiku/arm/packaging/repositories/HaikuPortsCross-build/sys-devel/gcc_bootstrap/work-8.3.0_2019_05_24/boot/cross-sysroot/arm/boot/system/develop/lib/init_term_dyn.o uses VFP register arguments, ./libgcc_s.so.1.tmp does not
/home/kallisti5/Code/haiku/generated.arm/cross-tools-arm/lib/gcc/arm-unknown-haiku/8.3.0/../../../../arm-unknown-haiku/bin/ld: failed to merge target specific data of file /home/kallisti5/Code/haiku/generated.arm/objects/haiku/arm/packaging/repositories/HaikuPortsCross-build/sys-devel/gcc_bootstrap/work-8.3.0_2019_05_24/boot/cross-sysroot/arm/boot/system/develop/lib/init_term_dyn.o

The issue above needs resolved, then we need to unbootstrap the new arm gcc_bootstrap package, then ensure the EFI code on arm uses the new libgcc_boot.a

To reproduce:

diff --git a/build/jam/repositories/HaikuPortsCross/arm b/build/jam/repositories/HaikuPortsCross/arm
index 1cb76bb458..a9184aedca 100644
--- a/build/jam/repositories/HaikuPortsCross/arm
+++ b/build/jam/repositories/HaikuPortsCross/arm
@@ -6,9 +6,9 @@ BootstrapPackageRepository HaikuPortsCross
        noto-20170202-7
        :
        # repository architecture packages (stage 0)
-       gcc_bootstrap-8.3.0_2019_05_24-1
-       gcc_bootstrap_syslibs-8.3.0_2019_05_24-1
-       gcc_bootstrap_syslibs_devel-8.3.0_2019_05_24-1
+       gcc_bootstrap-8.3.0_2019_05_24-4
+       gcc_bootstrap_syslibs-8.3.0_2019_05_24-4
+       gcc_bootstrap_syslibs_devel-8.3.0_2019_05_24-4
        :
        # repository architecture packages (stage 1)
        bash_bootstrap-4.4.023-1
@@ -25,12 +25,11 @@ BootstrapPackageRepository HaikuPortsCross
        freetype_bootstrap_devel-2.6.3-1
        gawk_bootstrap-3.1.8-2
        grep_bootstrap-2.14-1
-       icu_bootstrap-57.1-2
-       icu_bootstrap_devel-57.1-2
+       icu_bootstrap-57.2-2
+       icu_bootstrap_devel-57.2-2
        less_bootstrap-451-1
        m4_bootstrap-1.4.16-1
        make_bootstrap-4.1-2
-       mawk_bootstrap-1.3.4-1
        ncurses6_bootstrap-6.0-1
        ncurses6_bootstrap_devel-6.0-1
        python_bootstrap-2.7.6-1
@@ -53,7 +52,6 @@ BootstrapPackageRepository HaikuPortsCross
        findutils_bootstrap
        flex_bootstrap
        freetype_bootstrap
-       mawk_bootstrap
        gawk_bootstrap
        gcc_bootstrap
        grep_bootstrap

To try:

$ ../configure --cross-tools-source ../../buildtools/ --build-cross-tools arm --bootstrap ../../haikuporter/haikuporter ../../haikuports.cross/ ../../haikuports -j16
$ jam -q @bootstrap-raw

[waddlesplash]

There needs to be a separate "boot" version of init_term_dyn.o, it appears.

[davidkaroly]

Replying to waddlesplash:

There needs to be a separate "boot" version of init_term_dyn.o, it appears.

we could also use -nostartfiles to prevent gcc from linking libgcc-boot with init_term_dyn.o or any other object files from the glue folder.

If I apply the following change in haikuports.cross, I get a successful bootstrap:

--- a/sys-devel/gcc_bootstrap/gcc_bootstrap-8.3.0_2019_05_24.recipe
+++ b/sys-devel/gcc_bootstrap/gcc_bootstrap-8.3.0_2019_05_24.recipe
@@ -208,7 +208,7 @@ BUILD()
        local bootCcFlags
        if [ $effectiveTargetArchitecture == arm ]; then
                # EFI arm (32-bit) requires software fp
-               bootCcFlags+="-mfloat-abi=soft";
+               bootCcFlags+="-mfloat-abi=soft -nostartfiles";
        fi
 
        # build bootloader version of libgcc and libgcc_eh.a (no threads, TLS)

However... I haven't tried linking the bootloader against libgcc-boot so I can't tell whether it results in a usable libgcc-boot.a.

[pulkomandy]

The problem we have is that the current gcc does not provide libraries for soft-float support. If you try to use them, it complains that some files are missing. If you then use nostartfiles to not use these files, it will build something, but the crti/crto code for start and termination will be missing.

Since eventually the bootloader executable is not really in ELF format, but somehow converted to a Pe executable (same format used by Windows, and also used by EFI), you may be able to get this running. I'm not sure how much of the startup code is actually used in that case. But I would test carefully before adding a workaround on top of another. Otherwise it just makes things harder to untangle.

[davidkaroly]

I took one more look at it and it still looks like -nostartfiles could be a valid option.

• efi bootloader is build with -nostdlib, see KernelArchitectureSetup in build/jam/ArchitectureRules (the variable HAIKU_BOOT_$(bootTarget:U)_LDFLAGS is set in function KernelArchitectureSetup)
• what does nostdlib do according to gcc docs: "Do not use the standard system startup files or libraries when linking." i.e. -nostdlib is stronger than -nostartfiles

I also did an experiment, it's possible to successfully compile and link the bootloader with libgcc-boot.a using -nostartfiles. (even though it crashes shortly after starting, I think that's not related to the glue startup files) Some more details on the steps that I tried:

• I did bootsstrap with soft-float and nostartfiles for libgcc-boot and libsupc++-boot
• changed SystemLibraryRules to use libgcc-boot and libsupc++-boot for the bootloader (originally it uses libgcc-kernel, libsupc++-kernel)
• added -mfloat-abi=soft to HAIKU_BOOT_..._CCFLAGS/C++FLAGS in ArchitectureRules
• commented out FDT code and FPU initialization in the bootloader

This way I can build a bootloader with soft-float: compile and linking is successful. The bootloader starts, it can print out a one-liner debug message but then unfortunately it crashes, probably somewhere in vsnprintf.

[pulkomandy]

I think the best thing to do is let you continue fixing problems with the bootloader, if this approach continues working, we can merge the patch later, and otherwise we can try to find another solution?

I think tqh also ran into problems in vsnprintf while trying to format floating point values? But he would know more about that, and maybe your issue is different, or maybe it's the same issue but the cause is actually something else.

[davidkaroly]

Replying to pulkomandy:

I think the best thing to do is let you continue fixing problems with the bootloader, if this approach continues working, we can merge the patch later, and otherwise we can try to find another solution?

ok that sounds right

[davidkaroly]

this might be related

/home/david/haiku-projects/haiku/generated.arm/cross-tools-arm/bin/arm-unknown-haiku-ld: warning: build_packages/gcc_syslibs_devel-8.3.0_2019_05_24-1-arm/develop/lib/libgcc-boot.a(_udivmoddi4.o) uses 4-byte wchar_t yet the output is to use 2-byte wchar_t; use of wchar_t values across objects may fail

maybe something to do with -fshort-wchar

[tqh]

The values for printf could not be trusted for numbers.

printf("%d\n", a);

prints a 0

if (a==5) printf("ok\n");

prints ok

A lot of fun was had tracking that down.

[tqh]

Replying to davidkaroly:

this might be related

/home/david/haiku-projects/haiku/generated.arm/cross-tools-arm/bin/arm-unknown-haiku-ld: warning: build_packages/gcc_syslibs_devel-8.3.0_2019_05_24-1-arm/develop/lib/libgcc-boot.a(_udivmoddi4.o) uses 4-byte wchar_t yet the output is to use 2-byte wchar_t; use of wchar_t values across objects may fail

maybe something to do with -fshort-wchar

UEFI uses wide chars, so in bootloader we need to use it. I have not checked if UEFI for Arm is any different. The message implies we use objects with different wchar as well. So some objects or libs seems to be compiled wrong.

[tqh]

16 bit chars, same as Windows is what we want.

[davidkaroly]

ok, it's not the divmod. it was the memcpy from arch_string.S which uses FPU instructions.

after changing the build scripts to pick up the generic memcpy implementation from libroot, we get a working bootloader.

Recap - what needs to be changed to get a working bootloader with softfloat:

• adjust gcc_bootstrap recipe to use bootCcFlags+="-mfloat-abi=soft -nostartfiles -fshort-wchar";
• use generic memcpy in the bootloader instead of arch_string.S
• disable FPU initialization in arch_cpu.cpp, move it to entry.S instead
• add -mfloat-abi=soft to compile flags for bootloader in ArchitectureRules
• link bootloader with libgcc-boot and libsupc++-boot
• and finally, adjust the build scripts for boot_fdt.a to "grist" it correctly using MultiBootGristFiles (othewise the build system forgets the soft-float flag when compiling boot_fdt.a, giving us the linking issue again)

Once all these are in place, the bootloader can load and start the kernel. We get to the 4th icon on the splash screen, like on master.

[davidkaroly]

After adding -nostartfiles and -fshort-wchar I'm able to build a bootloader with soft-float that can start the kernel. Therefore I opened a pull request for haikuports.cross: ​https://github.com/haikuports/haikuports.cross/pull/8

Please let me know your thoughts.

[pulkomandy]

Change merged. If the kernel starts, that's all we need from the bootloader :)

[davidkaroly]

Thanks for the quick response.

I'll clean up my patches for compiling the bootloader in soft-float and submit them to gerrit review.

Meanwhile one more question: what shall we do with the u-boot based ARM bootloader?

• rebuild it with soft-float as well?
• or leave it as it is but then it will need different libraries?

[pulkomandy]

Do we need to support non-EFI boot on ARM at all? There are no working machines using it at the moment, if anyone needs it they can see how to fix it.

[davidkaroly]

I've submitted some reviews for moving the ARM bootloader to soft-float.

This will probably break the u-boot based ARM bootloader. Unfortunately I don't even know how to try it.

[davidkaroly]

I've added one more pull request to haikuports.cross to actually pack up the resulting libgcc-boot file in the bootstrap package. (previously I just copied it in the build directory manually)

With that I hope that we can do bootstrap, unbootstrap and then non-bootstrap build with the new packages.

--

re u-boot: I'm absolutely clueless, unfortunately I don't even know how to try to build Haiku with u-boot based bootloader. Is there a HOWTO for that perhaps? I'd like to try at least to see if it still builds...

[tqh]

The change looks good to me so I went ahead and merged it: ​https://github.com/haikuports/haikuports.cross/pull/9

We can improve if someone has more feedback.

[davidkaroly]

I haven't posted in a while... bootstrap for ARM now almost works, except for one (hopefully last) issue that came up today.

since the introduction of commit 7db2616, the bootstrap build for make package fails with the following error message:

compiling src/dir.c...
./src/dir.c: In function 'read_dirstream':
./src/dir.c:1229:44: error: invalid application of 'sizeof' to incomplete type 'char[]'
           size_t sz = sizeof (*d) - sizeof (d->d_name) + len;
                                            ^

If I revert the change in dirent.h then the bootstrap completes successfully.

Have you seen anything similar for arm64 or riscv64 bootstrap build?

[davidkaroly]

See also upstream gnu make sources: ​https://github.com/mirror/make/blob/master/src/dir.c#L1262

It's trying to do sizeof() on d_name which is now char[]. Apparently gcc-8.3.0 is not happy about this. I haven't tried it on gcc-11.

[pulkomandy]

It can simply be removed to use size_t sz = sizeof (*d) + len; instead.

A + 1 might be missing depending on how "len" is computed however? Make sure there is space for the NULL terminator at the end of the name.

[davidkaroly]

The same issue is there also for regular haikuports build on x86_64 with gcc-11. As it's not only arm related, I'll raise a ticket in haikuports repo.

Seems that patching would be needed for make in haikuports as well as make_bootstrap in haikuports.cross.

[davidkaroly]

of course: the ARM bootstrap "almost works" :)

some updates will be needed now that there's gcc11 also in haikuports.cross

[kallisti5]

yup. I made sure to include the bootloader libraries (with hard float) in the gcc11 bootstrap package.

Untested though... so if you run through a bootstrap, make sure they're correct on arm :-)

[davidkaroly]

I did one more bootstrap+unbootstrap on current tree (with ​https://review.haiku-os.org/c/haiku/+/4782 and ​https://github.com/haikuports/haikuports.cross/pull/14 applied on top of current master)

New packages are available on the "limerick" sftp server.

@kallisti5 can you perhaps do that special magic to make them available for a non-bootstrap build?

[kallisti5]

Done in ​https://git.haiku-os.org/haiku/log/?qt=range&q=9137be71a36a+%5Efbc211a25582

Let me know how it goes :-)

[kallisti5]

oh... you're likely missing zstd.

It needs added to the HaikuPortsCross/arm file, re-run the bootstrap (you should be able to just jam -q @bootstrap-raw again without rebuilding everything). Then re-unbootstrap.

You can upload just the zstd* package to limerick sftp.

[kallisti5]

nevermind. you did send it. I just didn't add it to the HaikuPorts/arm file. Done!

[davidkaroly]

One thing might be still missing: the build system is complaining because of zstd source packages.

Perhaps we need zstd added to the source package section in HaikuPorts/arm as well?

[kallisti5]

ack. Yup. That's on me. I didn't add zstd to the source section. Normally I would test these builds, but upstream isn't compiling.

@davidkaroly are there patches in gerrit that are waiting review to fix the arm build?

If so, i'll prioritize getting those reviewed / merged.

[davidkaroly]

it's only this one: ​https://review.haiku-os.org/c/haiku/+/4782

once that's merged, it should be possible to do arm builds again.

[kallisti5]

nice work davidkaroly!!!

I can confirm arm should build again with your changes as of hrev55765

[kallisti5]

Can confirm EFI haiku_loader is working on arm again:

qemu-system-arm -M virt -bios ~/Code/firmware/u-boot/arm/qemu/u-boot.bin -m 1G -drive file=haiku-mmc.image,format=raw,if=virtio -usb -device usb-ehci,id=echi -device usb-kbd -device usb-tablet

arch_smp_register_cpu()
cpu
  id: 0
GOP protocol not found
Welcome to the Haiku boot loader!
add_partitions_for(0x7bd93138, mountFS = no)
add_partitions_for(fd = 0, mountFS = no)
0x7bd93178 Partition::Partition
0x7bd93178 Partition::Scan()
check for partitioning_system: GUID Partition Map
.
.
.
PackageVolumeInfo::SetTo()
PackageVolumeInfo::_InitState(): failed to parse activated-packages: No such file or directory
load kernel kernel_arm...
maximum boot loader heap usage: 345616, currently used: 337256
Chosen UART:
  kind: pl011
  regs: 0x9000000, 0x1000
  irq: 1
  clock: 24000000
Chosen interrupt controller:
  kind: gicv2
  regs: 0x8000000, 0x10000
        0x8010000, 0x10000
kernel:
  text: 0x80000000, 0x193000
  data: 0x801a2000, 0x4a000
  entry: 0x8006ee0c
Kernel stack at 0x8241a000
System provided memory map:
  phys: 0x40000000-0x47f00000, virt: 0x40000000-0x47f00000, type: ConventionalMemory (0x7), attr: 0x8
  phys: 0x47f00000-0x48003000, virt: 0x47f00000-0x48003000, type: ACPIReclaimMemory (0x9), attr: 0x8
  phys: 0x48003000-0x7b789000, virt: 0x48003000-0x7b789000, type: ConventionalMemory (0x7), attr: 0x8
  phys: 0x7b789000-0x7dd93000, virt: 0x7b789000-0x7dd93000, type: LoaderData (0x2), attr: 0x8
  phys: 0x7dd93000-0x7ddf2000, virt: 0x7dd93000-0x7ddf2000, type: LoaderCode (0x1), attr: 0x8
  phys: 0x7ddf2000-0x7ddf6000, virt: 0x7ddf2000-0x7ddf6000, type: ReservedMemoryType (0x0), attr: 0x8
  phys: 0x7ddf6000-0x7ddf7000, virt: 0x7ddf6000-0x7ddf7000, type: BootServicesData (0x4), attr: 0x8
  phys: 0x7ddf7000-0x7ddf8000, virt: 0x7ddf7000-0x7ddf8000, type: RuntimeServicesData (0x6), attr: 0x8000000000000008
  phys: 0x7ddf8000-0x7ddfa000, virt: 0x7ddf8000-0x7ddfa000, type: BootServicesData (0x4), attr: 0x8
  phys: 0x7ddfa000-0x7ddfb000, virt: 0x7ddfa000-0x7ddfb000, type: ReservedMemoryType (0x0), attr: 0x8
  phys: 0x7ddfb000-0x7ddfe000, virt: 0x7ddfb000-0x7ddfe000, type: RuntimeServicesData (0x6), attr: 0x8000000000000008
  phys: 0x7ddfe000-0x7ddff000, virt: 0x7ddfe000-0x7ddff000, type: BootServicesData (0x4), attr: 0x8
  phys: 0x7ddff000-0x7de03000, virt: 0x7ddff000-0x7de03000, type: RuntimeServicesData (0x6), attr: 0x8000000000000008
  phys: 0x7de03000-0x7de04000, virt: 0x7de03000-0x7de04000, type: ReservedMemoryType (0x0), attr: 0x8
  phys: 0x7de04000-0x7de05000, virt: 0x7de04000-0x7de05000, type: BootServicesData (0x4), attr: 0x8
  phys: 0x7de05000-0x7de06000, virt: 0x7de05000-0x7de06000, type: ReservedMemoryType (0x0), attr: 0x8
  phys: 0x7de06000-0x7de08000, virt: 0x7de06000-0x7de08000, type: BootServicesData (0x4), attr: 0x8
  phys: 0x7de08000-0x7de09000, virt: 0x7de08000-0x7de09000, type: ReservedMemoryType (0x0), attr: 0x8
  phys: 0x7de09000-0x7de0a000, virt: 0x7de09000-0x7de0a000, type: BootServicesData (0x4), attr: 0x8
  phys: 0x7de0a000-0x7de0e000, virt: 0x7de0a000-0x7de0e000, type: ReservedMemoryType (0x0), attr: 0x8
  phys: 0x7de0e000-0x7de0f000, virt: 0x7de0e000-0x7de0f000, type: BootServicesData (0x4), attr: 0x8
  phys: 0x7de0f000-0x7de10000, virt: 0x7de0f000-0x7de10000, type: ReservedMemoryType (0x0), attr: 0x8
  phys: 0x7de10000-0x7ff51000, virt: 0x7de10000-0x7ff51000, type: LoaderData (0x2), attr: 0x8
  phys: 0x7ff51000-0x7ff53000, virt: 0x7ff51000-0x7ff53000, type: RuntimeServicesCode (0x5), attr: 0x8000000000000008
  phys: 0x7ff53000-0x80000000, virt: 0x7ff53000-0x80000000, type: LoaderData (0x2), attr: 0x8
sPageDirectory  = 0x7b754000
sFirstPageTable = 0x7b758000
sLastPageTable  = 0x7b788000
gKernelArgs.arch_args.phys_pgdir     = 0x7b754000
gKernelArgs.arch_args.vir_pgdir      = 0x82421000
gKernelArgs.arch_args.next_pagetable = 0x00018000
Calling ExitBootServices. So long, EFI!
phys memory ranges:
    0x40000000-0x47f00000, length 0x07f00000
    0x48003000-0x7b789000, length 0x33786000
    0x7ddf6000-0x7ddf7000, length 0x00001000
    0x7ddf8000-0x7ddfa000, length 0x00002000
    0x7ddfe000-0x7ddff000, length 0x00001000
    0x7de04000-0x7de05000, length 0x00001000
    0x7de06000-0x7de08000, length 0x00002000
    0x7de09000-0x7de0a000, length 0x00001000
    0x7de0e000-0x7de0f000, length 0x00001000
allocated phys memory ranges:
    0x7b751000-0x7dd93000, length 0x02642000
    0x7de10000-0x7ff51000, length 0x02141000
    0x7ff53000-0x80000000, length 0x000ad000
allocated virt memory ranges:
    0x7b758000-0x7b788000, length 0x00030000
    0x7b789000-0x7ddf2000, length 0x02669000
    0x7ddf7000-0x7ddf8000, length 0x00001000
    0x7ddfb000-0x7ddfe000, length 0x00003000
    0x7ddff000-0x7de03000, length 0x00004000
    0x7de10000-0x801ec000, length 0x023dc000
    0x82000000-0x82457000, length 0x00457000
virt memory ranges to keep:
    0x82456000-0x82457000, length 0x00001000
arch_enter_kernel(ttbr0: 0x7b754000, kernelArgs: 0x7ddeda64, kernelEntry: 0x8006ee0c, sp: 0x8241e000)
(hang.. or uart not setup?)

[davidkaroly]

I'm using TianoCore EDK2 and qemu-6.1.0, for me it reaches to the 4th icon on the boot splash screen using the command line below:

qemu-system-arm -bios /usr/share/edk2/arm/QEMU_EFI-pflash.raw -M virt -m 1G \
-device virtio-blk-device,drive=x0,bus=virtio-mmio-bus.0 \
-drive file="haiku-mmc.image",if=none,format=raw,id=x0 \
-device ramfb -usb -device qemu-xhci,id=xhci -device usb-mouse -device usb-kbd \
-serial stdio

Some printouts from the kernel starting up:

arch_enter_kernel(ttbr0: 0x7993c000, kernelArgs: 0x7c127a64, kernelEntry: 0x8006ee0c, sp: 0x8254e000)
Welcome to kernel debugger output!
Haiku revision: hrev55765, debug level: 2

Eventually I get a "PANIC: did not find any boot partitions!" which is expected on the current master as there are still a few minor things missing for the virtio-mmio block device to work on ARM.

If I try it with u-boot.bin, it hangs for me similarly, right after arch_enter_kernel. Quick check with qemu "info registers" shows abt32, the registers still contain the args passed on to the arch_enter_kernel trampoline function.

[kallisti5]

oh! Hot damn! nice work :-)

We'll need to figure out why u-boot 's efi can't enter the kernel. Pretty much all of the physical boards use u-boot for EFI (minus the Raspberry Pi 4 which has an early EDK2 port... lucky bastards)

[davidkaroly]

this might be the culprit?

​https://github.com/u-boot/u-boot/blob/master/configs/qemu_arm_defconfig#L9

​https://github.com/u-boot/u-boot/commit/d990f5c834f1b42293fb53e4fd7f3aa988184196

CONFIG_ARMV7_LPAE=y

If I understand it correctly, this also implies that u-boot is trying to initialize MMU in hypervisor mode (something to do with long descriptors and HTCR vs TTBCR)

[davidkaroly]

Yep, it was the the long descriptors. When starting Haiku with u-boot, TTBCR contains 0x8000...blabla, i.e. LPAE (aka EAE) is enabled. If I initialize TTBCR to zero, I start getting log messages from the kernel also on u-boot (and still 4th icon on the boot splash with TianoCore)

see: ​https://review.haiku-os.org/c/haiku/+/4852

[davidkaroly]

Other thing I noticed: u-boot does not initialize the frame buffer as it does not support qemu ramfb.

I recompiled u-boot from latest tag v2022.01 but that doesn't help either. It turns out that there are a set of patches to add qemu ramfb support in u-boot and most of them are already merged except for the last one.

So we need this one in addition: ​https://git.src.kameliya.ee/~kameliya/u-boot/commit/26f2e84b2b618cb6b12988361c2a9d858a1f827b

If I cherry-pick this changeset on top of u-boot v2022.01, I get the Haiku boot splash also in u-boot. (similarly to how it already works in TianoCore)

[davidkaroly]

Bootstrap build and then regular (non-bootstrap) build was successful and we get to the 4th icon on TianoCore and U-Boot, so I believe the original issue with bootstrap and soft-float libs is solved.

So perhaps we can close this ticket?

[tqh]

Yes, it seems bootstrap packages are built with softfloat, uboot needed a config-flag and lacks a efi framebuffer is the summary of the discussion. Nothing left to fix in this ticket.