Hybrid GPT/MBR Initial Program Loader

[meianoite]

Attached is a 432 bytes MBR IPL (Initial Program Loader) that looks for the UUID of a Haiku Boot GUID partition and boots it, falling back to MBR partitions if said UUID isn't found or if the (primary) GPT is corrupted or non-existent.

Its usefulness is in being a versatile, "universal" IPL that supports both partitioning styles, allowing it to be used transparently and even facilitating the conversion between partitioning styles, should the need for more partitions or volumes over 2TiB arise (for instance when cloning an older disk to a newer, more capacious one).

It also paves the way for Haiku to create and support booting from multiple volumes larger than 2TiB, which we're in the very privileged position of enjoying efficiently in the near future due to BFS. Another use case is taking a disk from a Intel EFI machine, plugging it on a BIOS machine and boot just fine; and vice-versa.

As mentioned, if there are valid partitions defined in the MBR, and the primary GPT becomes corrupt, it can fall back to loading the MBR partition with the active flag set, if one is defined.

Currently there's no provision for falling back to the GPT copy that lives in the end of the disk, due to the 512 bytes constraint; supporting this is unlikely given that the code is packed tight. An alternative would be disabling support for booting from MBR using BIOS calls other than Int13h function 42h, "Extended Read From Disk" (i.e., LBA mode). It's unlikely that any machine that Haiku supports won't have this BIOS function, but having an "universal" IPL should be quite useful to, say, people using Haiku to rewrite a corrupt MBR on another disk using the excellent DiskProbe.

The number of sectors loaded depends on the boot style. Booting from a MBR partition assumes that the Partition Boot Record is one sector long, whereas booting from a GPT partition assumes a partition exclusive for a system loader and will either copy its entirety into memory starting from address 0x7c00, or will copy up to circa 545KiB, whichever is smaller. Thus, it remains compatible with the FreeBSD gptloader and should work for loading Bootman from an exclusive Haiku boot manager partition as well.

It should be easy to adjust the UUID signature as needed. It lives at offset 0x1a0 (416), leaving plenty of space before the 32-bit disk signature at offset 0x1b8 (440), so compatibility with Microsoft Windows and other OSs is maintained.

[meianoite]

Updated with fixed comments and nicer formatting (set your editor to 4 spaces per tab)

[meianoite]

Changelog:

LBA pointer buffer is moved from 8 bytes on the MBR itself to a 64-bit buffer right after the stack. Makes more room for code and is more robust;

Deals with interrupts before and after relocation;

Work memory (0x600 to 0x7bff) is zero cleaned before relocation;

Deals with probable segment tainting by the BIOS. Nothing is assumed to be zeroed by default anymore;

Jumps to relocated code and loaded boot code made absolute (far);

Boot drive range check is disabled by default (turned into a compile-time option). Now assumes the BIOS is reasonable in this regard;

Test for BIOS extended disk read before attempting GPT read;

Reading arbitrary quantities of sectors bloated the code for no real gain. Now the amount of sectors to load depend on the partitioning scheme;

Restored some code from the FreeBSD loaders used as the basis for this one. Made the above item possible (slightly modified so GPT boot loader reading goes up to circa 545KiB now and stops there instead of erroring out);

In the process, noticed that error check after LBA disk read was removed. Duh! Restored it;

Changed some sections to make code more compact despite doing more work;

Comments are now more extensive. Magic numbers explained. Offsets explained. Exploited side effects are explicitly mentioned.

[meianoite]

Makefile

[meianoite]

Fixed a handful of comments, a typo in code, moved try_diskless to a better location and gave due credit to Starsseed

[meianoite]

Latest build

[axeld]

Now that you put all that work into it, how about a real patch, like the one from starsseed? One that I can apply, and it will update writembr, as well as the Intel partitioning, and share a single header for both applications as bonus? :-)

[meianoite]

Replying to axeld:

Now that you put all that work into it, how about a real patch, like the one from starsseed? One that I can apply, and it will update writembr, as well as the Intel partitioning, and share a single header for both applications as bonus? :-)

Here it is. One thing, however: writembr hasn't been written yet (see #5595). Moreover, I have no idea where the shared header should reside. I suppose the header would contain just the kBootCode array, so it's probably just a matter of excising it from PartitionMapWriter.cpp.

[meianoite]

in patch form

[bonefish]

Replying to meianoite:

Here it is. One thing, however: writembr hasn't been written yet (see #5595). Moreover, I have no idea where the shared header should reside. I suppose the header would contain just the kBootCode array, so it's probably just a matter of excising it from PartitionMapWriter.cpp.

If the assembly code can be munged by one of the assemblers the build system already uses (gcc/gas and, for x86, yasm), please do not add the compiled code in any form. There's the jam rule DataFileToSourceFile to create a source file defining an array from a binary file. It is already used in a similar case in src/apps/bootmanager/Jamfile.

[meianoite]

Replying to bonefish:

If the assembly code can be munged by one of the assemblers the build system already uses (gcc/gas and, for x86, yasm), please do not add the compiled code in any form. There's the jam rule DataFileToSourceFile to create a source file defining an array from a binary file. It is already used in a similar case in src/apps/bootmanager/Jamfile.

That's interesting. I didn't know such tool existed. I just changed what already exists in compiled form to account for the new binary.

In any case, I'm still unsure where the appropriate header should reside, were it to be shared. If I understand correctly, and were the header not to be shared, I'd have to grow src/add-ons/kernel/partitioning_systems/intel/Jamfile with something similar to:

DataFileToSourceFile [ FGristFiles MasterBootRecord.h ] : [ FGristFiles base_mbr.bin ]
	: kBootCode : 436 ;

and have the new header be included by PartitionMapWriter.cpp. That's simple enough, but the shared header case still eludes me. What would be the correct location then?

[bonefish]

Replying to meianoite:

In any case, I'm still unsure where the appropriate header should reside, were it to be shared. If I understand correctly, and were the header not to be shared, I'd have to grow src/add-ons/kernel/partitioning_systems/intel/Jamfile with something similar to:

DataFileToSourceFile [ FGristFiles MasterBootRecord.h ] : [ FGristFiles base_mbr.bin ]
	: kBootCode : 436 ;

The last parameter is a variable name too, declared as long long, containing the size of the array. It isn't needed when you create a header, since sizeof(kBootCode) works just fine in this case, but if one creates a source file and references it from other source files, one might find it handy.

and have the new header be included by PartitionMapWriter.cpp. That's simple enough, but the shared header case still eludes me. What would be the correct location then?

Since it's a generated header, it doesn't really matter. Just generate it in one of the Jamfiles where it is needed and refer to it from the other one. For that you need to 1. add the respective include path:

SubDirHdrs [ FDirName $(HAIKU_COMMON_PLATFORM_OBJECT_DIR) add-ons kernel partitioning_systems intel ] ;

and 2. explicitly state the inclusion relationship:

Includes [ FGristFiles myFile.cpp ]
    : <src!add-ons!kernel!partitioning_systems!intel>MasterBootRecord.h ;

Untested, so make sure you check the paths and grists.

[meianoite]

Replying to bonefish:

The last parameter is a variable name too, declared as long long, containing the size of the array. It isn't needed when you create a header, since sizeof(kBootCode) works just fine in this case, but if one creates a source file and references it from other source files, one might find it handy.

The compiled file generates a 512-byte long binary, of which we must take only 432 bytes. Is there a facility to clip the file size? I can't really modify the source to stop at 432 bytes as Anyboot needs the whole file as a MBR template.

Untested, so make sure you check the paths and grists.

Will do. But that will take a little while. I'm developing remotely over SSH on a quite limited machine (a Sheevaplug-like device with only 128MB of RAM) and only checked out the source, not compiled it. Building cross-tools for x86 took a whole three days. I'll have to find another machine I can code on.

[bonefish]

Replying to meianoite:

The compiled file generates a 512-byte long binary, of which we must take only 432 bytes. Is there a facility to clip the file size?

In place: truncate, copy: dd

I can't really modify the source to stop at 432 bytes as Anyboot needs the whole file as a MBR template.

It's not like the anyboot tool is closed source and couldn't be adjusted.

[meianoite]

Replying to bonefish:

In place: truncate, copy: dd

If you don't mind, I'll extend data_to_source.cpp and the Jam rule to take an optional size parameter.

[bonefish]

Replying to meianoite:

If you don't mind, I'll extend data_to_source.cpp and the Jam rule to take an optional size parameter.

Why not adjust anyboot? There's no real reason for the tool to expect anything more than the boot code and it can easily be fixed.

[meianoite]

Replying to bonefish:

Why not adjust anyboot? There's no real reason for the tool to expect anything more than the boot code and it can easily be fixed.

Because it is not broken. To assume a complete, valid MBR with no partitions set is not unreasonable IMHO. It even helps keeping the code simple since the input is always predictable.

OTOH, the current boot loaders are indeed broken and in need of fixing.

And in the same line of reasoning as above, data_to_source also doesn't need fixing, indeed, so I'll just write a Jam rule to use dd to extract the desired range instead. _That_ will probably be useful for many other situations.

[axeld]

Replying to meianoite:

Because it is not broken. To assume a complete, valid MBR with no partitions set is not unreasonable IMHO.

I'm not sure I can follow your line of reasoning: even if something is not broken, it's no argument to keep it as it is if it's not convenient to use, and requires extra work on another place.

But that aside, and correct me if I'm wrong, but the anyboot mechanism already needs to have a detailed knowledge about the MBR, as it needs to write the partition table itself -- and thus, requiring a complete 512 byte boot loader doesn't make much sense to me.

[meianoite]

Replying to axeld:

I'm not sure I can follow your line of reasoning: even if something is not broken, it's no argument to keep it as it is if it's not convenient to use, and requires extra work on another place.

This is not the point I'm trying to make. Virtually every source code for a MBR out there will produce a whole 512-byte binary, corresponding to a whole sector. The single place I can point that this is not convenient is embedding a range of those 512 bytes on another binary, which is a recent request; the patch I provided updated an _existing_ array of embedded code, but a Jam rule has since been created to deal with such situations more cleanly and I was asked to use that infrastructure instead. Fair enough. But that brought a new set of questions, because the existing infrastructure couldn't deal with an aspect of the problem at hand, that is, to copy only a range of bytes.

So at first I offered to expand that facility to account for that. Instead I'm being asked to change every other tool to conform, while the usual solution would be to confine those changes to a single point, preferably to a single tool that can be used whenever such style of embedding is necessary.

But that aside, and correct me if I'm wrong, but the anyboot mechanism already needs to have a detailed knowledge about the MBR, as it needs to write the partition table itself -- and thus, requiring a complete 512 byte boot loader doesn't make much sense to me.

That's beside the point. I asked whether there already existed a facility, as part of the existing Jam rules, that could extract a range of bytes from a binary for embedding. I got a tongue-in-cheek-y reply to use dd or truncate (or maybe I didn't make myself clear and that was a honest, straightforward answer). But on a second thought this is exactly what is needed. Since extracting a range of bytes from an arbitrary binary to a target file is a more general problem that tends to repeat itself (like embedding a bitmap image), I'm proposing that a more general/useful tool is incorporated into the build system and in return I'm being asked to change a bunch of semi-related code in order to satisfy a single instance where this problem is manifest, instead of keeping my changes to a minimum.

I'm not trying to avoid a problem, I'm instead willing to providing a more general solution.

[axeld]

Oh well, I forgot you like to write so wordy :-)

Replying to meianoite:

This is not the point I'm trying to make. [...]

That doesn't really matter; I only questioned your argumentation, not the contents.

So at first I offered to expand that facility to account for that. Instead I'm being asked to change every other tool to conform, [...]

Don't exaggerate. The only thing ever in question was anyboot.

But that aside, [...]

That's beside the point. [...]

No, it's an actual argument not to expect a 512 byte boot loader.

I got a tongue-in-cheek-y reply to use dd or truncate (or maybe I didn't make myself clear and that was a honest, straightforward answer). [...]

Of course it was a honest, and straightforward answer. There is certainly no need for being mean yet :-)

I'm not trying to avoid a problem, I'm instead willing to providing a more general solution.

FWIW, you managed to convince me. Not so much in the anyboot case, but in the more general solution thing. Unfortunately, you still haven't provided a patch ;-)

[meianoite]

Replying to axeld:

Oh well, I forgot you like to write so wordy :-)

=P

Don't exaggerate. The only thing ever in question was anyboot.

You believe it wouldn't hurt if it took just the IPL and reconstructed the rest of what makes an MBR, I believe it's valid for it to expect a full MBR minus partition table and just patch what it needs to. Let's just say we have a difference regarding the scope of anyboot, and leave it at that.

FWIW, you managed to convince me. Not so much in the anyboot case, but in the more general solution thing. Unfortunately, you still haven't provided a patch ;-)

As mentioned on comment:15, I'll have to find another machine I can code on. Actually, it just occurred to me that I could boot via a flash drive and use an external drive to hold the source (as I'll have to test it first, of course). That should work. I should have something ready by tomorrow.

[bonefish]

Replying to axeld:

Replying to meianoite:

I'm not trying to avoid a problem, I'm instead willing to providing a more general solution.

FWIW, you managed to convince me. Not so much in the anyboot case, but in the more general solution thing.

I'm still not convinced. There really is no reason for anyboot to require a complete MBR (*). It only needs the boot code and AFAI can see save for signature it writes all the other stuff itself, anyway. Writing the signature is like five additional lines of code. This makes a more general solution superfluous. And there's no reason to implement something that is not needed.

(*) The boot code could as well be compiled in. Earlier you sounded like you intended to do that anyway.

[modeenf]

so anything that we can apply? For a novice in this area what will this solve?

[phoudoin]

I'm with modeenf here (both novice and interested ;-): is this could unlock us poor GPT partitions guys?

AFAIK, our current boot loader don't support yet GPT, which needs

• to map/sync the MBR table with the GPT one,
• Haiku bootable partition(s) to live in the first 4 partitions

What this universal MBR will offer? No more MBR / GPT tables sync and no first 4 partitions limit?

[phoudoin]

Replying to bonefish:

Replying to axeld:

Replying to meianoite:

I'm not trying to avoid a problem, I'm instead willing to providing a more general solution.

FWIW, you managed to convince me. Not so much in the anyboot case, but in the more general solution thing.

I'm still not convinced. There really is no reason for anyboot to require a complete MBR (*). It only needs the boot code and AFAI can see save for signature it writes all the other stuff itself, anyway. Writing the signature is like five additional lines of code. This makes a more general solution superfluous. And there's no reason to implement something that is not needed.

AFAICT, anyboot currently don't mind to check MBR file's actual size, as it simply copy all its content at offset 0. Then, it write its built-in anyboot partition table over it..

What's matter is:

• anyboot assumes the MBR file is a complete 512 bytes-long, as it doesn't write itself the 0x55AA Boot Signature at end.
• anyboot don't overwrite the (optional) disk signature part - no big deal here.

Semantically, a MBR is a disk record of 512 bytes long. Period. It's the job of tools handling MBR to know how to skip, copy or overwrite bootstrap code and/or partitions table within.

I see no point to truncate hybrid_mbr.bin. So, either anyboot is changed to have the boot code compiled-in or it remains MBR agnostic like today, and then patching the partitions table of an external 512 bytes long data file make perfect sense.

I'm for keeping anyboot runtime MBR agnostic.

[phoudoin]

Regarding writembr system tool, what about moving the MBR in a ressource instead, via RDEF? That will allow to share the same MBR, while not relying on recompilation or data to header trick?

And writembr, contrary to anyboot, run from a BFS volume, right?

[phoudoin]

Same could be done for Boot Manager, BTW.

[meianoite]

Hoy. Almost two years. Brings some memories of the time when Real Life was quite tame, in hindsight.

Replying to phoudoin:

I'm with modeenf here (both novice and interested ;-): is this could unlock us poor GPT partitions guys?

This is for legacy BIOS. In theory what this IPL enables is that one could take a GPT-partitioned disk from a EFI system and plug it into a BIOS system, or vice versa, and things would Just Work(tm).

AFAIK, our current boot loader don't support yet GPT, which needs

• to map/sync the MBR table with the GPT one,
• Haiku bootable partition(s) to live in the first 4 partitions

What this universal MBR will offer? No more MBR / GPT tables sync and no first 4 partitions limit?

Kind of; what it does is:

• Test the BIOS for the availability of LBA Packet mode; proceed to GPT boot if available; else proceed to MBR boot.
• Look up the GPT for the partition signature written at byte offset 416;
  • Load up to 576kb or the entire contents of that partition, whichever is smaller, into RAM starting from address 0x7c00 (flat addressing), and jump to that address. Hopefully either the boot loader or the code from the first several sectors in the partition boot loader would fit that much memory. For those old enough to remember, this lies within the bounds of the old DOS conventional memory limits.
• If none found, proceed to MBR boot;
  • MBR boot sequence happens as usual, potentially falling back to CHS mode for royally legacy BIOSes (pre-Pentium II).

This is mostly code taken straight from FreeBSD, only modified to work with both MBR and GPT schemes, so it should be pretty compatible. However a bunch of stuff was moved around, so: caveat emptor, YMMV, THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS AS IS AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED, etc.

[jessicah]

FWIW, I've used this MBR to boot Haiku from a pure GPT partitioned disk successfully, although applying the binary file directly to my disk did break GPT temporarily (thankfully super easy to recover: parted, rename a partition, save/exit).

If there are no objections other than making sure that anyboot, writembr, etc. do the right thing, I'll look into replacing our current MBR code with this new version that's been long overdue.

Still need to look at making makebootable GPT aware though...

[jessicah]

Implemented as of hrev49238.