bootstrap is complex and error prone

[kallisti5]

Our current bootstrap process is extremely complex and error prone. We really need to reduce some of the complexity of the initial bootstrap. Port progress is at a standstill as you have to have an entire bootstrap completed before doing any port work on bootloader, kernel, etc.

Ideas:

• Create a new single repo with the most basic tools forked. We tune the build scripts to be easily executable within the initial bootstrap environment. We generate one haiku_bootstrap_env hpkg that provides a basic environment. (coreutils,awk,bison,curl,mawk,flex,m4,make,sed)
• Fork busybox to provide all of the core utils + sed,awk,wget,etc. I have a busybox binary generated on Haiku after some tweaking... their build system is really rough however and can be flaky when automated.
• Introduce a new @basic-raw target which doesn't require a bootstrap. Just include bootloader, filesystem drivers, kernel.

[kallisti5]

arm bootstrap after two days of work.

[kallisti5]

busybox example.

make haiku_defconfig make

/Builds/busybox-1.23.1> uname -a
Haiku kallbuild01 1 hrev48786 Feb  8 2015 19:48:00 x86_64 x86_64 Haiku
/Builds/busybox-1.23.1> ./busybox
BusyBox v1.23.1 (2015-02-27 14:46:56 CST) multi-call binary.
BusyBox is copyrighted by many authors between 1998-2012.
Licensed under GPLv2. See source distribution for detailed
copyright notices.

Usage: busybox [function [arguments]...]
   or: busybox --list
   or: function [arguments]...

	BusyBox is a multi-call binary that combines many common Unix
	utilities into a single executable.  Most people will create a
	link to busybox for each function they wish to use and BusyBox
	will act like whatever it was invoked as.

Currently defined functions:
	[, [[, awk, base64, basename, cat, chgrp, chmod, chown, chroot, cksum,
	cp, cut, date, dd, diff, dirname, echo, env, expand, expr, false, fold,
	grep, groups, head, id, install, ln, logname, md5sum, mkdir, mv, nohup,
	patch, printenv, printf, pwd, readlink, realpath, rm, rmdir, sed, seq,
	sha1sum, sha256sum, sha3sum, sha512sum, sleep, sort, split, stty, sum,
	sync, tac, tail, tee, test, touch, tr, true, tty, uname, uniq, unlink,
	usleep, uudecode, uuencode, wc, which, whoami, xargs, yes

/Builds/busybox-1.23.1> uname -a | ./busybox awk '{ print $1 }'
Haiku

[kallisti5]

busybox patches for Haiku

[bonefish]

Replying to kallisti5:

Our current bootstrap process is extremely complex and error prone. We really need to reduce some of the complexity of the initial bootstrap. Port progress is at a standstill as you have to have an entire bootstrap completed before doing any port work on bootloader, kernel, etc.

Save for Ithamar's BeGeistert spurts of activity and the x86-64 port, port progress has been at a standstill for about a decade. And that really doesn't have anything to do with our bootstrap process. In fact we didn't even have one before PM, i.e. the situation has much improved already.

For a pre-userland boot -- which is the state of all non x86* ports -- you don't even need to build a complete image. You should get away with defining a minimal image and adapting/hacking a few source packages (probably only zlib and libsolv).

Ideas:

• Create a new single repo with the most basic tools forked. We tune the build scripts to be easily executable within the initial bootstrap environment. We generate one haiku_bootstrap_env hpkg that provides a basic environment. (coreutils,awk,bison,curl,mawk,flex,m4,make,sed)

That is basically what we already do. haikuports.cross is that repository. I don't see any advantage in building everything into a single package instead of building multiple packages as we do now. The bootstrap process obviously builds a lot more than you need for just booting the system (since it needs to be a complete build environment). But that's just a question of defining a respective image, so that only the required subset is built.

• Fork busybox to provide all of the core utils + sed,awk,wget,etc. I have a busybox binary generated on Haiku after some tweaking... their build system is really rough however and can be flaky when automated.

Sure, add a recipe to haikuports.cross.

• Introduce a new @basic-raw target which doesn't require a bootstrap. Just include bootloader, filesystem drivers, kernel.

Yep, that makes a lot of sense. As mentioned above, you will need a few source packages already, though. I wouldn't call the profile basic. We already have minimal which contains a lot more. I'd suggest something like kernel-only.

[pulkomandy]

You don't even need to do the bootstrap to build a minimal image. I have uploaded packages to the Haiku ARM repository and they are enough to get the minimal image built and running.

The script used to do this converts the bootstrap packages to non-bootstrap ones, and is available here: ​http://cgit.haiku-os.org/haiku/tree/3rdparty/pulkomandy/unbootstrap.sh . This is good enough to get started on kernel work without touching the bootstrap process at all.

Of course since begeistert (when this was last used) there were more packages outsourced, and the unbootstrap script can be used to add more of the packages to the repo.

[haiqu]

What I'd like to see is a concise description of the bootstrap process as it has been done by whoever has been there before. So far I can get a single package:

haiku_cross_devel_sysroot_stage0_arm.hpkg

I can't find any description of what to do with it or where to go from there.

Downloading prebuilt stuff doesn't constitute bootstrapping and teaches nothing about the process at all. In fact anyone considering that as a solution should go look up the definition of 'bootstrap' and contemplate it for about a week.

I've got an Arm compiler that works, the haikuports and haikuports.cross repositories set up, and have followed every decription of a "working procedure" on the internet published in the past 3 years. None of them produce a result.

[pulkomandy]

The real bootstrap process needs Haiku to be advanced enough to run userland, because the packages will be built there. For the ARM port, this is not the case yet. So doing a real bootstrap build of the ARM port is simply not possible yet.

If you use a known configuration, that is, x86 or x86_64, building from a Linux host (this is the only thing that ever worked, and it worked only once to my knowledge), it may be possible to get the bootstrap process to run through.

On the ARM port, and again, only from a Linux host (getting this to work from a Haiku host would be a lot more difficult), the best you can get is a minimal set of "bootstrap" packages. These would normally be used to build a bare minimum Haiku userland, which you can then boot. Inside this environment, if it worked, you could run haikuporter to complete the bootstrap process by building all the final packages. However, currently the ARM port barely has a running kernel, and no userland. So, this won't work in the current state.

[pulkomandy]

Removed "patch" bit since this does not target HAiku sources (it is a patch for busybox). A recipe for busybox may be a good idea, however.

[pulkomandy]

I think we can close this. The bootstrap is still complex, but there isn't a lot more we can do about this, we have the dependencies quite reduced already. We have succesfully used it for various architectures now, so I think we can close this.