Clean installs of > hrev53021 are broken, bootloader packagefs: "Bad data"

[Evgen]

I have updated form hrev53021 to hrev53042 using via bootable USB. With USB hrev53042 boots normally. Main HDD (where Haiku is installed) is visible and can be mounted. When I reboot machine Haiku Boot Loader stops and brings the menu where it says: Select boot volume (Current: None) and option "Continue booting" is inaccessible. If I plug USB and try to rescan to find bootable media, it fails to do that. When I revert back to hrev53021, it boots normally. I wasn't able to find log file, but it seems that problem is with packagefs.cpp file

[Evgen]

End of the log

[waddlesplash]

The bootloader packagefs code has not changed in some time, so, I'm guessing your disk is actually corrupted. Could you mount it from the USB and run a checkfs?

[Evgen]

checkfs results

[Evgen]

Done. checkfs did not report any trouble with HDD where HAIKU lives (see attachment). I will try latest build and tell here how it goes, but again - Haiku Boot Loader fails to rescan and detect this bootable USB which I used right now

[Evgen]

No, latest build (hrev53052) doesn't work. I wonder, what has changed after hrev53021 that broke my setup? hrev53024 already is broken for me. Can you generate an anyboot ISO 32-bit for me for hrev53022 and hrev53023, so I can install them and we'll have more accurate info where trouble comes from?

[waddlesplash]

Can you upload a copy of the "activated-packages" file from your main HDD, please?

[Evgen]

list of packages

[Evgen]

Sorry, not sure that I know how to do that. With HaikuDepot I obviously can see what packages are installed, which are marked as Active and as Available, but I cannot generate list of activated-packages. What I did then is

cd /Haiku1/system/packages (my main HDD)

ls > all_packgaes.txt (attached)

Not quite sure that it is exactly what you want me to do. If not, please give a hint

[diver]

Please attach this file /system/packages/administrative/activated-packages.

[Evgen]

There is no such file in this directory (neither in HDD, nor in booted USB). Screenshot attached

[waddlesplash]

Please try moving the "administrative" directory on your HDD to somewhere else on the filesystem (e.g. ~) so that it is ignored. (Don't delete it altogether just yet.)

[Evgen]

Booting from HDD (hrev53062), end of the log

[Evgen]

No, it does not help. Yes, kernel complains in log that it cannot open this directory (see attachment) and Haiku Boot Loader i.s stuck Moreover (if it may help), in bootable hrev53021 this file /system/packages/administrative/activated-packages does not exist too.

[waddlesplash]

Well, I am stumped; nothing particularly changed with the bootloader packagefs as I said. So I don't really understand what could have possibly broken here.

[waddlesplash]

Ah wait, I just realized the last "bad data" is coming from here: ​http://xref.plausible.coop/source/xref/haiku/src/system/boot/loader/file_systems/packagefs/packagefs.cpp#838

So this means one of your package files is corrupt. We probably should log a message there when that happens.

[Evgen]

Look, there is a bunch of changes made in here ​https://git.haiku-os.org/haiku/commit/?h=hrev53022&id=f10ebdb1f74093594a5440651ca26be231a9bd2b I wonder is it still possible to generate an ISO image with this particular hrev that I can test?

[waddlesplash]

These changes are completely irrelevant; none affect the bootloader packagefs. If they somehow affect this, then that still is not the real problem.

Since nobody else has reported boot failures, and the final error in your log is that one of your package files can't be read (actually looking at the code more closely, it's the system package).

As you can boot off a USB stick, this further implies that your package files on the main HDD are somehow corrupt.

[Evgen]

What package file you're talking about? I AM REPORTING BOOT FAILURE HERE! Could you please to re-read the whole ticket again

[waddlesplash]

The main Haiku package, namely "haiku-hrev1~beta1_hrev53052-1-x86_gcc2.hpkg" from your ls. The bootloader is trying to read it, and finds that it's invalid. That's what the last error message in your picture means.

Since you can boot successfully off a USB stick, this probably means the package file on your HDD is just corrupt.

[Evgen]

Sir, look, I do the full wipe out and reinstall with the latest hrev. To tackle the issue I even did the full reformatting the HDD and making new partition table etc., as documentation advices. All get crooked after with HDD reboot. If I revert back to hrev53021 all is shining.

[waddlesplash]

I really don't know what to tell you; the problem is obviously not a hardware compatibility change or the like since you can boot from USB.

Can you try getting the checksum of the package file on your HDD?

[Evgen]

checksums

[Evgen]

This file haiku-hrev1~beta1_hrev53052-1-x86_gcc2.hpkg does not exist. Please see screenshot to pick the right one. Upper terminal shows output for bootable USB, bottom one for unbootable HDD. So, what else you suggest to do?

[Evgen]

Let me outline things the way I see it:

1. All hrevs are bootable via USB.
2. All hrevs after 53021 are stumbling up on Haiku Boot Loader.
3. File /system/packages/administrative/activated-packages you have asked to show does not exit neither on bootable USB not on HDD after fresh install in any hrev, including ones which boots normally from HDD. Do you want me to ask people at forum if it's available in their 32-bit systems?
4. Checksums are identical, which means that if they are wrong (for particular package you have asked) and package is broken, it's broken already in nightly ISO build.

[waddlesplash]

Since you can boot from USB just fine, I really don't know what the problem could be here. As stated before, the bootloader did not particularly change.

[Evgen]

Yes, that's pretty strange. I'm looking through changes made in both 53022 and 53023 and do not see anything crucial. Still something is wrong. Again, can you build two ISO for me for these two revisions?

[waddlesplash]

I am really busy and don't particularly have time to make a full custom build and upload it right now. Can you please try and do it yourself?

[waddlesplash]

Actually, perhaps try this first: Move the "administrative" directory out of /system/packages, then put the *older* "haiku_loader" hpkg there, and remove the *newer* one. If it's a problem with the loader somehow, that will expose it.

[Evgen]

No, replacing "new" with "old" hpkg you have mentioned did not help. It obviously helped when I replaced the whole set of "new" packages with "old" ones. I may in principle find the troubling package this way, but will it help to actually fix this issue?

[waddlesplash]

I added some debugging code in hrev53090 which should add more information to the output besides "Bad data". Please try retesting with this version and take a new picture of the output.

[Evgen]

line 144-167

[Evgen]

line 168-191

[Evgen]

192-215

[Evgen]

201-224

[Evgen]

Done. Please see 4 shots of hrev53090 log (end part). Is it looks like one package has wrong size?

[korli]

a difference of 1302 bytes at ​http://xref.plausible.coop/source/xref/haiku/src/kits/package/hpkg/v1/PackageReaderImplV1.cpp#454

[waddlesplash]

The "v1" code is not compiled into the bootloader, nor the kernel packagefs. The real error comes from ​ReaderImplBase.

[waddlesplash]

From the forums, mounty ​reports seeing this also:

Error: Invalid package file: Total size in header (51237355) doesn't agree with total file size (51230912)

So here the difference is -6443 bytes, unlike Evgen's case, where it is indeed 1302 bytes larger.

[waddlesplash]

Actually, mounty reported that there was a typo in that post, the correct second number is 51238912. So then the difference is +1557 bytes.

It's worth noting that 51238912 and 45326336, the sizes reported by the file system, are both exact multiples of 2048, the most common BFS block size. So perhaps there is a bug in the bootloader where it will occasionally report the block length allocated for the file, not the file's true size? I really don't know much about the bootloader BFS code.

[waddlesplash]

The bootloader and the kernel BFS drivers both use the same bfs_inode structure, and both bootloader Stream::Size and kernel Inode::Size just return the raw "bfs_inode.data.size" value. So I really don't know what could be happening here. korli, any ideas?

[korli]

No idea, hrev53022 and hrev53023 should be built and tested, this should help a bit.

[waddlesplash]

haiku-r1~beta1_hrev53021-1-x86_gcc2.hpkg from the package repositories has a size of 45253742, which is of course not a multiple of 2048. I don't get why things are suddenly being rounded to that, then.

Again, the errors are occurring in the loader; and Evgen said above that using the old "loader" package did not fix the problem. There is again nothing in that range that affects the loader at all; the kernel packagefs is changed, but this only is read after the package file is opened, which of course it is erroring out on.

[waddlesplash]

It's also worth noting that on the forums, mounty reported that the on-disk size (as Linux BFS saw it) of the file matched the size in the header.

[waddlesplash]

So, other users are starting to report this. What appears to be common:

1) Running the anyboot directly works. 2) After installation, it doesn't boot, with the errors described here. 3) Installing hrev53021 or earlier and then using pkgman to update works. (No reports on whether a second update after the first works or not.)

I wonder if the O_NOCACHE flag causes some subtle BFS brokenness, i.e., if one inode is opened NOCACHE, and then another process calls stat(), it gets an incorrect size? That would explain things.

CC'ing axeld for his analysis.

[waddlesplash]

​https://review.haiku-os.org/c/haiku/+/1420/2 is potentially the fix for this.

[waddlesplash]

Please retest after hrev53100.

[Evgen]

Clean install of hrev53101 on HDD is bootable.

[waddlesplash]

Good!

[Evgen]

Yes! Awesome! Thank you. The only complain in the log I see is: PackageVolumeInfo::_InitState(): failed to parse activated-packages:No such file or directory

[mounty]

hrev53101 fixes it for me as well. Thank you!

[pulkomandy]

Replying to Evgen:

Yes! Awesome! Thank you. The only complain in the log I see is: PackageVolumeInfo::_InitState(): failed to parse activated-packages:No such file or directory

This is expected on first boot, as the activated-packages file is not there yet. It's created after the first boot to remember which packages to enable, avoiding to redo the dependency resolution at every boot.