Opened 2 years ago

Closed 2 years ago

Last modified 2 years ago

#14993 closed bug (fixed)

Clean installs of > hrev53021 are broken, bootloader packagefs: "Bad data"

Reported by: Evgen Owned by: nobody
Priority: blocker Milestone: R1/beta2
Component: System/Boot Loader Version: R1/Development
Keywords: Cc: axeld
Blocked By: Blocking:
Platform: All

Description

I have updated form hrev53021 to hrev53042 using via bootable USB. With USB hrev53042 boots normally. Main HDD (where Haiku is installed) is visible and can be mounted. When I reboot machine Haiku Boot Loader stops and brings the menu where it says: Select boot volume (Current: None) and option "Continue booting" is inaccessible. If I plug USB and try to rescan to find bootable media, it fails to do that. When I revert back to hrev53021, it boots normally. I wasn't able to find log file, but it seems that problem is with packagefs.cpp file

Attachments (10)

9.JPG (416.8 KB ) - added by Evgen 2 years ago.
End of the log
IMG_1299.JPG (110.0 KB ) - added by Evgen 2 years ago.
checkfs results
all_packages.txt (5.3 KB ) - added by Evgen 2 years ago.
list of packages
IMG_1301.JPG (48.1 KB ) - added by Evgen 2 years ago.
IMG_1304.JPG (222.8 KB ) - added by Evgen 2 years ago.
Booting from HDD (hrev53062), end of the log
IMG_1307.JPG (692.2 KB ) - added by Evgen 2 years ago.
checksums
IMG_1309.JPG (88.7 KB ) - added by Evgen 2 years ago.
line 144-167
IMG_1310.JPG (96.2 KB ) - added by Evgen 2 years ago.
line 168-191
IMG_1311.JPG (93.2 KB ) - added by Evgen 2 years ago.
192-215
IMG_1312.JPG (93.8 KB ) - added by Evgen 2 years ago.
201-224

Download all attachments as: .zip

Change History (53)

by Evgen, 2 years ago

Attachment: 9.JPG added

End of the log

comment:1 by waddlesplash, 2 years ago

The bootloader packagefs code has not changed in some time, so, I'm guessing your disk is actually corrupted. Could you mount it from the USB and run a checkfs?

by Evgen, 2 years ago

Attachment: IMG_1299.JPG added

checkfs results

comment:2 by Evgen, 2 years ago

Done. checkfs did not report any trouble with HDD where HAIKU lives (see attachment). I will try latest build and tell here how it goes, but again - Haiku Boot Loader fails to rescan and detect this bootable USB which I used right now

comment:3 by Evgen, 2 years ago

No, latest build (hrev53052) doesn't work. I wonder, what has changed after hrev53021 that broke my setup? hrev53024 already is broken for me. Can you generate an anyboot ISO 32-bit for me for hrev53022 and hrev53023, so I can install them and we'll have more accurate info where trouble comes from?

comment:4 by waddlesplash, 2 years ago

Can you upload a copy of the "activated-packages" file from your main HDD, please?

by Evgen, 2 years ago

Attachment: all_packages.txt added

list of packages

comment:5 by Evgen, 2 years ago

Sorry, not sure that I know how to do that. With HaikuDepot I obviously can see what packages are installed, which are marked as Active and as Available, but I cannot generate list of activated-packages. What I did then is

cd /Haiku1/system/packages (my main HDD)

ls > all_packgaes.txt (attached)

Not quite sure that it is exactly what you want me to do. If not, please give a hint

comment:6 by diver, 2 years ago

Please attach this file /system/packages/administrative/activated-packages.

by Evgen, 2 years ago

Attachment: IMG_1301.JPG added

comment:7 by Evgen, 2 years ago

There is no such file in this directory (neither in HDD, nor in booted USB). Screenshot attached

comment:8 by waddlesplash, 2 years ago

Please try moving the "administrative" directory on your HDD to somewhere else on the filesystem (e.g. ~) so that it is ignored. (Don't delete it altogether just yet.)

by Evgen, 2 years ago

Attachment: IMG_1304.JPG added

Booting from HDD (hrev53062), end of the log

comment:9 by Evgen, 2 years ago

No, it does not help. Yes, kernel complains in log that it cannot open this directory (see attachment) and Haiku Boot Loader i.s stuck Moreover (if it may help), in bootable hrev53021 this file /system/packages/administrative/activated-packages does not exist too.

comment:10 by waddlesplash, 2 years ago

Well, I am stumped; nothing particularly changed with the bootloader packagefs as I said. So I don't really understand what could have possibly broken here.

comment:11 by waddlesplash, 2 years ago

Ah wait, I just realized the last "bad data" is coming from here: http://xref.plausible.coop/source/xref/haiku/src/system/boot/loader/file_systems/packagefs/packagefs.cpp#838

So this means one of your package files is corrupt. We probably should log a message there when that happens.

comment:12 by Evgen, 2 years ago

Look, there is a bunch of changes made in here https://git.haiku-os.org/haiku/commit/?h=hrev53022&id=f10ebdb1f74093594a5440651ca26be231a9bd2b I wonder is it still possible to generate an ISO image with this particular hrev that I can test?

comment:13 by waddlesplash, 2 years ago

These changes are completely irrelevant; none affect the bootloader packagefs. If they somehow affect this, then that still is not the real problem.

Since nobody else has reported boot failures, and the final error in your log is that one of your package files can't be read (actually looking at the code more closely, it's the system package).

As you can boot off a USB stick, this further implies that your package files on the main HDD are somehow corrupt.

comment:14 by Evgen, 2 years ago

What package file you're talking about? I AM REPORTING BOOT FAILURE HERE! Could you please to re-read the whole ticket again

comment:15 by waddlesplash, 2 years ago

The main Haiku package, namely "haiku-hrev1~beta1_hrev53052-1-x86_gcc2.hpkg" from your ls. The bootloader is trying to read it, and finds that it's invalid. That's what the last error message in your picture means.

Since you can boot successfully off a USB stick, this probably means the package file on your HDD is just corrupt.

comment:16 by Evgen, 2 years ago

Sir, look, I do the full wipe out and reinstall with the latest hrev. To tackle the issue I even did the full reformatting the HDD and making new partition table etc., as documentation advices. All get crooked after with HDD reboot. If I revert back to hrev53021 all is shining.

comment:17 by waddlesplash, 2 years ago

I really don't know what to tell you; the problem is obviously not a hardware compatibility change or the like since you can boot from USB.

Can you try getting the checksum of the package file on your HDD?

by Evgen, 2 years ago

Attachment: IMG_1307.JPG added

checksums

comment:18 by Evgen, 2 years ago

This file haiku-hrev1~beta1_hrev53052-1-x86_gcc2.hpkg does not exist. Please see screenshot to pick the right one. Upper terminal shows output for bootable USB, bottom one for unbootable HDD. So, what else you suggest to do?

comment:19 by Evgen, 2 years ago

Let me outline things the way I see it:

  1. All hrevs are bootable via USB.
  2. All hrevs after 53021 are stumbling up on Haiku Boot Loader.
  3. File /system/packages/administrative/activated-packages you have asked to show does not exit neither on bootable USB not on HDD after fresh install in any hrev, including ones which boots normally from HDD. Do you want me to ask people at forum if it's available in their 32-bit systems?
  4. Checksums are identical, which means that if they are wrong (for particular package you have asked) and package is broken, it's broken already in nightly ISO build.

comment:20 by waddlesplash, 2 years ago

Since you can boot from USB just fine, I really don't know what the problem could be here. As stated before, the bootloader did not particularly change.

comment:21 by Evgen, 2 years ago

Yes, that's pretty strange. I'm looking through changes made in both 53022 and 53023 and do not see anything crucial. Still something is wrong. Again, can you build two ISO for me for these two revisions?

comment:22 by waddlesplash, 2 years ago

I am really busy and don't particularly have time to make a full custom build and upload it right now. Can you please try and do it yourself?

comment:23 by waddlesplash, 2 years ago

Actually, perhaps try this first: Move the "administrative" directory out of /system/packages, then put the *older* "haiku_loader" hpkg there, and remove the *newer* one. If it's a problem with the loader somehow, that will expose it.

comment:24 by Evgen, 2 years ago

No, replacing "new" with "old" hpkg you have mentioned did not help, as well as moving the "administrative" directory. It obviously helped when I replaced the whole set of "new" packages with "old" ones. I may in principle find the troubling package this way, but will it help to actually fix this issue?

Last edited 2 years ago by Evgen (previous) (diff)

comment:25 by waddlesplash, 2 years ago

I added some debugging code in hrev53090 which should add more information to the output besides "Bad data". Please try retesting with this version and take a new picture of the output.

by Evgen, 2 years ago

Attachment: IMG_1309.JPG added

line 144-167

by Evgen, 2 years ago

Attachment: IMG_1310.JPG added

line 168-191

by Evgen, 2 years ago

Attachment: IMG_1311.JPG added

192-215

by Evgen, 2 years ago

Attachment: IMG_1312.JPG added

201-224

comment:26 by Evgen, 2 years ago

Done. Please see 4 shots of hrev53090 log (end part). Is it looks like one package has wrong size?

comment:28 by waddlesplash, 2 years ago

The "v1" code is not compiled into the bootloader, nor the kernel packagefs. The real error comes from ReaderImplBase.

comment:29 by waddlesplash, 2 years ago

From the forums, mounty reports seeing this also:

Error: Invalid package file: Total size in header (51237355) doesn't agree with total file size (51230912)

So here the difference is -6443 bytes, unlike Evgen's case, where it is indeed 1302 bytes larger.

comment:30 by waddlesplash, 2 years ago

Actually, mounty reported that there was a typo in that post, the correct second number is 51238912. So then the difference is +1557 bytes.

It's worth noting that 51238912 and 45326336, the sizes reported by the file system, are both exact multiples of 2048, the most common BFS block size. So perhaps there is a bug in the bootloader where it will occasionally report the block length allocated for the file, not the file's true size? I really don't know much about the bootloader BFS code.

comment:31 by waddlesplash, 2 years ago

The bootloader and the kernel BFS drivers both use the same bfs_inode structure, and both bootloader Stream::Size and kernel Inode::Size just return the raw "bfs_inode.data.size" value. So I really don't know what could be happening here. korli, any ideas?

comment:32 by korli, 2 years ago

No idea, hrev53022 and hrev53023 should be built and tested, this should help a bit.

comment:33 by waddlesplash, 2 years ago

haiku-r1~beta1_hrev53021-1-x86_gcc2.hpkg from the package repositories has a size of 45253742, which is of course not a multiple of 2048. I don't get why things are suddenly being rounded to that, then.

Again, the errors are occurring in the loader; and Evgen said above that using the old "loader" package did not fix the problem. There is again nothing in that range that affects the loader at all; the kernel packagefs is changed, but this only is read after the package file is opened, which of course it is erroring out on.

comment:34 by waddlesplash, 2 years ago

It's also worth noting that on the forums, mounty reported that the on-disk size (as Linux BFS saw it) of the file matched the size in the header.

comment:35 by waddlesplash, 2 years ago

Cc: axeld added
Platform: x86All

So, other users are starting to report this. What appears to be common:

1) Running the anyboot directly works. 2) After installation, it doesn't boot, with the errors described here. 3) Installing hrev53021 or earlier and then using pkgman to update works. (No reports on whether a second update after the first works or not.)

I wonder if the O_NOCACHE flag causes some subtle BFS brokenness, i.e., if one inode is opened NOCACHE, and then another process calls stat(), it gets an incorrect size? That would explain things.

CC'ing axeld for his analysis.

comment:36 by waddlesplash, 2 years ago

Milestone: UnscheduledR1/beta2
Priority: normalblocker
Summary: Latest 32-bit build fails to bootClean installs of > hrev53021 are broken, bootloader packagefs: "Bad data"

comment:37 by waddlesplash, 2 years ago

https://review.haiku-os.org/c/haiku/+/1420/2 is potentially the fix for this.

comment:38 by waddlesplash, 2 years ago

Please retest after hrev53100.

comment:39 by Evgen, 2 years ago

Clean install of hrev53101 on HDD is bootable.

comment:40 by waddlesplash, 2 years ago

Resolution: fixed
Status: newclosed

Good!

comment:41 by Evgen, 2 years ago

Yes! Awesome! Thank you. The only complain in the log I see is: PackageVolumeInfo::_InitState(): failed to parse activated-packages:No such file or directory

comment:42 by mounty, 2 years ago

hrev53101 fixes it for me as well. Thank you!

in reply to:  41 comment:43 by pulkomandy, 2 years ago

Replying to Evgen:

Yes! Awesome! Thank you. The only complain in the log I see is: PackageVolumeInfo::_InitState(): failed to parse activated-packages:No such file or directory

This is expected on first boot, as the activated-packages file is not there yet. It's created after the first boot to remember which packages to enable, avoiding to redo the dependency resolution at every boot.

Note: See TracTickets for help on using tickets.