Allow using Hpkg files as regular archives

[X512]

This is hrev53967.

It will be nice to allow using hpkg files as regular archives, not only as packages. Unlike other archive formats, Hpkg archives supports all BFS features and mounting. It can be used in same way as WIM archives in Windows and squashfs in Linux. It can be also used for BFS volume backups.

Problems need to be solved:

1. package command: allow creating packages without .PackageInfo.
2. Mount on click non-package hpkg files in same way as BFS images.

[X512]

stub package info

[X512]

Attached stub.PackageInfo can be used as workaround to create hpkg archive.

[waddlesplash]

-1.

packagefs images are block-compressed and indexed so they are optimized for fast random access, which makes them very computationally expensive to create and not as efficient for data storage as standard archives.

You can already mount BFS image files as partitions, and then compress these with any archive tool (and of course you can modify them, too) making them much more suitable for this purpose.

[X512]

Replying to waddlesplash:

packagefs images are block-compressed and indexed so they are optimized for fast random access, which makes them very computationally expensive to create and not as efficient for data storage as standard archives.

Compression can be disabled if this is an issue. Something like hpkg.xz can be used if size is important and mounting is not needed.

You can already mount BFS image files as partitions, and then compress these with any archive tool (and of course you can modify them, too) making them much more suitable for this purpose.

There are no method for creating BFS image from folder including automatic size detection. HPKG is also more simple (less risk of errors and possibility to make extracting software for different platforms) and space efficient than BFS. Making non-compressed HPKG is much faster than copying files to different BFS volume.

Creating HPKG can be also piped by ssh to backup server.

[waddlesplash]

There are no method for creating BFS image from folder including automatic size detection.

Then we should make one, it should not be that hard to do, in fact. Again, packagefs are really not designed for this usecase, nor do we want to use it for this usecase.

Making non-compressed HPKG is much faster than copying files to different BFS volume.

Only when the file cache is used I think; probably mounting a volume with the cache disabled is as fast as your I/O can make it.

[X512]

Replying to waddlesplash:

Only when the file cache is used I think; probably mounting a volume with the cache disabled is as fast as your I/O can make it.

When creating HPKG archive, data is written sequentially, while in BFS data is written at random locations, that is slower and make piping impossible. BFS also don't handle a lot of small files well, so BFS image with many small files will be much larger than HPKG.

[waddlesplash]

TAR supports extended attributes: ​https://www.gnu.org/software/tar/manual/html_node/Extended-File-Attributes.html

We have patches for ZIP format to make it support BeOS typed extended attributes (and it does not support xattrs at all); since TAR supports xattrs natively, we should be able to add support for typed extended attributes to it, too. This should be a much better solution overall, since everything reads/writes TAR, and whatever does not support xattrs, that's OK.

[pulkomandy]

Yes, just use zip or tar as everyone is doing since BeOS is available. No need to use hpkg for that, the format is not designed for it.

[X512]

Replying to pulkomandy:

Yes, just use zip or tar as everyone is doing since BeOS is available. No need to use hpkg for that, the format is not designed for it.

ZIP fails with zip error: Out of memory (was adding files to zip file), TAR don't support attributes. HPKG is only working archiving solution for now.

[waddlesplash]

zip works just fine here (package version is 3.0-4.) We should, as I said, patch TAR. We greatly prefer the right solution over the easy solution, as a matter of course...

[X512]

Replying to waddlesplash:

package version is 3.0-4.

Same for me, but it fails to create archive from folder with a lot of small files about 10GB total on 32 bit system. It seems to load file structure to RAM.

HPKG created successfully with package create -0 -i /path/to/stub.PackageInfo /boot/Backup.hpkg, but extracting failed with Error: Malformed TOC strings section. Attempt to put it in packages folder cause KERN: package_daemon [3860316314: 945] failed to init package for file "Backup.hpkg".

[X512]

Replying to waddlesplash:

We should, as I said, patch TAR.

Created ticket on haikuPorts: ​https://github.com/haikuports/haikuports/issues/4793.