Package Management Ideas
This page is a place to hash out ideas regarding Haiku's package management (and creation). The following is a draft specification for the package management system to be included in R1. It is based on (1), (2) and the discussion in (5). The draft does not yet cover everything from (1), however.
This section describes the intended user experience.
HaikuBits is a complete directory of software for the Haiku platform. It is the one place a user needs when looking for Haiku software. While it does not host all software binaries, it does list 99.9% of all software available for Haiku.
Software is classified into a number of categories. For each software, HaikuBits provides a short description, one or more screenshots, a link to the author's homepage and a download link. Additionally, a software's page has a community rating and important information such as security notices.
A bundle is a single file that contains the binaries, data files and documentation. This makes software self-contained and easy to handle. An application can be run by double-clicking the bundle icon. Obvious exceptions to this rule are drivers and libraries. These have to be installed to be of any use. The contents of a bundle can be inspected by opening the bundle by means of a context-menu option which opens the bundle in Tracker just like a directory.
Optionally, an application bundle can be installed by moving it to
/boot/apps (system-wide) or in
/boot/home/<user>/apps (user-local). Another option is to right-click the icon and select "install for everyone" (only admins) or "install only for me". Any initial configuration (accepting a license) can be performed the first time a bundle is being run.
Libaries can be installed the same way. The user normally does not have to install libraries manually, as the package manager will do so when it is needed, asking the user for permission.
When an application bundle has been installed, shortcuts to the application appear in the Deskbar menu. This menu is subdivided into a number of a predefined categories (games, graphics, internet, ...) that match those on HaikuBits.
Drivers ... ?
Uninstallation is performed by simply removing the bundle from
A user's application settings are kept when a bundle is uninstalled or deleted. The system however provides a comprehensive listing of applications for which user settings exist. The user can choose to delete settings for each of the bundles.
Application folders (
/boot/home/<user>/apps) display the list of installed bundles including information (description, availability of updates, security risk warning), just like the mail folder in BeOS displays emails.
The user does not have to worry about dependencies. If a bundle depends on other bundles, the package manager should make sure they are fulfilled (on run). If a dependency is not fulfilled, the system will ask the user for confirmation and automatically install any required bundles. This requires an internet connection.
Alternatively, a user can download a fat bundle that includes the dependencies as well.
All installed software can be checked for new versions. For each of the installed applications, the changes with respect to the installed version can be displayed. Updates are flagged as 'strongly recommended' when security problems are known. The user can select which applications he/she would like to update and have the system perform the updates.
By default, the updater tool does not show libraries in order to keep the list of updates as short as possible and understandable by the non-technical user. Libaries should only be updated when there are known problems with them.
In this section the implementation of the system is discussed.
A bundle is a compressed disk image that contains:
- application executable(s) & data
- hash (integrity-check)
- category (for grouping applications in the Deskbar menu)
- shortcuts to appear in the Deskbar menu
- a default shortcut to run when the bundle is being "run"
- right-clicking bundle could offer the option of opening a help document about the application
There are a number of different bundle types:
Library, driver and font bundles have to be installed.
Install or not?
As bundles have to be compressed for distribution, they will need to be uncompressed at some point. There are two options:
- Before using a bundle, it is decompressed. This is very similar to installing, which many wanted to avoid.
- The bundle's contents are decompressed on access. This is less efficient as decompression needs to be performed on each access, as opposed to only one time during installation. Performance of large applications and heavy games in particular will suffer.
It is possible to split application bundles into two sub-types; those that have to be installed, and those that can be run as-is.
File System Organization
Native Haiku applications are contained in their own directory as before. Organization of ports is more complicated.
By means of a union pkgfs (3). All ports are mounted under
It is not clear how multiple versions of libraries and applications can be handled in this scheme.
By means of assignfs (4). Each port receives its own unique assign:
global settings/user settings
Multiple Application Version
how to handle
Dependency hell is a problem mostly for ports. That does not mean it can be ignored. At least in the early years of Haiku, ports will be an important source of software.
Bundles are always fat bundles. All required libraries are included in the bundle. Problem solved! However, this very area-inefficient. Nor is it a realistic solution for bundles that depend on large packages like Python or Perl.
In order to solve conflicting dependencies, it is necessary to be able to have multiple versions of a library installed. Even worse, some libraries can be built with different options.
To make this work, it is obvious that a central bundle repository is required: HaikuBits. Alongside offering a browsable directory of software like BeBits, it stores information about dependencies. Dependency information (problems arising from certain combinations of bundles) is updated by the community.
An example. When ABC-1.0 is released, its dependency libfoo is at version 1.2.10. Bundle ABC-1.0 specifies "libfoo >= 1.2.10" as a dependency. Later, when libfoo 1.2.12 is released, it appears that this breaks ABC-1.0. HaikuBits is updated to indicate this: "libfoo >= 1.2.10 && != 1.2.12". When ABC-1.0 is now downloaded from HaikuBits, the bundle contains the updated information. A software updating tool can also check HaikuBits to see whether dependencies are still OK.
Because the act of porting can introduce additional incompatibilities, each port should be tagged with a revision number to uniquely identify it. Revisions can also be used to differentiate between ports with different build options. Specifying build options in the dependency information seems overkill anyway, as we should strive to have as few port revisions as possible (developers should have dependencies installed as bundles before porting).
The bundle metadata needs to be extended to include information about the dependencies:
- minimum/maximum version
- preferred version/revision
- non-working versions/revisions
While bundles will not be available for download for retail software, it still makes sense to record dependency information about it on HaikuBits.
Having an application use a particular library version can be done by manipulating LD_LIBRARY_PATH or by virtually placing the library in the applications directory by means of assignfs or pkgfs.
Note the important difference with typical Linux package management systems. In Linux, the repository typically offers only one version of a particular package. This is the result of keeping all packages in the repository in sync, in order to avoid conflicting dependencies. In the proposed system however, the user is free to install any version of a bundle, as there is no need for any global synchronization of all bundles.
(and it's tool: pacman)
libalpm is the package management library used on ArchLinux, most people know it as "pacman" since that's the main tool to use, however, all the functionality is part of the libalpm library which could be utilized to create a nice GUI frontend for the package manager. It of course can also be adapted.
It uses libarchive to extract archives, and either libdownload or libfetch to download files - although one can also have it use an external command, like curl or wget.
The current status is this:
As far as libfetch is concerned: compiles and is linked to, but it doesn't really work, so I'm using curl instead - it works like a charm.
The important part: libarchive needed some work to support zip files in a useful way. Basically, it now supports seeking (which it didn't before), the central-directory headers for ZIP files (so it supports stuff like symlinks), and BeOS file attributes! Also, when reading from a source which doesn't allow seeking (... which are... none - on our case) it simply reads the local headers, but can also - if explicitly requested - provide "update"-entries to update the raw data when the central directory is reached (but those are of no importance anymore).
What's good about libalpm? Well, it provides useful configuring mechanisms, it stores dependencies and can also give you a list of which packages require a certain package. It keeps a database containing package information, including a file list. Configuration files in packages can be listed as such, which causes them to be installed as *.pacnew when they are upgraded (unless the new and old files equal - an md5 sum check is used there.) It provides the ability to use different database directories which allows us to have an automated way of creating package bundles. For instance, I can set the installdir to /tmp/mypackage and install the game "einstein" including its dependencies there, then move /tmp/mypackage/einstein/common/lib to /tmp/mypackage/einstein/lib, remove the unnecessary manpages, share files (well, usually anything else which is in the common/ folder), and then strip those dependencies from einstein's .PKGINFO file and create a bundled package which I can then install normally to say /boot/apps.
Another useful feature is the possibility to change the root directory. When a package contains a .INSTALL script, libalpm chroot()s into the root directory, cd()s into the installation directory, and then executes the .INSTALL script (which means, that install-scripts can and should work relative to the installation directory, although, if necessary, the absolute path is available in $PWD)
Also, libalpm works similar to an actual database. It doesn't blindly attempt to install a package, but first check for file conflicts, see if any files need backups or configuration files need to be installed as .pacnew, and then installs a package. If you install multiple packages at once, then it only either installs all of them, or none. It allows you to find the owning package of a file in the filesystem as well as listing all the files and dependencies of a package.
Where does it get the packages form? Two possibilities: One can use package files directly - which could be made in such a way that you could also just unzip them. In fact, it might be useful to put the .PKGINFO into the zip file as some extra data which is not unzipped when simply using `unzip´, although package creation is easier if it's just a file. The other one being repositories. The pacman utility currently allows you to list repositories like this in pacman.conf:
[core] Server = file:///MyRepositories [devel] Server = file:///MyRepositories [public] Server = http://www.public-repository.com/
When you synchronize the repository databases, pacman downloads the file <Server>/<Reponame>.db.tar.gz which contains a list of packages with dependencies.
When you install a package from such a repository, it downloads them from the very same location: <Server>/<Package File>
The repo.db.tar.gz files are currently created using the tools
repo-remove provided in the pacman package. Those extract information from the .PKGINFO file and put it into the database which can be used as a repository then.
Here's a little log of using pacman to install a package file, and bundle a package with dependencies together into one package file. http://stud4.tuwien.ac.at/~e0725517/using-pacman-on-haiku.log.txt
- It has been used on archlinux for a long time - so it works.
- It's obviously possibly to compile and use it on haiku
- Since most of its functionality is part of a library, it can be reused to build a GUI-application utilizing libalpm
- pacman also provides scripts for building packages using a PKGBUILD script.
- Likely to cause unwanted restrictions in the package management system.
Brainstorming Results from BeGeistert
These features were discussed/written down at BeGeistert:
- Integrated app to add packages
- Knows about repositories
- Defined protocol to add repositories that everyone can use (for example through description files with a special MIME type)
- Multiple installations of the same package (if the software supports it)
- Maintain shared libraries
- runtime_loader uses package info to resolve libraries
- Install packages per user
- Repositories support keys to verify packages
- Package database of installed packages
- Packagemanagement API
- Binary diffing for packages
- Export/publish the set of installed packages to another system
- System updates
- Quality of packages (QA integrated into the process of releasing a package)
- Property of live update possible
A package format has to meet the following requirements:
- It must be able to store BeOS/Haiku file attributes.
- If the package shall be used directly (i.e. without prior extracting) by a package file system, fast random access to the file data must be possible. This disqualifies zipped TAR like formats.
[PackageFormat The Haiku Package format] specifies a format meeting these requirements.
Other package managers to steal ideas from:
- OSNews: Decentralised Installation Systems - article by the 0install author
- Package management system