Changes between Version 16 and Version 17 of Obsolete/MovedToTree/PackageManagement/FileFormat


Ignore:
Timestamp:
May 25, 2013, 2:12:34 AM (12 years ago)
Author:
bonefish
Comment:

Version 2 changes

Legend:

Unmodified
Added
Removed
Modified
  • Obsolete/MovedToTree/PackageManagement/FileFormat

    v16 v17  
    2626   the file, it specifies meta data of the package as a whole.
    2727
     28The TOC and Package Attributes sections aren't really separate sections, as they are stored at the end of the heap.
     29
    2830All numbers in the HPKG are stored in big endian format or [http://en.wikipedia.org/wiki/LEB128 LEB128] encoding.
    2931
     
    3941        uint16  version;
    4042        uint64  total_size;
    41 
    42         // package attributes section
    43         uint32  attributes_compression;
    44         uint32  attributes_length_compressed;
    45         uint32  attributes_length_uncompressed;
     43        uint16  minor_version;
     44
     45        uint16  heap_compression;
     46        uint32  heap_chunk_size;
     47        uint64  heap_size_compressed;
     48        uint64  heap_size_uncompressed;
     49
     50        uint32  attributes_length;
    4651        uint32  attributes_strings_length;
    4752        uint32  attributes_strings_count;
    48 
    49         // TOC section
    50         uint32  toc_compression;
    51         uint64  toc_length_compressed;
    52         uint64  toc_length_uncompressed;
     53        uint32  reserved1;
     54
     55        uint64  toc_length;
    5356        uint64  toc_strings_length;
    5457        uint64  toc_strings_count;
     
    5962   The string 'hpkg' (B_HPKG_MAGIC).
    6063 header_size::
    61    The size of the header.
     64   The size of the header. This is also the absolute offset of the heap.
    6265 version::
    6366   The version of the HPKG format the file conforms to. The current version is
     
    6568 total_size::
    6669   The total file size.
    67 
    68  attributes_compression::
    69    The compression algorithm used for the package attributes section.
    70  attributes_length_compressed::
    71    The compressed size of the package attributes section. Equals
    72    attributes_length_uncompressed, if the section is not compressed.
    73  attributes_length_uncompressed::
     70 minor_version::
     71   The minor version of the HPKG format the file conforms to. The current minor version is
     72   0 (B_HPKG_MINOR_VERSION). Additions of new attributes to the attributes or TOC sections should
     73   generally only increment the minor version. When a file with a greater minor version is
     74   encountered, the reader should ignore unknown attributes.
     75
     76 heap_compression::
     77   Compression format used for the heap.
     78 heap_chunk_size::
     79   The size of the chunks the uncompressed heap data are divided into.
     80 heap_size_compressed::
     81   The compressed size of the heap. This includes all administrative data (the chunk size array).
     82 heap_size_uncompressed::
     83   The uncompressed size of the heap. This is only the size of the raw data (including the TOC
     84   and attributes section), not including administrative data (the chunk size array).
     85
     86 attributes_length::
    7487   The uncompressed size of the package attributes section.
    7588 attributes_strings_length::
     
    7891   The number of entries in the strings subsection of the package attributes section.
    7992
    80  toc_compression::
    81    The compression algorithm used for the TOC section.
    82  toc_length_compressed::
    83    The compressed size of the TOC section. Equals
    84    toc_length_uncompressed, if the section is not compressed.
    85  toc_length_uncompressed::
     93 reserved1::
     94   Reserved for later use.
     95
     96 toc_length::
    8697   The uncompressed size of the TOC section.
    8798 toc_strings_length::
     
    89100 toc_strings_count::
    90101   The number of entries in the strings subsection of the TOC section.
     102
     103
     104=== Heap ===
     105
     106The heap provides storage for arbitrary data. Data from various sources are concatenated without padding or separator, forming the uncompressed heap. A specific section of data is usually referenced (e.g. in the TOC and attributes sections) by an offset and the number of bytes. These references always point into the uncompressed heap, even if the heap is actually stored in a compressed format. The `heap_compression` field in the header specifies which format is used. The following values are defined:
     107
     108||0||B_HPKG_COMPRESSION_NONE||no compression||
     109||1||B_HPKG_COMPRESSION_ZLIB||zlib (LZ77) compression||
     110
     111The uncompressed heap data are divided into equally sized chunks (64 KiB). Each individual chunk may be stored compressed or not. A uint16 array at the end of the heap contains the actual in-file (compressed) size of each chunk, save for the last one, which is omitted since it is implied. A chunk is only stored compressed, if compression actually saves space. That is if the chunk's compressed size equals its uncompressed size, the data aren't compressed.
     112
     113The TOC and the package attributes sections are stored (in this order) at the end of the uncompressed heap. The offset of the package attributes section data is therefore `heap_size_uncompressed - attributes_length` and the offset of the TOC section data `heap_size_uncompressed - attributes_length - toc_length`.
    91114
    92115
     
    129152The strings subsections consists of a list of null-terminated UTF-8 strings. The section itself is terminated by a 0 byte.
    130153
    131 Each string is implicity assigned the (null-based) index at which the it appears in the list, i.e. the nth string has the index n - 1. The string is referenced by this index in the main TOC subsection.
     154Each string is implicitly assigned the (null-based) index at which it appears in the list, i.e. the nth string has the index n - 1. The string is referenced by this index in the main TOC subsection.
    132155
    133156==== Main TOC ====
     
    143166
    144167The attribute tag encodes four pieces of information:
    145   {{{(encoding << 10) + (hasChildren << 9) + (dataType << 6) + id + 1}}}
     168  {{{(encoding << 11) + (hasChildren << 10) + (dataType << 7) + id + 1}}}
    146169
    147170 encoding::
     
    173196
    174197   ||0||B_HPKG_ATTRIBUTE_ENCODING_RAW_INLINE||unsigned LEB128: size; followed by raw bytes||
    175    ||1||B_HPKG_ATTRIBUTE_ENCODING_RAW_HEAP||unsigned LEB128: size; unsigned LEB128: offset into heap||
     198   ||1||B_HPKG_ATTRIBUTE_ENCODING_RAW_HEAP||unsigned LEB128: size; unsigned LEB128: offset into the uncompressed heap||
    176199
    177200
     
    180203The package attributes section contains a list of attribute trees, just like
    181204the TOC section. The structure of this section follows the TOC, i.e. there's a subsection for shared strings and a subsection that stores a list of attribute entries terminated by a 0 byte. An entry has the same format as the ones in the TOC (only using different attribute IDs).
    182 
    183 
    184 === Section Compression ===
    185 
    186 The TOC and the package attributes section can be compressed. Which compression algorithm is used is specified by the {{{toc_compression}}} respectively the {{{attributes_compression}}} field in the header. The following values are defined:
    187 
    188 ||0||B_HPKG_COMPRESSION_NONE||no compression||
    189 ||1||B_HPKG_COMPRESSION_ZLIB||zlib (LZ77) compression||
    190 
    191205
    192206
     
    267281 - '''Value:''' Name of the user owning the file.
    268282 - '''Allowed Values:''' Any non-empty string.
     283 - '''Default Value:''' The user owning the installation location where the package is activated.
    269284 - '''Child Attributes:''' none
    270285
     
    273288 - '''Value:''' Name of the group owning the file.
    274289 - '''Allowed Values:''' Any non-empty string.
     290 - '''Default Value:''' The group owning the installation location where the package is activated.
    275291 - '''Child Attributes:''' none
    276292
     
    331347 - '''Type:''' data
    332348 - '''Value:''' Raw data of a file or attribute.
    333  - '''Allowed Values:''' Any value, if uncompressed, otherwise see below.
    334  - '''Child Attributes:'''
    335    - B_HPKG_ATTRIBUTE_ID_DATA_COMPRESSION: The compression algorithm used for
    336      storing the data.
    337    - B_HPKG_ATTRIBUTE_ID_DATA_SIZE: The size of the uncompressed data.
    338    - B_HPKG_ATTRIBUTE_ID_DATA_CHUNK_SIZE: The size of an uncompressed data chunk.
    339 
    340 ==== B_HPKG_ATTRIBUTE_ID_DATA_COMPRESSION ("data:compression") ====
    341  - '''Type:''' uint
    342  - '''Value:''' ID of the data compression algorithm.
    343  - '''Allowed Values:'''
    344 
    345    ||0||B_HPKG_COMPRESSION_NONE||no compression||
    346    ||1||B_HPKG_COMPRESSION_ZLIB||zlib (LZ77) compression||
    347  - '''Default Value:''' B_HPKG_COMPRESSION_NONE
    348  - '''Child Attributes:''' none
    349 
    350 ==== B_HPKG_ATTRIBUTE_ID_DATA_SIZE ("data:size") ====
    351  - '''Type:''' uint
    352  - '''Value:''' Size of the uncompressed data.
    353  - '''Allowed Values:''': Any value.
    354  - '''Default Value:''' Size of the compressed data.
    355  - '''Child Attributes:''' none
    356 
    357 ==== B_HPKG_ATTRIBUTE_ID_DATA_CHUNK_SIZE ("data:chunk_size") ====
    358  - '''Type:''' uint
    359  - '''Value:''' Size of an uncompressed data chunk.
    360  - '''Allowed Values:''': Any value.
    361  - '''Default Value:'''
    362     - If not compressed: 0
    363     - If B_HPKG_COMPRESSION_ZLIB compressed: 64 * 1024
    364  - '''Child Attributes:''' none
     349 - '''Allowed Values:''' Any value.
    365350
    366351==== B_HPKG_ATTRIBUTE_ID_SYMLINK_PATH ("symlink:path") ====
     
    375360
    376361The TOC can directly contain any number of attributes of the B_HPKG_ATTRIBUTE_ID_DIRECTORY_ENTRY type, which in turn contain descendant attributes as specified in the previous section. Any other attributes are ignored.
    377 
    378 
    379 === Data Compression ===
    380 
    381 Data referred to by an B_HPKG_ATTRIBUTE_ID_DATA attribute will be the raw data, if uncompressed. If compressed, the data have a special format, that allows for fast random access.
    382 
    383 ==== B_HPKG_COMPRESSION_ZLIB ====
    384 
    385 The original data are split into equally sized chunks and compressed individually. The compressed data chunks are stored (in order) without padding, preceded by an uint64 array specifying the relative offsets of the compressed data of each chunk. The offsets are relative to the first byte following the offset array. Since the first chunk is always at offset 0, its array element is omitted. Therefore uncompressed data split into n chunks will have n - 1 offset array elements.
    386 
    387 The size of the compressed chunks is implied by the offset differences (respectively for the last chunk the difference to the total size). A compressed chunk is always shorter than the uncompressed chunk. If the chunk data couldn't be compressed, the data are stored uncompressed and the size of the stored chunk will be equal to the uncompressed chunk size.
    388362
    389363
     
    494468   ||1||B_PACKAGE_ARCHITECTURE_X86||x86, 32-bit, built with gcc4||
    495469   ||2||B_PACKAGE_ARCHITECTURE_X86_GCC2||x86, 32-bit, built with gcc2||
     470   ||3||B_PACKAGE_ARCHITECTURE_SOURCE||source code, doesn't depend on the system architecture||
    496471 - '''Child Attributes:''' none
    497472
     
    629604 - '''Child Attributes:''' none
    630605
     606==== B_HPKG_ATTRIBUTE_ID_PACKAGE_GLOBAL_SETTINGS_FILE ("package:global-settings-file") ====
     607 - '''Type:''' string
     608 - '''Value:''' Relative path of a global settings file either included in the package or created by the included software. If the settings file is included in the package, it will be installed upon activation. In this case the attribute must contain a B_HPKG_ATTRIBUTE_ID_PACKAGE_SETTINGS_FILE_UPDATE_TYPE child attribute.
     609 - '''Allowed Values:''' Installation location relative path (i.e. "settings/...").
     610 - '''Child Attributes:'''
     611   - B_HPKG_ATTRIBUTE_ID_PACKAGE_SETTINGS_FILE_UPDATE_TYPE: Specifies what to do with the settings file on package update.
     612
     613==== B_HPKG_ATTRIBUTE_ID_PACKAGE_USER_SETTINGS_FILE ("package:user-settings-file") ====
     614 - '''Type:''' string
     615 - '''Value:''' Relative path of a user settings file created by the included software or required by the software to be created by the user.
     616 - '''Allowed Values:''' Installation location relative path (i.e. "settings/...").
     617 - '''Child Attributes:'''
     618   - B_HPKG_ATTRIBUTE_ID_PACKAGE_SETTINGS_FILE_TEMPLATE: A template for the settings file.
     619
     620==== B_HPKG_ATTRIBUTE_ID_PACKAGE_SETTINGS_FILE_UPDATE_TYPE ("package:settings-file-update-type") ====
     621 - '''Type:''' uint
     622 - '''Value:''' Specifies what to do on package update when the settings file provided by the package has been changed by the user.
     623 - '''Allowed Values:'''
     624
     625   ||0||B_SETTINGS_FILE_UPDATE_TYPE_KEEP_OLD||the old settings file shall be kept||
     626   ||1||B_SETTINGS_FILE_UPDATE_TYPE_MANUAL||the old settings file needs to be updated manually||
     627   ||2||B_SETTINGS_FILE_UPDATE_TYPE_AUTO_MERGE||an automatic three-way merge shall be attempted||
     628 - '''Child Attributes:''' none
     629
     630==== B_HPKG_ATTRIBUTE_ID_PACKAGE_SETTINGS_FILE_TEMPLATE ("package:settings-file-template") ====
     631 - '''Type:''' string
     632 - '''Value:''' Relative path of an included template file for the user settings file.
     633 - '''Allowed Values:''' Installation location relative path of a file included in the package.
     634 - '''Child Attributes:''' none
     635
     636==== B_HPKG_ATTRIBUTE_ID_PACKAGE_USER ("package:user") ====
     637 - '''Type:''' string
     638 - '''Value:''' Name of a user required by the package. Upon package activation the user will be created, if necessary.
     639 - '''Allowed Values:''' Any valid user name, i.e. must be non-empty composed of <alphanum_underline>.
     640 - '''Child Attributes:'''
     641   - B_HPKG_ATTRIBUTE_ID_PACKAGE_USER_REAL_NAME: The user's real name.
     642   - B_HPKG_ATTRIBUTE_ID_PACKAGE_USER_HOME: The user's home directory.
     643   - B_HPKG_ATTRIBUTE_ID_PACKAGE_USER_SHELL: The user's shell.
     644   - B_HPKG_ATTRIBUTE_ID_PACKAGE_USER_GROUP: The user's group(s).
     645
     646==== B_HPKG_ATTRIBUTE_ID_PACKAGE_USER_REAL_NAME ("package:user.real-name") ====
     647 - '''Type:''' string
     648 - '''Value:''' The real name of the user.
     649 - '''Allowed Values:''' Any string.
     650 - '''Default Value:''' The user name.
     651 - '''Child Attributes:''' none
     652
     653==== B_HPKG_ATTRIBUTE_ID_PACKAGE_USER_HOME ("package:user.home") ====
     654 - '''Type:''' string
     655 - '''Value:''' The path to the home directory of the user.
     656 - '''Allowed Values:''' Any valid path.
     657 - '''Child Attributes:''' none
     658
     659==== B_HPKG_ATTRIBUTE_ID_PACKAGE_USER_SHELL ("package:user.shell") ====
     660 - '''Type:''' string
     661 - '''Value:''' The path to the shell to be used for the user.
     662 - '''Allowed Values:''' Any valid path.
     663 - '''Default Value:''' "/bin/bash".
     664 - '''Child Attributes:''' none
     665
     666==== B_HPKG_ATTRIBUTE_ID_PACKAGE_USER_GROUP ("package:user.group") ====
     667 - '''Type:''' string
     668 - '''Value:''' A group the user belongs to. At least one must be specified.
     669 - '''Allowed Values:''' Any valid group name, i.e. must be non-empty composed of <alphanum_underline>.
     670 - '''Default Value:''' The default group for users.
     671 - '''Child Attributes:''' none
     672
     673==== B_HPKG_ATTRIBUTE_ID_PACKAGE_GROUP ("package:group") ====
     674 - '''Type:''' string
     675 - '''Value:''' Name of a group required by the package. Upon package activation the group will be created, if necessary.
     676 - '''Allowed Values:''' Any valid group name, i.e. must be non-empty composed of <alphanum_underline>.
     677 - '''Child Attributes:''' none
     678
     679==== B_HPKG_ATTRIBUTE_ID_PACKAGE_POST_INSTALL_SCRIPT ("package:post-install-script") ====
     680 - '''Type:''' string
     681 - '''Value:''' Relative path of a script that shall be executed after package activation.
     682 - '''Allowed Values:''' Installation location relative path of a file included in the package.
     683 - '''Child Attributes:''' none
     684
     685
    631686== Haiku Package Repository Format ==
    632687
     
    643698 Header::
    644699   Identifies the file as HPKR file and provides access to the other sections.
     700 Heap::
     701   Contains the next two sections.
    645702 Repository Info::
    646703   A section containing an archived BMessage of a BRepositoryInfo object.
     
    648705   A section just like the package attributes section of the HPKG, only that this section contains the package attributes of all the packages contained in the repository (not just one).
    649706
     707The Repository Info and Package Attributes sections aren't really separate sections, as they are stored at the end of the heap.
    650708
    651709==== Header ====
     
    659717        uint16  version;
    660718        uint64  total_size;
     719        uint16  minor_version;
     720
     721        // heap
     722        uint16  heap_compression;
     723        uint32  heap_chunk_size;
     724        uint64  heap_size_compressed;
     725        uint64  heap_size_uncompressed;
    661726
    662727        // repository info section
    663         uint32  info_compression;
    664         uint32  info_length_compressed;
    665         uint32  info_length_uncompressed;
     728        uint32  info_length;
     729        uint32  reserved1;
    666730
    667731        // package attributes section
    668         uint32  packages_compression;
    669         uint64  packages_length_compressed;
    670         uint64  packages_length_uncompressed;
     732        uint64  packages_length;
    671733        uint64  packages_strings_length;
    672734        uint64  packages_strings_count;
     
    677739   The string 'hpkr' (B_HPKG_REPO_MAGIC).
    678740 header_size::
    679    The size of the header.
     741   The size of the header. This is also the absolute offset of the heap.
    680742 version::
    681743   The version of the HPKR format the file conforms to. The current version is
    682    1 (B_HPKG_REPO_VERSION).
     744   2 (B_HPKG_REPO_VERSION).
    683745 total_size::
    684746   The total file size.
    685 
    686  info_compression::
    687    The compression algorithm used for the repository info section.
    688  info_length_compressed::
    689    The compressed size of the repository info section. Equals
    690    info_length_uncompressed, if the section is not compressed.
    691  info_length_uncompressed::
     747 minor_version::
     748   The minor version of the HPKR format the file conforms to. The current minor version is
     749   0 (B_HPKG_REPO_MINOR_VERSION). Additions of new attributes to the attributes section should
     750   generally only increment the minor version. When a file with a greater minor version is
     751   encountered, the reader should ignore unknown attributes.
     752
     753 heap_compression::
     754   Compression format used for the heap.
     755 heap_chunk_size::
     756   The size of the chunks the uncompressed heap data are divided into.
     757 heap_size_compressed::
     758   The compressed size of the heap. This includes all administrative data (the chunk size array).
     759 heap_size_uncompressed::
     760   The uncompressed size of the heap. This is only the size of the raw data (including the repository info
     761   and attributes section), not including administrative data (the chunk size array).
     762
     763 info_length::
    692764   The uncompressed size of the repository info section.
    693765
    694  packages_compression::
    695    The compression algorithm used for the package attributes section.
    696  packages_length_compressed::
    697    The compressed size of the package attributes section. Equals
    698    attributes_length_uncompressed, if the section is not compressed.
    699  packages_length_uncompressed::
     766 reserved1::
     767   Reserved for later use.
     768
     769 packages_length::
    700770   The uncompressed size of the package attributes section.
    701771 packages_strings_length::