From d7c70837b94e46feaa07c0ed98f300ac0a3cf72b Mon Sep 17 00:00:00 2001 From: Dylan Araps Date: Mon, 5 Oct 2020 08:23:47 +0300 Subject: [PATCH] docs: Added package-manager.txt --- docs/package-manager.txt | 739 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 739 insertions(+) create mode 100644 docs/package-manager.txt diff --git a/docs/package-manager.txt b/docs/package-manager.txt new file mode 100644 index 0000000..34df073 --- /dev/null +++ b/docs/package-manager.txt @@ -0,0 +1,739 @@ +Package Manager +________________________________________________________________________________ + +The KISS package manager is a self-contained POSIX shell script which is written +in a highly portable way. The entire utility comes in at under 1000 lines of +code (excluding blank lines/comments of which there are many). + +The package manager is merely an implementation of the package format, its +requirements and some added sugar on top. + +Source: $/kisslinux/kiss + + +[0.0] Index +________________________________________________________________________________ + +- Usage [1.0] +- Dependencies [2.0] + +- Interesting Features [3.0] + - Runtime dependency detector built around 'ldd' [3.1] + - Fully dynamic (and automatic) alternatives system [3.2] + - 3-way handshake for files in /etc/ [3.3] + +- Configuration [4.0] + +- Repositories [5.0] + - What is a repository? [5.1] + - Enabling a remote repository [5.2] + - Preventing a package from receiving updates [5.3] + - Package fallbacks [5.4] + - Bypassing KISS_PATH [5.5] + +- Package Manager Hooks [6.0] + - Usage [6.1] + - Removing unneeded files from packages [6.2] + - Building some packages in memory (tmpfs) [6.3] + - Logging build duration [6.4] + +- Package Manager Extensions [7.0] + +- Tips and Tricks [8.0] + - Swap grep implementations for a major speed up [8.1] + + +[1.0] Usage +________________________________________________________________________________ + ++------------------------------------------------------------------------------+ +| kiss | ++------------------------------------------------------------------------------+ +| | +| -> kiss [a|b|c|d|i|l|r|s|u|v] [pkg]... | +| -> alternatives List and swap to alternatives | +| -> build Build a package | +| -> checksum Generate checksums | +| -> download Pre-download all sources | +| -> install Install a package | +| -> list List installed packages | +| -> remove Remove a package | +| -> search Search for a package | +| -> update Update the system | +| -> version Package manager version | +| | +| Run "kiss help-ext" to see all actions | +| | ++------------------------------------------------------------------------------+ + + +[2.0] Dependencies +________________________________________________________________________________ + +POSIX utilities are used where appropriate _and_ where they exist to solve a +particular problem. Utilities of which there is only a single and cross-platform +implementation are considered "portable" (git, curl, etc) + +If a dependency can be made optional, it will be made so. Dependencies are also +kept to a minimum (though we must also remain realistic). + ++------------------+------------------------------------------------+----------+ +| Dependency | Reason for use | Required | ++------------------+------------------------------------------------+----------| +| POSIX utilities | Used throughout | Yes | +| git | Remote repositories and | Yes [1] | +| | package git sources | | +| curl | Source downloads | Yes | +| gpg1 or gpg2 | Repository signing | No | +| sha256sum | Checksums | Yes [2] | +| tar | Sources, packages, etc | Yes [3] | +| unzip | Zip sources (very rare) | No | +| | | | +| | | | +|------------------+------------------------------------------------+----------| +| Compression | | | +|------------------+------------------------------------------------+----------| +| gzip, bzip2, xz | Tarball compression | Yes [4] | +| zstd, lzma, lzip | Tarball compression | No | +| | | | +| | | | +|------------------+------------------------------------------------+----------| +| Privileges | | | +|------------------+------------------------------------------------+----------| +| su, sudo, doas, | Privilege escalation | No [5] | +| sls | Privilege escalation | No [5] | +| | | | +|------------------+------------------------------------------------+----------| +| C Library | | | +|------------------+------------------------------------------------+----------| +| ldd | Dependency Fixer | No [6] | +| | | | +|------------------+------------------------------------------------+----------| +| Binutils | | | +|------------------+------------------------------------------------+----------| +| strip | Binary Stripping | No [6] | +| readelf | Dependency Fixer | No [6] | +| | | | ++------------------+------------------------------------------------+----------+ + +[1] Git is also required for contribution to the distribution itself. Strictly + speaking, nothing forces you to use git. Remote repositories and git based + sources will simply become unusable. + +[2] There is no standard utility for the generation of sha256 checksums. While + sha256sum is listed above, the package manager also supports sha256, shasum + and openssl as fallbacks. + +[3] The tar command has no standard! This came as a shock. The POSIX equivalent + is "pax" though this isn't in wide use (at least on Linux). + + Our usage of tar is merely, cf, xf and tf. A patch is applied to sbase's + tar so that it supports "dashless" arguments (as all others do). Our usage + of tar cannot become any more basic than it is now. Portability should no + longer be a concern. + + Tested tar implementations include: busybox, toybox, sbase, GNU and + libarchive (though all tar implementations should work in theory). + +[4] These three compression methods are required as pretty much every package + source uses them as the tarball compression method. + + The other compression methods are optional as no package sources (in the + official repositories) make use of them. + + If a compression method has 1-3 uses (hasn't yet happened), the compression + method will simply become a 'make' dependency of the package until usage + increases to a "normality". + +[5] A privilege escalation utility is only needed when using the package + manager as a normal user for system-wide package installation. + + Installation to a user-writable directory does not require root access. + + Root usage of the package manager (chroot usage for example) does not + require these utilities. + +[6] If these are missing, binary stripping and/or the dependency fixer will + simply be disabled. If readelf is not available, ldd will be used in its + place for the dependency fixer. + + Regarding 'strip'; It has a POSIX specification, though the spec doesn't + contain any arguments whatsoever. + + This makes our usage of 'strip' non-POSIX. That being said, our usage is + compatible with these 'strip' implementations. + + strips: binutils, elfutils, elftoolchain, llvm, etc. + + +[3.0] Interesting Features +________________________________________________________________________________ + + + [3.1] Runtime dependency detector built around 'readelf' or 'ldd' + ____________________________________________________________________________ + + Dynamic dependencies brought in by build systems (which are missing from the + package's dependency list) are fixed on-the-fly by checking which libraries + link to the package's files. + + This prevents an incomplete dependency list from causing system breakage as + the package manager is able to complete the list. + + A lot of packages make use of this "implicit" to "explicit" dependency list + "conversion" to provide optional dependencies. + + Example output: + + +--------------------------------------------------------------------------+ + | | + | -> libXmu Checking for missing dependencies | + | --- /home/dylan/conf/cache/kiss/build-4477/d | + | depends | + | @@ -1,3 +1,8 @@ | + | +libX11 | + | +libXau | + | libXext | + | libXt | + | +libxcb | + | xorg-util-macros make | + | | + +--------------------------------------------------------------------------+ + + + [3.2] Fully dynamic (and automatic) alternatives system + ____________________________________________________________________________ + + Any file conflicts between two packages automatically become choices in the + alternatives system. + + This allows one to swap providers of files without needing to explicitly + tell the package manager that two packages conflict, provide the same + utilities, etc. + + In other words, no changes need to be made to packages. In fact, nothing + needs to be done at all. It's entirely automatic. + + +--------------------------------------------------------------------------+ + | List available alternatives ('a' is an alias to 'alternatives'). | + +--------------------------------------------------------------------------+ + | | + | $ kiss a | + | gnugrep /usr/bin/grep | + | ncurses /usr/bin/clear | + | | + +--------------------------------------------------------------------------+ + | Swap to GNU grep. | + +--------------------------------------------------------------------------+ + | | + | $ kiss a gnugrep /usr/bin/grep | + | -> Swapping '/usr/bin/grep' from busybox to gnugrep | + | | + +--------------------------------------------------------------------------+ + | Swap back to busybox grep. | + +--------------------------------------------------------------------------+ + | | + | $ kiss a busybox /usr/bin/grep | + | -> Swapping '/usr/bin/grep' from gnugrep to busybox | + | | + +--------------------------------------------------------------------------+ + | Swap to all alternatives for a given package (sbase for example). | + +--------------------------------------------------------------------------+ + | | + | $ kiss a | grep ^sbase | kiss a - | + | -> Swapping '/usr/bin/cat' from busybox to sbase | + | -> Swapping '/usr/bin/cut' from busybox to sbase | + | -> Swapping '/usr/bin/yes' from busybox to sbase | + | ...Many more lines of output... | + | | + +--------------------------------------------------------------------------+ + + The above command works as the output of the alternatives listing is + directly usable as input to 'kiss a'. + + + [3.3] 3-way handshake for files in /etc/ + ____________________________________________________________________________ + + Files in /etc/ are handled differently to those elsewhere on the system. A + reinstallation or update to a package will not always overwrite these files. + + Instead, a 3-way handshake happens during installation to determine how the + new /etc/ file should be handled. + + If the user has made modifications to the file and those modifications + differ to the to-be-installed package's file, the file is installed with the + suffix '.new' + + If the user hasn't touched the file, it will be automatically overwritten by + the package manager as it will contain updated/new contents.. + + If the user has touched the file but the file has not changed between + package versions, it will simply be skipped over. + + +[4.0] Configuration +________________________________________________________________________________ + +The package manager has no configuration files and no changes need to be made to +the system prior to its use. While there is no configuration file, this does not +mean that there is no possibility for configuration. + +The package manager can be configured via the use of environment variables. I +believe this to be the best configuration method (where realistic). Environment +variables can be set system-wide, per-user, conditionally, for a single +invocation, etc, etc. + +They require little to no extra code in the package manager to support them. + ++-----------------+------------------------------------------------------------+ +| Variable | Description | ++-----------------+------------------------------------------------------------+ +| | | +| KISS_PATH | List of repositories. This works exactly like '$PATH' | +| | (a colon separated list of paths). | +| | | +| KISS_FORCE | Force installation/removal of package by bypassing | +| | dependency checks, etc. Set to '1' to enable. | +| | | +| KISS_CHOICE | Set to '0' to disable the alternatives system and error on | +| | any detected file conflicts. | +| | | +| KISS_HOOK | Hook into the package manager. Set to the full path to the | +| | script. See @/wiki/kiss/package-manager-hooks | +| | | +| KISS_ROOT | Where installed packages will go. Can be used to install | +| | packages to somewhere other than '/'. | +| | | +| KISS_COLOR | Enable/Disable colors. Set to '0' to disable colors. | +| | | +| KISS_PROMPT | Skip all prompts. Set to '0' to say 'yes' to all prompts | +| | from the package manager. | +| | | +| KISS_COMPRESS | Compression method to use for built package tarballs | +| | (defaults to 'gz'). Valid: bz2, gz, lzma, lz, xz, zst | +| | | +| KISS_SU | Force usage of a different sudo tool. | +| | Valid: su, sudo, doas | +| | | +| KISS_STRIP | Enable/Disable package stripping globally. | +| | Set to '0' to disable. | +| | | +| KISS_DEBUG | Keep temporary directories around for debugging purposes. | +| | Set to '1' to enable. | +| | | +| KISS_KEEPLOG | Keep build logs around for successful builds and not just | +| | failing ones. Set to '1' to enable. | +| | | +| KISS_TMPDIR | Temporary directory for builds. Can be set to a tmpfs so | +| | builds happen in memory. | +| | | +| KISS_PID | Use a reproducible cache naming scheme instead of | +| | build-$pid. If set to test, the result will be build-test. | +| | | ++-----------------+------------------------------------------------------------+ + +There are also a myriad of "3rd-party" environment variables which control GCC, +Make, CMake, etc. These aren't used by the package manager. They're used by the +tools called by the package's build script. + ++-----------------+------------------------------------------------------------+ +| Variable | Description | ++-----------------+------------------------------------------------------------+ +| XDG_CACHE_HOME | Cache directory location. | +| | | +| CC | C compiler. | +| CXX | C++ compiler. | +| AR | Archive tool. | +| NM | Symbol tool. | +| RANLIB | Index tool. | +| | | +| CFLAGS | C compiler flags. | +| CXXFLAGS | C++ compiler flags. | +| LDFLAGS | Linker flags. | +| MAKEFLAGS | Make flags. | +| SAMUFLAGS | Samurai flags. | +| RUSTFLAGS | Rust compiler flags. | +| | | +| CMAKE_GENERATOR | 'Unix Makefiles' or 'Ninja'. | +| | | ++-----------------+------------------------------------------------------------+ + + +[5.0] Repositories +________________________________________________________________________________ + +Repository management in KISS is very simple. Repositories are configurable via +an environment variable. This environment variable can be set system-wide, +per-user, conditionally (via a script or program), for a single invocation, etc. + +The environment variable is called '$KISS_PATH' and is functionally identical to +the '$PATH' variable. A colon separated list of paths in other words. + ++------------------------------------------------------------------------------+ +| Example KISS_PATH | ++------------------------------------------------------------------------------+ +| | +| $ KISS_PATH=/var/db/kiss/repo/core:/var/db/kiss/repo/extra | +| | ++------------------------------------------------------------------------------+ + +In the above example, two repositories are enabled (Core and Extra). The package +manager will search this list for packages in the order it is written. + +Repositories can live anywhere on the system. In your '$HOME' directory, +somewhere system-wide in '/', etc. The only requirement is that a full path be +used. + + + [5.1] What is a repository? + ____________________________________________________________________________ + + A KISS repository is simply a directory of directories. The quickest way to + get started is as follows. + + +--------------------------------------------------------------------------+ + | 1. Create the repository | + +--------------------------------------------------------------------------+ + | | + | $ mkdir -p repo + | $ cd repo + | | + +--------------------------------------------------------------------------+ + | 2. Let's 'fork' a few packages into our new repository. | + +--------------------------------------------------------------------------+ + | | + | $ kiss fork curl | + | $ kiss fork xz | + | $ kiss fork zlib | + | | + +--------------------------------------------------------------------------+ + + This is now a fully usable repository and it can be added to your KISS_PATH. + + + [5.2] Enabling a remote repository + ____________________________________________________________________________ + + Let's assume that our KISS_PATH matches the above example (Core and Extra). + As an example, we'll be enabling the Community repository. + + +--------------------------------------------------------------------------+ + | 1. Clone the repository | + +--------------------------------------------------------------------------+ + | | + | # This can live anywhere on the system. | + | $ git clone https://github.com/kisslinux/community | + | | + +--------------------------------------------------------------------------+ + | 2. Enable the repository | + +--------------------------------------------------------------------------+ + | | + | $ export KISS_PATH=$KISS_PATH:/path/to/community/community | + | | + +--------------------------------------------------------------------------+ + + + [5.3] Preventing a package from receiving updates + ____________________________________________________________________________ + + Preventing a package from receiving updates can be accomplished in a myriad + of different ways. The easiest method is to leverage a user repository. + + +--------------------------------------------------------------------------+ + | 1. Create a new repository | + +--------------------------------------------------------------------------+ + | | + | $ mkdir -p no_updates | + | $ cd no_updates | + | | + +--------------------------------------------------------------------------+ + | 2. Copy the package to the new repository | + +--------------------------------------------------------------------------+ + | | + | $ cp -r /var/db/kiss/installed/PKG_NAME /path/to/no_updates | + | | + +--------------------------------------------------------------------------+ + | 3. Add the new repository to the /START/ of your KISS_PATH | + +--------------------------------------------------------------------------+ + | | + | $ export KISS_PATH=/path/to/no_updates:$KISS_PATH | + | | + +--------------------------------------------------------------------------+ + + The package manager will search KISS_PATH in order. It will see that the + no_updates repository provides PKG_NAME and the version matches that which + is installed. No updates will again happen for the package. + + + [5.4] Package fallbacks + ____________________________________________________________________________ + + If you would like to package something in your own repository but would + like the package manager to prefer other repositories before yours, simply + add your repository to the end of KISS_PATH. + + The moment that your package is available elsewhere, the package manager + will prefer the new location to yours. The list is searched (in order) and + the first match is picked. + + + [5.5] Bypassing KISS_PATH + ____________________________________________________________________________ + + There is a special case where one can bypass the regular KISS_PATH for a + single invocation of the package manager. This has been called "CRUX-like + usage" by users. + + +--------------------------------------------------------------------------+ + | | + | $ cd /path/to/myrepo/firefox | + | $ kiss b | + | $ kiss i | + | | + +--------------------------------------------------------------------------+ + + As seen above, various package manager commands will work without arguments, + so long as you are in a package's repository directory. This will prepend + the current directory to '$KISS_PATH' _only_ for this invocation. + + +[6.0] Package Manager Hooks +________________________________________________________________________________ + +KISS' package manager is extensible via hooks which fire at various different +places inside the utility. Hooks allow the user to modify behavior, add new +features or conditionally do things on a per-package basis. + +This setting is controlled by the '$KISS_HOOK' environment variable which takes +the full path to a POSIX shell script as its value. + +Example: + ++------------------------------------------------------------------------------+ +| | +| export KISS_HOOK=$HOME/.local/bin/kiss-hook | +| | ++------------------------------------------------------------------------------+ + + + [6.1] Usage + ____________________________________________________________________________ + + The script is sourced at each hook and is given three variables as input. + POSIX shell does not allow sourced scripts to receive arguments, these + variables are instead defined via the script's environment. + + - '$PKG': The name of the current package. + - '$TYPE': The type of hook. + - '$DEST': The location where 'make install' will put files. + + Valid values for '$TYPE' include: + + pre-build, post-build, build-fail, + pre-install, post-install, pre-extract, + post-package. + + As the hook script is sourced (instead of being executed in its own shell), + the script has the ability to override package manager internals and the + package manager's environment. + + + [6.2] Removing unneeded files from packages + ____________________________________________________________________________ + + Packages can contain files which you will have no use for. A simple hook can + be defined to remove them from packages. + + NOTE: This is the default 'KISS_HOOK' script. If defining your own, be sure + to include this if you would like to continue to remove these files. + + +--------------------------------------------------------------------------+ + | | + | case $TYPE in | + | post-build) | + | # Ensure that '$DEST' is set. | + | : "${DEST:?DEST is unset}" | + | | + | rm -rf "$DEST/usr/share/gettext" \ | + | "$DEST/usr/share/polkit-1" \ | + | "$DEST/usr/share/locale" \ | + | "$DEST/usr/share/info" \ | + | "$DEST/usr/lib/charset.alias" | + | ;; | + | esac | + | | + +--------------------------------------------------------------------------+ + + + [6.3] Building some packages in memory (tmpfs) + ____________________________________________________________________________ + + This hook runs on pre-extract and conditionally runs the source extraction, + package build process and resulting tarball creation in memory. The benefit + of this is a nice speedup throughout (especially when used alongside + ccache [0]). + + NOTE: '/tmp' must be mounted as 'tmpfs' for this to work. + + +--------------------------------------------------------------------------+ + | | + | case $TYPE in | + | pre-extract) | + | case $PKG in | + | # Run everything on disk for these memory hungry packages. | + | firefox|firefox-esr|rust|llvm|clang) | + | mak_dir=$KISS_TMPDIR/$pid/build | + | pkg_dir=$KISS_TMPDIR/$pid/pkg | + | ;; | + | | + | *) | + | log "$PKG" "Activating tmpfs" | + | | + | mak_dir=/tmp/$pid/build | + | pkg_dir=/tmp/$pid/pkg | + | ;; | + | esac | + | | + | mkdir -p "$mak_dir" "$pkg_dir/$PKG/var/db/kiss/installed" | + | ;; | + | | + | post-build) | + | rm -rf "$mak_dir" | + | ;; | + | | + | post-package) | + | rm -rf "$pkg_dir" | + | ;; | + | esac | + | | + +--------------------------------------------------------------------------+ + + + [6.4] Logging build duration + ____________________________________________________________________________ + + This hook adds a new message post-build with the total build duration in a + human readable format (00h 00m). Similar code is used in the boot process + of the system to calculate boot time. + + +--------------------------------------------------------------------------+ + | | + | case $TYPE in | + | pre-build) | + | IFS=. read -r _start _ < /proc/uptime | + | ;; | + | | + | post-build) | + | IFS=. read -r _end _ < /proc/uptime | + | | + | ( | + | _s=$((_end - _start)) | + | _h=$((_s / 60 / 60 % 24)) | + | _m=$((_s / 60 % 60)) | + | | + | [ "$_h" = 0 ] || _u="${_u}${_h}h " | + | [ "$_m" = 0 ] || _u="${_u}${_m}m " | + | | + | log "$PKG" "Build finished in ${_u:-${_s}s}" | + | ) | + | ;; | + | esac | + | | + +--------------------------------------------------------------------------+ + + +[7.0] Package Manager Extensions +________________________________________________________________________________ + +Anything in the user's '$PATH' which matches the glob 'kiss-*' will be directly +usable via the package manager. For example, 'kiss-size' is also usable as +'kiss size' (and even 'kiss si') (the shortest available alias). + +The detected 'kiss-*' utilities will appear in the package manager's help output +with the second line in the script acting as a doc-string. + +Example help output: + ++--------------------------------------------------------------------------+ +| kiss extensions | ++--------------------------------------------------------------------------+ +| | +| -> kiss help-ext | +| -> extensions (kiss-* in PATH) | +| chroot Enter a kiss chroot | +| depends Display a package's dependencies | +| export Turn an installed package into a KISS tarball | +| fork Copy a package's repository files into PWD. | +| help Read KISS documentation | +| link Link a repository file to another repository | +| maintainer Find the maintainer of a package | +| manifest Display all files owned by a package | +| new Create a boilerplate package | +| old shellcheck source=/dev/null | +| orphans List orphaned packages | +| outdated Check repository for outdated packages | +| owns Check which package owns a file | +| reset Remove all packages except for the base | +| revdepends Display packages which depend on package | +| size Show the size on disk for a package | +| | ++--------------------------------------------------------------------------+ + +These are in effect, optional utilities which interact with the package system +in one way or another. My hope behind them is to act as an example as to how +easy it is to interface with the plain-text and "static" package system. + +Example utility: + ++------------------------------------------------------------------------------+ +| kiss-depends (kiss depends, kiss de) | ++------------------------------------------------------------------------------+ +| | +| #!/bin/sh -ef | +| # Display a package's dependencies | +| | +| # Ignore shellcheck as we want the warning's behavior. | +| # shellcheck disable=2015 | +| [ "$1" ] && kiss l "${1:-null}" >/dev/null || { | +| printf 'usage: kiss-depends [pkg]\n' | +| exit 1 | +| } | +| | +| cat "$KISS_ROOT/var/db/kiss/installed/$1/depends" 2>/dev/null | +| | ++------------------------------------------------------------------------------+ + + +[8.0] Tips and Tricks +________________________________________________________________________________ + +A lot of the package manager's features are hard to discover or otherwise +non-obvious to its users. This section will document these features, how to use +them and the benefits they bring. + + + [8.1] Swap grep implementations for a major speed up. + ____________________________________________________________________________ + + The default grep implementation in KISS is busybox grep. This version of + grep works very well and supports a large number of features. The one issue + is that it is painfully slow when compared to other popular implementations. + + A fairly major speedup can be attained by swapping to a different grep via + the alternatives system. The fastest grep implementation around is GNU grep + which is available in Community as 'gnugrep'. + + +--------------------------------------------------------------------------+ + | 1. Install GNU grep | + +--------------------------------------------------------------------------+ + | | + | $ kiss b gnugrep && kiss i gnugrep | + | | + +--------------------------------------------------------------------------+ + | 2. Swap to GNU grep | + +--------------------------------------------------------------------------+ + | | + | $ kiss a gnugrep /usr/bin/grep | + | | + +--------------------------------------------------------------------------+