forklift

README

---

Composable, reprovisionable, decentralized management of apps & configs on Raspberry Pis and other embedded Linux systems

Note: this is still an experimental prototype and is not yet ready for general use.

Introduction

Forklift is a software deployment and configuration system providing a simpler, easier, and safer mechanism for provisioning, updating, reconfiguring, recomposing, and extending browser apps, network services, and system files on single-computer systems (such as a Raspberry Pi or a laptop). The design of Forklift makes tradeoffs specific to the ways in which many open-source scientific instruments need to be deployed and operated (e.g. intermittent internet access, independent administration by individual people, decentralized management & customization). For a quick three-minute overview of the motivation for Forklift and how it can be used, refer to this demo video: https://www.youtube.com/watch?v=4lHh_NDlFKA

For end-users operating open-source instruments with application services (e.g. network APIs or browser-based interfaces) and/or system services (for e.g. data backups/transfer, hardware support, computer networking, monitoring, etc.), Forklift aims to provide an experience for installing, updating, uninstalling software and OS customizations similar what is achieved by app stores for mobile phones - but with more user control. Forklift also simplifies the process of keeping software up-to-date and the process of rolling software back to older versions if needed; this reduces the need to (for example) re-flash a Raspberry Pi's SD card with a new OS image just to update the application software running on the instrument while ensuring the validity of the resulting state of the system.

For open-hardware project developers, Forklift enables Linux-based devices and appliances (especially those based on the Raspberry Pi) to be retrofitted and extended with an open ecosystem of containerized software - device-specific or general-purpose, project-maintained or third-party. Forklift also provides an incremental path for migrating project-specific application/system services and OS configs into management by Forklift so that they can be configured, distributed, installed, and replaced by users just like any other app managed by Forklift - i.e. with version control and easy upgrades/rollbacks. The PlanktoScope, an open-source microscope for quantitative imaging of plankton, uses Forklift as foundational infrastructure for software releases, deployment, and extensibility in the PlanktoScope OS, a hardware-specific operating system based on the Raspberry Pi OS; and Forklift was designed specifically to solve the OS/software maintenance and operations challenges experienced in the PlanktoScope project.

For indie software developers and sysadmins familiar with DevOps and cloud-native patterns, Forklift is just a GitOps-inspired system which is small and simple enough to work beyond the cloud - using Docker Compose to avoid the unnecessary architectural complexity and overhead even minimal Kubernetes distributions like k3s for situations where a container workload orchestrator is unnecessary; and bundling app deployment with the deployment of system files, executables, and systemd units from configuration files version-controlled in Git repositories. Thus, Forklift allows hassle-free management of software configurations on one or more machines with only occasional internet access (or, in the future, no internet access at all!) and no specialized ops or platform team.

For people who are Very Into Linux, Forklift is a way to bring some of the architectural benefits of atomic/immutable OSes (such as ChromeOS, Fedora Atomic Desktops, Universal Blue, Fedora CoreOS/IoT, NixOS, Flatcar Container Linux, etc.) to more traditional non-atomic Linux distros (such as Raspberry Pi OS, Debian, etc.), by enabling atomic, composable, and reprovisionable changes for a subset of the OS. The specific scope of what Forklift manages depends on how Forklift gets integrated with the OS, but Forklift is intended to enable sysadmins to incrementally reduce reliance on system packages, and to incrementally reduce any responsibilities of the base OS beyond acting as a container host with hardware drivers (e.g. for Wi-Fi).

To learn more about the design of Forklift, please refer to Forklift's design document.

Project Governance

Currently, design and development of Forklift prioritizes the needs of the PlanktoScope community and the PlanktoScope project's values for its infrastructural software. Thus, for now decisions will be made by the PlanktoScope software's lead maintainer (currently @ethanjli) as a "benevolent dictator"/"mad scientist" in consultation with the PlanktoScope community in online meetings and discussion channels open to the entire community. This will remain the governance model of Forklift while it's still an experimental tool and still only used for the standard/default version of the PlanktoScope's operating system, in order to ensure that Forklift develops in a cohesive way consistent with the values mentioned above and with intended use of Forklift for the PlanktoScope community. Once Forklift starts being used for delivering/maintaining variants of the PlanktoScope's operating system, for integration of third-party apps from the PlanktoScope community, or for software configuration by ordinary users, then governance of the github.com/PlanktoScope/forklift repository will transition from benevolent dictatorship into the PlanktoScope project's consensus-based proposals process. In the meantime, we encourage anyone who is interested in using/adapting Forklift to fork this repository for experimentation and/or to create new discussion posts in this repository, though we can't make any guarantees about the stability of any APIs or about our capacity to address any external code contributions or feature requests.

If other projects beyond the PlanktoScope community decide to use Forklift as part of their software delivery/deployment infrastructure, we can talk about expanding governance of Forklift beyond the PlanktoScope community - feel free to start a discussion in this repository's GitHub Discussions forum.

Usage

For a more guided demo showing off Forklift's capabilities, you can try out Forklift on a Raspberry Pi or in a VM:

• Raspberry Pi: github.com/ethanjli/rpi-forklift-demo provides an SD card image of the Raspberry Pi OS with Forklift already integrated into it, so that Forklift can be used to provision/update/deprovision Docker Compose apps as well as system config files, scripts, and binaries in the /etc and /usr directories.
• VM: github.com/ethanjli/ublue-forklift-sysext-demo provides an ISO installer of Bluefin (i.e. a custom image of Fedora Silverblue) with Forklift already integrated into it, so that Forklift can be used to provision/update/deprovision Docker Compose apps as well as systemd system/configuration extensions.

If you don't want to try out Forklift in either of the above ways, the below instructions will help you to set up Forklift on your own system to Docker Compose apps on your system:

Download/install forklift

First, you will need to download the forklift tool, which is available as a single self-contained executable file. You should visit this repository's releases page and download an archive file for your platform and CPU architecture; for example, on a Raspberry Pi 4, you should download the archive named forklift_{version number}_linux_arm.tar.gz (where the version number should be substituted). You can extract the forklift binary from the archive using a command like:

tar -xzf forklift_{version number}_{os}_{cpu architecture}.tar.gz forklift

Then you may need to move the forklift binary into a directory in your system path, or you can just run the forklift binary in your current directory (in which case you should replace forklift with ./forklift in the example commands listed below), or you can just run the forklift binary by its absolute/relative path (in which case you should replace forklift with the absolute/relative path of the binary in the example commands listed below).

Deploy a published pallet

To deploy a particular version of a published pallet to your computer, you will need to clone a pallet and stage it to be applied, and then you will need to apply the staged pallet. Pallets are identified by the path of their Git repository and a version query (which can be a Git branch name, a Git tag name, or an abbreviated or full Git commit hash). For example, the most recent commit on the main branch of the github.com/forklift-run/pallet-example-minimal can be identified as github.com/forklift-run/pallet-example-minimal@main - this is what we use in the example commands in this section.

If you are running Docker in rootless mode or your user is in the docker group):

• If you want to apply the pallet immediately, you can run forklift pallet switch --apply with your specified pallet. For example:

  forklift pallet switch --apply github.com/forklift-run/pallet-example-minimal@main
  

• If you want to apply the pallet later, you can first stage the pallet, and then later apply the staged pallet using a separate forklift stage apply command. For example:

  # Run now:
  forklift pallet switch github.com/forklift-run/pallet-example-minimal@main
  # Run when you want to apply the pallet:
  forklift stage apply
  

If you aren't running Docker in rootless mode and your user isn't in a docker group, we recommend a slightly different set of commands:

• If you want to apply the pallet immediately, you can run forklift pallet switch --no-cache as a regular user, and then run forklift stage apply as root; the --no-cache flag prevents forklift pallet switch from attempting to make Docker pre-download all container images required by the pallet, as doing so would require root permissions for Forklift to talk to Docker. For example:

  forklift pallet switch --no-cache-img github.com/forklift-run/pallet-example-minimal@main
  sudo -E forklift stage apply
  

• If you want to apply the pallet later, you can first stage the pallet and pre-download all container images required by the pallet, and then later apply the staged pallet using a separate forklift stage apply command (which you can then run even when you don't have internet access). For example:

  # Run now:
  forklift pallet switch --no-cache-img github.com/forklift-run/pallet-example-minimal@main
  sudo -E forklift stage cache-img
  # Run when you want to apply the pallet:
  sudo -E forklift stage apply
  

Note: in the above commands, you can replace forklift pallet with forklift plt if you want to type fewer characters when running those commands.

Work on a development pallet

First, you will need to make/download a pallet somewhere on your local file system. For example, you can use git to clone the latest unstable version (on the main branch) of the github.com/forklift-run/pallet-example-minimal pallet using the command:

git clone https://github.com/forklift-run/pallet-example-minimal

Then you will need to download/install the forklift tool (see instructions in the "Download/install forklift" section above). Once you have forklift, you can run commands using the dev plt subcommand; if forklift is in your system path, you can simply run commands within the directory containing your development pallet, or any subdirectory of it. For example, if your development pallet is at /home/pi/dev/pallet-example-minimal, you can run the following commands to see some information about your development pallet:

cd /home/pi/dev/pallet-example-minimal
forklift dev plt show

You can also run the command from anywhere else on your filesystem by specifying the path of your development pallet. For example, if your forklift binary is in /home/pi, you can run any the following sets of commands to see the same information about your development pallet:

cd /home/pi/
./forklift dev --cwd ./dev/pallet-example-minimal plt show

cd /etc/
/home/pi/forklift dev --cwd /home/pi/dev/pallet-example-minimal plt show

You can also use the forklift dev plt require-repo command to require additional Forklift repositories for use in your development pallet, and/or to change the versions of Forklift repositories already required by your development pallet.

You can also run commands like forklift dev plt cache-all and forklift dev plt stage --no-cache-img (with appropriate values in the --cwd flag if necessary) to download the Forklift repositories specified by your development pallet into your local cache and stage your development pallet to be applied with sudo -E forklift stage apply. This is useful if, for example, you want to make some experimental changes to your development pallet and test them on your local machine before committing and pushing those changes onto GitHub.

Finally, you can run the forklift dev plt check command to check the pallet for any problems, such as violations of resource constraints between package deployments.

You can also override cached repos with repos from your filesystem by specifying one or more directories containing one or more repos; then the repos in those directories will be used instead of the respective repos from the cache, regardless of repo version. For example:

cd /home/pi/
/home/pi/forklift dev --cwd /home/pi/dev/pallet-example-minimal plt --repos /home/pi/forklift/dev/device-pkgs check

Similar projects

The following projects solve related problems with containers for application software, though they make different trade-offs compared to Forklift:

• poco enables Git-based management of Docker Compose projects and collections (catalogs) of projects and repositories and provides some similar functionalities to forklift.
• Terraform (an early inspiration for this project) has a Docker Provider which enables declarative management of Docker hosts and Docker Swarms from a Terraform configuration.
• swarm-pack (an early inspiration for this project) uses collections of packages from user-specified Git repositories and enables templated configuration of Docker Compose files, with imperative deployments of packages to a Docker Swarm.
• SwarmManagement uses a single YAML file for declarative configuration of an entire Docker Swarm.
• Podman Quadlets enable management of containers, volumes, and networks using declarative systemd units
• FetchIt enables Git-based management of containers in Podman.
• Projects developing GitOps tools such as ArgoCD, Flux, etc., store container environment configurations as Git repositories but are generally designed for Kubernetes.

The following projects solve related problems in the base OS, though they make different trade-offs compared to Forklift (especially because of the PlanktoScope project's legacy software):

• systemd-sysext and systemd-confext provide a more structured/opinionated way (compared to Forklift) to atomically overlay system files onto the base OS, but they don't specify a way to distribute/provision published sysext/confext images, and they are not available on Raspberry Pi OS 11 (bullseye). Forklift can be used together with systemd-sysext/confext, as a way to download sysext/confext images, constrain which images can be deployed together/separately, and manage which images are available to systemd - see this demo.
• systemd's Portable Services pattern and portablectl tool provide a more structured/constrained/sandboxed way (compared to Forklift) to atomically add system services, but they are only designed for self-contained system services. Forklift can probably used together with systemd Portable Services, as a way to deploy published images onto an OS.
• NixOS enables declarative specification of the entire configuration of the system with atomic updates, but it requires learning a bespoke functional programming language and is rather complex. Forklift is inspired by some of NixOS's design decisions, and it attempts to provide a few of NixOS's benefits but in a much more approachable, simple, and backwards-compatible system design by sacrificing some theoretical rigor for practical considerations.
• OSTree enables atomic updates of the base OS, but it is not supported by Raspberry Pi OS, and delivery of the OS requires operating a server to host OSTree repositories. Forklift can be used together with OSTree-based systems, as a way to provision/replace/deprovision a layer of /etc config files over (and independently of) OSTree's 3-way merge mechanism for /etc, and as a way to manage Docker Compose apps running on the system.
• The bootc project enables the entire operating system to be delivered as a bootable OCI container image, but currently it relies on bootupd, which currently only works on RPM-based distros. Forklift can be used together with bootc-based systems, as a way to manage Docker Compose apps (and/or systemd sysexts/confexts) on the system.
• gokrazy enables atomic deployment of Go programs (and also of software containers!), but it has a very different architecture compared to traditional Linux distros.

Other related OS-level projects can be found at github.com/castrojo/awesome-immutable.

Licensing

Except where otherwise indicated, source code provided here is covered by the following information:

Copyright Ethan Li and PlanktoScope project contributors

SPDX-License-Identifier: Apache-2.0 OR BlueOak-1.0.0

You can use the source code provided here either under the Apache 2.0 License or under the Blue Oak Model License 1.0.0; you get to decide. We are making the software available under the Apache license because it's OSI-approved, but we like the Blue Oak Model License more because it's easier to read and understand.