Immutable store and garbage collector¶
Everything Lattice builds is kept in the store, a single directory in which each entry is written once and never changed. Software is never upgraded in place. A new version is a new entry beside the old one, and the system switches from one to the other. Old entries stay until nothing needs them, and then the garbage collector removes them.
Store paths¶
Each entry is a directory named /lattice/store/<hash>-<name>, for example /lattice/store/3x9k…-hello-2.12.1. The hash is the derivation hash, so it covers every input to the build. A derivation with several outputs gets one entry per output, and every output other than out adds its name, as in …-hello-2.12.1-dev. Two different builds can never claim the same path, and if a path already exists the build it represents has already been done.
Publishing a build¶
---
config:
themeVariables:
fontSize: 16px
flowchart:
useMaxWidth: false
sequence:
useMaxWidth: false
---
flowchart TB
out["Output in the<br/>sandbox tmpfs"] -->|"copy"| stage["Staging directory<br/>same filesystem as store"]
stage -->|"atomic rename"| path["/lattice/store/hash-name"]
path -->|"remove write bits"| ro["Read-only path"]
ro -->|"one transaction"| db[("Register path<br/>and references")]
A rename is atomic only within one filesystem, and the sandbox output lives in memory. Lattice therefore first copies the output to a staging directory on the same filesystem as the store, then moves it into place with a single rename. Anyone looking at the store sees either no entry or a complete one, never a partial one. Lattice then removes write permission from every file and directory in the entry, so any later attempt to change it fails with a permission error.
Tracking dependencies¶
An SQLite database records every store path, every run-time dependency between paths, and the roots that must be kept, such as the current system and anything a user has pinned. Run-time dependencies are found by scanning a new entry's files for the hashes of the paths it was built from. A program linked against a library contains that library's store path, and so the dependency is found without being declared.
| Table | Holds |
|---|---|
store_paths |
One row per entry, with its derivation hash, a hash of its contents and its size |
references |
One row per dependency, from the entry that needs it to the entry it needs |
gc_roots |
The named entries that must never be collected |
An entry can only depend on entries that existed before it was built, or on another output of the same build that does not depend on it in turn, so these dependencies form a directed acyclic graph. The hash of an entry's contents covers file names, contents, symbolic link targets and the executable bit, and nothing that differs between machines, so lattice verify can later check that an entry is unchanged.
Garbage collection¶
The collector works in two phases. It first marks every entry reachable from any root by following dependencies through the graph, using a recursive query inside SQLite so the graph does not have to be loaded into memory. It then sweeps, deleting every entry that was not marked, starting from those that depend on others so that no remaining record ever points at a deleted entry.
Store entries are read-only, so the collector restores write permission on each one immediately before deleting it. It removes the files first and the database record second, so an interrupted collection can at worst leave an unregistered directory behind for the next run to remove. Collection holds an exclusive lock on the store, so it never runs at the same time as a build that has not yet registered its result.
Profiles and stack versions¶
The store activates nothing by itself. What a machine uses is decided by profiles. A profile is a sequence of generations, each one a complete environment in the store, and switching to a new generation or back to an old one is a single atomic rename. Every kept generation is a GC root.
Large software stacks, such as an NVIDIA HPC stack made of driver libraries, CUDA, cuDNN and NCCL, are combined into one stack per version. Any number of versions can sit in the store together, and each is assigned to a named slot such as nvidia. A slot is a single symlink at /run/lattice/stacks/<slot>, so exactly one version is active at a time.
---
config:
themeVariables:
fontSize: 16px
flowchart:
useMaxWidth: false
sequence:
useMaxWidth: false
---
flowchart TB
gen["System profile<br/>generation 42"] -->|"stacks: nvidia"| slot["/run/lattice/stacks/nvidia"]
slot --> s570["nvidia-570<br/>CUDA 12.8"]
s560["nvidia-560<br/>CUDA 12.6"]
old["Generation 41"] -.->|"rollback"| s560
Switching versions changes one line of the system description and switches the profile with lattice switch system.pyb. When the kernel driver changes too, Lattice replaces the kernel modules only if the new ones were built for the running kernel, no process has the GPU devices open and no old module is in use. Otherwise the new version takes effect at the next boot, lattice slots shows both versions, and lattice activate-boot makes the change when the machine starts. If loading a new module fails, Lattice puts the old modules back and the profile stays where it was. Running jobs keep the version they started with, and the garbage collector treats every store path a running process uses as a root, so an old stack is never deleted from under a job.
Binary caches and shallow paths¶
Because a store path is named by its inputs, Lattice knows the name of every path before building it, and can first ask a binary cache whether a trusted machine has already built it. A cache is either a directory of static files served over HTTPS, holding a signed metadata file and a compressed archive per path, or a CernVM-FS repository. Lattice accepts a path only with a signature from a key the machine trusts. Each published path also carries signed provenance in the SLSA format, recording the derivation, inputs, build machine and sandbox backend it was built from, and a machine can refuse any path without it.
CernVM-FS fetches each file the first time it is opened and keeps it in a local cache of fixed size, and a file shared by two stack versions is stored once. Lattice can register a path from it as a shallow path, a symlink from the store into /cvmfs, so several versions of a stack of several gigabytes take almost no disk until they are used. This suits virtual machines and container images. Kernel modules are always copied locally, because they are needed at boot, and a shallow path can be turned into a full local copy before a machine goes offline.
---
config:
themeVariables:
fontSize: 16px
flowchart:
useMaxWidth: false
sequence:
useMaxWidth: false
---
flowchart TB
eval["Evaluated derivation<br/>hash known"] --> local{"In local store?"}
local -->|"yes"| done["Use it"]
local -->|"no"| http["HTTP cache<br/>full copy"]
local -->|"no"| cvmfs["CernVM-FS<br/>full copy or shallow"]
local -->|"no cache has it"| build["Build in sandbox"]
http --> done
cvmfs --> done
build --> done
A build machine creates a key once and pushes what it builds, and every other machine lists the cache and trusts the key.
lattice generate-key cache.example.org-1 /etc/lattice/cache.key # prints the public key
lattice push --to https://cache.example.org /lattice/store/<hash>-hello
# /etc/lattice/settings.toml on a machine that uses the caches
substituters = ["cvmfs://sw.example.org", "https://cache.example.org"]
trusted_keys = ["cache.example.org-1:..."]
require_provenance = true
[cache.shallow]
slots = ["nvidia"] # stacks in these slots, and their contents
packages = ["cuda-*"] # derivation names
lattice build then fetches every path a trusted cache holds and builds the rest. lattice substitute <path> fetches one path, lattice substitute --shallow <path> links it from CernVM-FS, and lattice materialise <path> turns a shallow path into a full copy.
Implementation¶
The store is the lattice.store package, written in Python on top of the standard sqlite3 module.
| Module | Contents |
|---|---|
layout.py |
Where a store keeps its files, and the checks on store path names |
lock.py |
The store lock and the lock of each derivation |
verify.py |
The content hash of an entry, and the files that differ between two entries |
references.py |
Finds the store paths an entry refers to while it is hashed |
database.py |
The SQLite schema, registration, roots and the reachability queries |
realise.py |
Normalises an output, publishes it and registers it |
fetch.py |
Downloads a file whose SHA-256 digest is known in advance |
builder.py |
Orders the builds a derivation needs and runs them in parallel |
gc.py |
The garbage collector |
combine.py |
Builds stacks and profile environments as directories of symlinks with a manifest |
Binary caches are the lattice.cache package.
| Module | Contents |
|---|---|
signing.py |
Ed25519 keys and signatures, with pyrixos-ed25519 |
metadata.py |
The signed metadata of a cache entry and the lattice-cache.json file |
archive.py |
Deterministic archives, and unpacking that refuses unsafe entries and limits the size |
provenance.py |
SLSA provenance in an in-toto statement, signed in a DSSE envelope |
http.py and cvmfs.py |
Reading and writing HTTP, directory and CernVM-FS caches |
substitute.py |
Fetches, checks and registers paths from caches, and materialises shallow paths |
push.py |
Signs and publishes paths, their closures and their provenance |
The scan of running processes for store paths in use is lattice.procscan. Profiles, generations and slots are the lattice.profiles package (generations.py, slots.py and activation.py), and reading and loading kernel modules is the lattice.kmod package, which calls the Linux finit_module, init_module and delete_module system calls directly.
The schema, locking rules, deletion order, activation steps and cache format are specified in RFC 0001 §10 to §14.