Day-two operations

Once it's running, a fleet mostly looks after itself: hosts update from Git, back themselves up, and report health. These are the controls you'll use when it doesn't.

Updates

kittens-auto-update, which every host runs from your main branch

Hosts

Drain, resume and retire with the Marshall operator CLI

Backups

kittens-backup, restic to S3-compatible storage

Automatic updates

Each host builds a revision once, stages its secrets, applies compatible migrations, switches, and checks health twice, five seconds apart. A failing revision backs off from thirty seconds to fifteen minutes; pushing a fix skips the wait.

sudo kittens-auto-update status
sudo systemctl start kittens-auto-update.service
sudo journalctl -u kittens-auto-update.service
sudo systemctl stop kittens-auto-update.timer   # before manual recovery

Hosts don't coordinate their restarts with each other. Kernel changes and incompatible schema or protocol changes need planned maintenance.

Drain a host for maintenance

bin/marshall eval 'Marshall.Operations.CLI.main()' drain HOST_UUID REQUEST_UUID ops:ticket-123
bin/marshall eval 'Marshall.Operations.CLI.main()' drain-status REQUEST_UUID
bin/marshall eval 'Marshall.Operations.CLI.main()' resume-host REQUEST_UUID
bin/marshall eval 'Marshall.Operations.CLI.main()' retire-host REQUEST_UUID

Drain before rebooting or replacing a host. Stopping Lilly's VM slice doesn't count as confirmed cleanup. Guests keep running when the Lilly daemon itself restarts.

Move machines between hosts

Owners, managers and team admins can relocate a stopped machine with ssh <gateway> relocate NAME. Marshall chooses the destination host and transfer method. The machine keeps its identity, name, sharing and private address.

Backups

services.kittensBackups = {
  enable = true;
  postgres.enable = true;
  s3 = {
    endpoint = "https://objects.example.com";
    bucket = "node-recovery";
  };
};

kittens-backup list
kittens-backup check
kittens-backup read-check

Only enabled roles add their state. By default that's the host identity and receipts; Lilly's journals, image templates and every ready guest disk (from native ZFS snapshots); a logical dump of Marshall's PostgreSQL; Porsche and Audi keys; and ACME state. A stopped guest is never started just to back it up. A running guest is asked to sync first, but isn't paused.

Logs and metrics

services.kittensLogging sends journals from Marshall, Lilly, Porsche, HAProxy and the system to CloudWatch, OTLP or a fleet log store, with a bounded on-disk queue. Ziggy and Galileo record usage per account.

Credentials

Rotate a Lilly credential with nix run .#host -- rotate-credential, using the same arguments as prepare-enrollment. Rotation revokes the old credential immediately, so deploy the new ciphertext promptly. Signing secrets, the database URL and the active Cloak key can't change through automatic updates; rotating them needs the coordinated procedure.

Full reference: docs/host-maintenance.md ↗

Full reference: docs/backups.md ↗

Full reference: docs/observability.md ↗