Troubleshoot

Start with sandcastle check. It prints one line per item: a FAIL line keeps the installation from working, a warn line is something to know. Every line, and what to do about it, is in Check messages.

Then look at the journals and at the job’s record:

journalctl --user -u sandcastle -f        # the core: one line per poll that found work and per job
journalctl --user -u sandcastle-keep -f   # the keep: one line per request, with destination, path and status
journalctl --user -u sandcastle-gate -f   # the gate: what it let through and what it refused
ls ~/.local/share/sandcastle/jobs/<source>/<job id>/

install refuses the machine

this machine does not serve, nothing was changed:
  …

Each line under it is one thing to fix, with the version found and the one needed. The least versions are in Supported systems.

• The cgroup controller … is not delegated to your user. A box’s limits need the cpu, memory and pids controllers. Delegate them to every user’s systemd, as root:

  sudo mkdir -p /etc/systemd/system/user@.service.d
  printf '[Service]\nDelegate=cpu cpuset io memory pids\n' | sudo tee /etc/systemd/system/user@.service.d/delegate.conf
  sudo systemctl daemon-reload
  

  Then log out and in again, and check with cat /sys/fs/cgroup/user.slice/user-$(id -u).slice/user@$(id -u).service/cgroup.controllers.

• /etc/subuid has no range for you. Add one, as root: sudo usermod --add-subuids 100000-165535 --add-subgids 100000-165535 "$USER", then podman system migrate.

• The link or egress range is already in use. Choose two free IPv4 /24s with --link and --egress. install remembers them. See Network names and ranges.

install stops halfway

After the images are built, install undoes nothing: it stops at the step that failed and says which. Fix the cause and run install again.

systemd did not load sandcastle-keep.service: Quadlet found a problem in a unit; …

A drop-in of yours is the likeliest cause. The line gives the command that shows the problem.

The keep does not answer

FAIL  keep: the keep does not answer at …/sandcastle-keep/keep.sock; start it with sandcastle keep

Under the installation, the keep is a service, so look at it and at the gate, which it waits for:

systemctl --user status sandcastle-keep sandcastle-gate
journalctl --user -u sandcastle-keep -u sandcastle-gate -n 50

The gate does not start when its preparation fails, for one when the configuration has no destination or cannot be read. The core waits for the keep without ending, and carries on by itself once the keep answers.

The gate lost its networks

FAIL  gate: the container lost the networks [sandcastle-egress sandcastle-link]; restart it with systemctl --user restart sandcastle-gate

The gate runs, but with no network, so the keep reaches nothing and every source fails with connect: no route. Nothing gets out: a gate without its networks is closed.

A podman network reload does this. It gives every container its networks anew, and Podman fails for the gate, whose address in the link is fixed. Some tools run it with --all on their own, when the machine’s DNS changes, for one at boot or when a VPN connects. Find which with journalctl --user -b | grep 'network reload'.

The gate’s repair, sandcastle-gate-repair.timer, finds this within a minute and restarts the gate, and the journal says so:

journalctl --user -u sandcastle-gate-repair

To mend it at once: systemctl --user restart sandcastle-gate. Keep the tool from reloading every network if you can: only its own need it.

A source stopped answering

When the gate said no, check shows it at the end, with the likeliest cause:

warn  gate: refused notes 3 time(s) in the last hour, last at 10:15:46 (…): the name resolves to the internal network; …

• The name resolves to the internal network. The gate refuses an address inside your network unless the destination has an internal exception. If the source does live inside, give it one, as in Reach a service inside your network. If it does not, its DNS changed.
• The name is not on the gate’s list. The configuration changed after the gate started: systemctl --user restart sandcastle.target.

A source the gate refused is asked again at its poll_interval. One that cannot be reached is asked again later each time, up to every 10 minutes; restart the core to ask it at once: systemctl --user restart sandcastle.

A token or a key was rejected

FAIL  source notes: the token was rejected; create a new worker token and set it with sandcastle secret set notes

Set the new value with sandcastle secret set. The keep reads it at the next request; nothing needs a restart. A rejected Claude key releases the jobs and the capability rests for its cooldown.

An image is missing or of another build

FAIL  capability ocr: the image sandcastle-ocr:… of this build is missing; run sandcastle install

The binary changed and its images were not built, or the ocr languages changed. Run sandcastle install. While a capability has no image, its jobs are given back (released) and the capability rests.

A job failed

The job’s folder holds what happened: see A job’s record. meta.json has the status, the error and the start of what the box wrote on stderr. The box’s own words never go to the journal: they are hostile, and are kept only in the record.