Test matrix

Each promise of the threat model, the attacks tried against it, and where the proof lives. The ids are stable: a test cites the ids it proves in a comment that starts with them (// S1, N4: …), or at the start of its name or a subtest’s, and a test of the repository holds this page to the tests. Each id needs at least one proof of each of its levels, the proof names every test file that cites the id, and an id a test cites must be on this page.

The level says where the attack runs:

U: a unit test, run by go test ./... on any machine;
I: an integration test, run by go test -tags integration ./... against an installed sandcastle with real Podman, only on the reference machine;
M: a step done by hand for each version, in the manual list.

An attack counts as stopped only when three things hold: it fails; it fails for the reason expected (the gate’s reason, 137 at the memory limit, “read-only file system”, EACCES), never a timeout or any error; and a control, the same attack without the lock or against an allowed destination, succeeds.

Test runs records each version’s run of this matrix. The most recent is the one at the top of that page.

A compromised source, and the content of a job

Id	Promise	Attacks	Level	Proof
S1	reads no key or token	a redirect to the attacker’s server, waiting for the token; a `destination` or `url` in the envelope; text that looks like a token in the errors; the decoy secrets looked for everywhere an attacker sees	U, I	`hostile_integration_test.go`, `internal/keep/keep_test.go`, `main_test.go`
S2	reaches no internal address	a redirect to `192.168.x`, `169.254.169.254` and `localhost`; the source’s name starts to resolve inside; internal URLs in the content	U, I	`dns_integration_test.go`, `hostile_integration_test.go`, `internal/config/config_test.go`
S3	leaves nothing behind the job	the capability writes files and starts processes; afterwards no box with the label, no volume, no file outside the job’s record	I	`attacker_integration_test.go`
S4	spends no more than its limit	jobs without end; content above the maximum; a prompt above `max_input`	U	`internal/box/core_test.go`, `internal/runner/runner_test.go`
S5	sees and changes no job of another source	a job with another source’s job id; with an honest source and an attacking one, the attacker never gets the other’s result or token	U, I	`hostile_integration_test.go`, `internal/keep/keep_test.go`, `internal/source/client_test.go`, `main_test.go`

The program in a box

Id	Promise	Attacks	Level	Proof
B0	runs with every lock of its box	inspecting a running box: no network, a read-only root, no capabilities, `no-new-privileges`, a user that is not the operator’s, `--memory-swap` equal to the memory; an image of another build is refused	I	`integration_test.go`, `internal/box/core_test.go`, `internal/box/podman_test.go`, `internal/config/config_test.go`
B1	reads no key or token	`/proc/self/environ` and `/proc/1/environ`; the mounts; looking for the decoy; the keep’s and Podman’s sockets; the machine’s processes	I	`attacker_integration_test.go`
B2	reaches no network	the list of ways out; abstract Unix sockets; a second request out; choosing the destination, the model or the headers	U, I	`attacker_integration_test.go`, `internal/box/core_test.go`
B3	leaves no file or process	writing to the root; `/tmp` unseen by the next box; a process in the background (`setsid`, a double fork); the core killed midway and the box removed at the next start	I	`attacker_integration_test.go`, `integration_test.go`, `internal/box/podman_test.go`
B4	reads no record of another job	looking for the `data_dir` and the previous job’s result	I	`attacker_integration_test.go`
B5	spends nothing without a limit	memory with swap free; a fork bomb; the processor until its time; an endless `sleep`; filling `/tmp`; flooding stdout and stderr; a huge line with no line break	I	`attacker_integration_test.go`, `integration_test.go`, `internal/box/podman_test.go`

A model’s reply

Id	Promise	Attacks	Level	Proof
M1	is never run	shell commands and “ignore the instructions” reach the source as text; fields outside the schema are refused	U, I	`hostile_integration_test.go`, `internal/capability/claude_test.go`, `main_test.go`
M2	goes only to the source of the job	a reply that asks for another source, or holds another job’s id	U	`main_test.go`
M3	causes no request and no read	URLs and paths (`file:///etc/passwd`) in the reply; the keep’s log shows one request	I	`hostile_integration_test.go`
M4	costs one request at most	a request to go on, `max_tokens`, `tool_use`; no second request leaves	U, I	`hostile_integration_test.go`, `internal/box/core_test.go`, `internal/capability/claude_test.go`

Another account without privileges

Id	Promise	Attacks	Level	Proof
A1	reads no key, token or record	another account reads the configuration, the secrets folder and the `data_dir`	M	the manual list
A2	sees no secret in the processes	`ps auxww` and `/proc/*/environ` during a job, looking for the decoy	M	the manual list
A3	hands sandcastle no job	connecting to the keep’s socket; writing to the `data_dir`	M	the manual list
A4	cannot use the gate	connecting to the gate’s address and port; scanning the machine’s ports	M	the manual list

The network

Id	Promise	Attacks	Level	Proof
N1	the internal network is closed after resolving	an allowed name resolves to each internal range and to `::ffff:` mapped addresses; rebinding with a TTL of 0; a CNAME inside; a mixed answer connects only to the public address	I	`dns_integration_test.go`, `online_integration_test.go`
N2	a sealed address never opens	the configuration refuses `internal:` with each sealed address, loopback and link-local among them; the gate’s files never hold one	U	`internal/config/internal_test.go`, `internal/ranges/ranges_test.go`, `setup_test.go`
N3	an exception opens only its `ip:port`	another port of the same IP; the neighbouring IP; a destination without `internal` resolving inside; another destination pointed at the open `ip:port` dies in the keep’s TLS before its secret is sent	I	`hostile_integration_test.go`
N4	the keep never follows a redirect	covered by S1 and S2	—	`hostile_integration_test.go`, `internal/keep/keep_test.go`, `internal/source/client_test.go`, `main_test.go`
N5	the keep checks the port	the keep only asks for the `name:port` of the configuration; control: the gate alone lets port 22 through	U, I	`dns_integration_test.go`, `internal/keep/gate_test.go`, `internal/keep/keep_test.go`
N6	only the `default` rule	the gate’s file has one rule; a client without a certificate falls in it and in no other	U, I	`install_integration_test.go`, `internal/gate/gate_test.go`, `setup_test.go`
N7	the keep and the gate run with their locks	inspecting the keep (no DNS, no IPv6, only on the link) and the gate (no ports, IPv6 off), and the egress network (subnet, routes, no IPv6); the keep tries to leave without the gate and fails	I	`install_integration_test.go`, `internal/install/inspect_test.go`, `internal/install/units_test.go`
N8	the egress network refuses the inside without the gate	the attacker’s image, on the egress network with the gate’s locks, connects to each of the fourteen ranges and gets `EACCES` at once; `ip route del` fails; no IPv6 route; with an exception, the `/32` is reached and the neighbouring IP is not. Control: a network without the routes is reached	I	`egress_integration_test.go`

Outside the promise

These attacks touch a point outside the promise. They are written down so nobody mistakes them for a gap in the matrix; nothing proves them stopped.

Point	Next to	What an attacker can do
3, a tampered sandcastle	N8	a tampered gate binds a socket to the egress interface (`SO_BINDTODEVICE`) and gets past the routes, as far as Podman’s own addresses
7, an internal destination	N3	read and change what goes to an `http://` internal destination, which nothing encrypts
8, money	S4, M4	spend up to the limit set at the model’s provider, one request per job
14, the operator’s mistakes	B5	use a box’s limits set above their defaults

The manual list

Done by hand on the reference machine, for each version, in the same session as the automatic tests. Each step’s result goes in the version’s record in Test runs.

A1: as sctest-other, an account with no privileges, try to read the configuration folder, the secrets folder and the data_dir. Each read fails with “permission denied”. Control: the operator reads them.
A2: during a job, as sctest-other, run ps auxww and read /proc/*/environ, looking for the decoy SCTEST-CANARY-…. It is never found. Control: the operator finds the decoy in its secret file.
A3: as sctest-other, connect to the keep’s socket and write to the data_dir. Both fail with “permission denied”.
A4: as sctest-other, connect to the gate’s address and port, and scan the machine’s ports. Nothing answers; sandcastle listens on no port of the machine.
Traffic: capture with tcpdump, inside the rootless Podman network namespace, during the integration run. Every flow is keep → gate or gate → an allowed destination.
Boot: restart the machine without logging in. The gate, the keep and the core are up (linger and the enabled sandcastle.target).
Private names: the publication script finds no private name in the git history, on GitHub or in the built site.

Keyboard shortcuts

sandcastle