snowglobe

Running outside a browser

Read this when you want a site in your terminal, in CI or on a server. The same snapshot runs in a container or natively, from a folder, a tarball or a deployed site's URL. Each named run is an instance.

Sessions and instances

snowglobe run dist/elixir                    # a session; ctrl-] quits
printf 'Enum.sum(1..100)\n' | snowglobe run dist/elixir
snowglobe run dist/elixir --name pg --detach
snowglobe exec pg 'sql "SELECT 1"'           # prints only what the command printed
snowglobe attach pg                          # join it; ctrl-] detaches
snowglobe ps
snowglobe stop pg
  • In a container by default (--runtime docker): unprivileged, read-only, memory-capped.
  • Natively with --runtime native (or SNOWGLOBE_RUNTIME=native): it runs snowglobe-vm itself, with no Docker.

Piped input is typed in one line at a time, each waiting until the app goes quiet. Each instance starts fresh from the snapshot.

Sites from a URL

A site doesn't have to be local. snowglobe run also takes:

  • a deployed site's URL, which it mirrors into a cache, or
  • a tarball made with snowglobe pack, local or by URL.

Downloads happen on the host and are revalidated. The second run starts from the cache, and the sandbox stays offline.

snowglobe run https://filipecabaco.github.io/snowglobe/python/
snowglobe pack dist/elixir -o elixir.tar.gz
snowglobe run https://cdn.example.com/elixir.tar.gz

Serving from the guest

A site built with --network fetch can serve. -p HOST:GUEST forwards a port on your machine to one in the guest. Host ports bind to 127.0.0.1 unless you give an address: -p 0.0.0.0:8080:4000.

One executable

snowglobe bundle makes a site a single executable: snowglobe-vm with the site appended. It needs no Docker, hypervisor or KVM, so it runs on a laptop, in CI or inside someone else's container.

Build with --sandbox for a bundle. The guest then keeps its file cache in the snapshot, and exercises run before it.

snowglobe build examples/python --sandbox --out dist/python
snowglobe bundle dist/python -o python        # Linux; --target darwin for Apple silicon
echo 'sum(i*i for i in range(10))' | ./python

A bundle whose app is a shell also runs one-shot jobs:

  • --workspace DIR is written into a fresh machine, --cmd runs there, and its output, exit status and out/ come back.
  • --serve SOCKET --jobs-root DIR restores the snapshot once and answers each job with a fork of it, sharing the restored machine copy-on-write.
  • --server SOCKET is that server's client.
./toolkit --workspace job --cmd 'sh main.sh' --env 'N=42'
./toolkit --serve /run/toolkit.sock --jobs-root /var/lib/jobs &
./toolkit --server /run/toolkit.sock --workspace /var/lib/jobs/1 --cmd 'sh main.sh'

A job's limits

Past its --timeout, nothing a job may use is limited unless asked. Each limit is a flag of a one-shot job and the same field of a request to --serve, where every forked job has its own:

  • --disk-limit MB (disk_limit_mb): what the guest's file writes may hold (its filesystem is in the process's memory). Past it, writes fail with ENOSPC and df shows what is left. What the job prints has a limit of its own, outside it.
  • --max-rss MB (max_rss_mb): the process's (or the fork's) resident memory. Past it the job ends at once, out of memory (exit 137).
  • --memory-limit MB, --cpus-limit N, --pids-limit N (memory_limit_mb, cpus_limit, pids_limit): the command runs in a cgroup v2 in the guest, made at boot and reused, so a limit costs a job a few writes, not a process. A guest OOM kill ends it with 137, and whatever the job left running ends with it. The memory limit needs a site built with snowglobe build --memory-limits (its guest boots with the kernel's memory controller, which costs fork-heavy jobs about 5%); the CPU and process limits work on any site built by this snowglobe. A guest that can't enforce one (a site built without it, before its kernel had cgroups, or with no kernel) fails the job, saying so.
  • --cpu-time S (cpu_time_s): the process's CPU time, every thread. Past it the job ends (exit 152); RLIMIT_CPU backs it up.
  • --output-limit BYTES (output_limit): past it the command is killed and its output cut there (exit 153).
  • --copy-limit MB (copy_limit_mb, default 2048): the workspace copied in, and out/ copied back, each. out/ comes back even from a job out of time, and is never written through a symbolic link.
  • --read-only-in (read_only_in): in/ is mounted read-only in the guest.
  • --egress none|https|internet|private (egress): https is TCP 443 to public addresses only. A forked job's is at most its server's.

--status-file PATH writes the result as JSON as the run ends: status (null if the command didn't end by itself), exit, timed_out, killed, oom, output_truncated, disk_full, cpu_time_exceeded, out_truncated, ms. A server's answer carries the same fields. Each forked job is confined to its own workspace (Landlock, or Seatbelt), not the rest of the jobs root, can't connect to any Unix socket, and is cancelled when its client's connection closes. A job writes nothing its server shares with other jobs: a server's --log-dir has the server's own events (each job's log handles are closed before it runs), and the JIT profile a server starts from is only read, never added to by its jobs (a single-tenant snowglobe run or one-shot job still adds to it).

./toolkit --workspace job --cmd 'sh main.sh' --memory-limit 256 --disk-limit 512 --status-file job.json

Instances share their memory

Native instances of one site don't each hold a copy of its snapshot:

  1. The first one writes the snapshot's memory to an image in the cache.
  2. Every instance after it maps that image copy-on-write. Pages no instance writes are kept once for all of them. A page an instance writes becomes its own.
  3. An instance restores in a few milliseconds instead of decompressing the snapshot first.
Ten instances of the Python runEach, beforeEach, now
Memory on Linux (PSS)111 MB46 MB
Memory on macOS (physical footprint)175 MB96 MB
Restore60–74 ms4–6 ms
  • Where: ram/ under the cache (SNOWGLOBE_CACHE, by default ~/.cache/snowglobe), one image per snapshot, named by its hash.
  • Disk: about as much as the snapshot's memory in use (71 MB for the Python run). Images unused for two weeks go when a new one is made.
  • Forks: machines forked from a template, by --fork or the daemon, already shared the template's memory and still do. Their template server maps the image too.
  • Off: SNOWGLOBE_RAM_IMAGE=off restores into memory of the instance's own.

How the image is kept from changing under a machine is on Isolation.

Next: drive an instance from a program on For agents and programs, or run many machines on Orchestration and fleets.