How it works

This page describes the mechanism, for someone deciding whether to trust it with the only durable copy of their session history. Recall archives AI coding sessions by copying files. There is no server, no daemon, and no database server; nothing on the preservation path ever calls an LLM or a cloud API. An hourly scheduler unit runs one pure-stdlib Python script straight from the repo checkout, and correctness comes from the shape of the data and a small set of preservation rules, not from a coordinator.

Three sources feed the archive: a ~/.claude-shaped tree (Claude Code), ~/.gemini (Gemini CLI and the Antigravity IDE), and ~/.codex (Codex CLI). Each is mirrored faithfully — the tool’s own layout, unchanged — into one dataset root. Concepts defines the vocabulary; this page explains why the archive is safe to leave alone for years.

The local root is really local

Every machine archives to a machine-local root that no cloud mount, permission fence, or network outage can deny. The installer proposes a platform default — ~/Library/Application Support/pibmo/recall on macOS, ${XDG_DATA_HOME:-~/.local/share}/pibmo/recall on Linux and WSL (on WSL, explicitly on ext4, never /mnt/*) — and refuses cloud-synced and remote paths for it. The refusal is not doctrine for its own sake; each listed risk took down a real machine in the fleet this tool is dogfooded on:

REFUSED as the local dataset root: ~/Library/CloudStorage/Dropbox/Datasets/pibmo
  - inside ~/Library/CloudStorage (a macOS cloud-provider mount)
  - path component 'Dropbox' looks cloud-synced (dropbox)
Risks of a cloud-synced local root (the 2026-06/07 fleet outage class):
  - schedulers can be permission-fenced off the path (macOS TCC denies
    launchd and cron access to ~/Library/CloudStorage) -> total, silent outage
  - the cloud client can evict files to dataless placeholder stubs the
    sync cannot read
  - client rewrites and conflict copies violate append-only preservation
  - when the client is not running, nothing is preserved at all
The local root must be a plain local path -- a hard rule of the
preservation design. A cloud-synced folder is welcome as the CONSOLIDATED
root instead -- that is replication's job, not preservation's.
Detection is heuristic: re-run with --force to override eyes-open.

Detection is heuristic by design (a custom Dropbox location or a symlinked alias is not always detectable), so --force exists as an informed override rather than a wall that teaches users to disguise paths. A cloud-synced folder has a legitimate role — as the consolidated root that joins machines, where the vendor’s client is doing replication’s job, not preservation’s. See configuration for how the root and the consolidated root are set.

On-disk layout

One dataset root holds everything. Each machine writes exactly one subtree, named after itself; shared config sits at the top:

<root>/
  machines.json                    # machine registry: hardware UUID -> name (installer-owned)
  projects.json                    # process-time grouping map -- never sync routing
  profiles.json                    # denylist processing profiles
  <machine>/                       # one single-writer subtree per machine
    machine.json                   # this machine's identity record
    pibmo.db                       # provenance: sync runs, file observations, schema evolution
    logs/cron.log                  # every scheduled tick, appended
    claude/projects/<encoded>/     # faithful mirror of ~/.claude/projects
      <uuid>.jsonl                 # one session transcript
      <uuid>.fork-<sha8>.jsonl     # a divergent rewrite, preserved as a sibling
      <uuid>/subagents/            # subagent transcripts
    gemini/                        # mirror of ~/.gemini -- Gemini CLI and Antigravity
    codex/                         # mirror of ~/.codex -- Codex CLI
  processed/<key>/<profile>/       # converter output: merged Markdown per project

The raw mirror is faithful: no renames, no key remapping, no folder reorganisation. claude/projects/-Users-alice-code-myapp/ in the archive is byte-named after ~/.claude/projects/-Users-alice-code-myapp/ on disk. Mapping those encoded paths to human project names is the converter’s job at process time (via projects.json), never the sync’s — so a wrong or stale mapping can mislabel Markdown output but can never misroute or rename raw data.

What never enters the archive

Exclusions are by name; include is the default. The sync excludes exactly three classes — credentials, application code and caches, and mutable derived state — and everything else flows through:

Source	Mirrored from	Excluded, by name
Claude Code	`~/.claude/projects/`	each project’s agent-memory subtree (mutable, derived — not raw session data)
Gemini CLI + Antigravity	`~/.gemini/`	`oauth_creds.json`, `google_accounts.json`, `antigravity-oauth-token`; `extensions/` (installed code); the Chromium browser profile; per-data-dir `browser_recordings/`, `bin/`, `cache/`, `builtin/` (media, binaries, caches)
Codex CLI	`~/.codex/`	`auth.json`; `.tmp/`, `tmp/`, `cache/` (ephemera)

Credential files never enter the archive. The lists above are measured against the real trees, not guessed — with the awkward cases kept, not dropped: Gemini CLI stores session data under ~/.gemini/tmp/ (tmp/<project>/logs.json, chats/), so Gemini’s tmp/ stays in even though Codex’s tmp/ is excluded as genuine ephemera. When in doubt, the bias is to archive: a denylist fails visibly (noise you can see), an allowlist fails silently (data you never knew you lost).

Why nothing is ever lost

A file in the archive is preserved, never overwritten or shrunk. The preservation engine routes every source file through one of three primitives by type:

File type	Primitive	Rule
`*.jsonl` transcripts	Line lineage	identical -> skip; append -> extend the archived copy; shorter than archived -> skip, never shrink; divergent rewrite -> preserved as a sibling fork
SQLite databases (`.db`, `.sqlite`)	Logical snapshot	a raw snapshot (taken via the SQLite backup API, so WAL content is included) is kept only when the deterministic logical dump hash changes; a `.sql` text dump is stored beside it for diff and grep
Everything else	Content-addressed versions	every distinct content hash becomes an immutable `{stem}.{sha8}{suffix}` version, plus a `{name}` pointer tracking the latest

The fork rule is what makes destructive tool behaviour survivable. When a tool rewrites a transcript in place — compaction is the common case — the archived lineage and the new content share a prefix and then diverge. The sync writes the new content as a sibling, {uuid}.fork-{sha8}.jsonl, where sha8 hashes the first divergent line. That key is stable as the fork grows, so a growing fork extends its own file instead of spawning duplicates; a fork that itself diverges chains ({uuid}.fork-aaaa.fork-bbbb.jsonl). Both histories survive. Nothing decides which one was “right”.

Two mechanical guarantees underneath:

All writes are atomic: temp file in the destination directory, fsync, rename. A crash mid-write cannot leave a corrupt or truncated file in the archive.
Unchanged files are skipped without being read: a (size, mtime_ns) fast path makes the hourly tick cheap on a tree of thousands of transcripts.

The sync lock

Before touching the root, the sync takes a non-blocking exclusive flock. Be precise about what this does and does not protect:

Protects	Does not protect
Serializes syncs on one machine: a second run against the same root (including the legacy importer) exits immediately instead of interleaving writes — which is why running the sync by hand is safe at any time	Anything across machines — an `flock` is local to one host’s kernel

Cross-machine safety does not need a lock at all: each machine writes only its own {root}/{machine}/ subtree, so two machines’ writers touch disjoint paths by construction. Note also where the lock lives — in a local directory (/tmp by default), keyed by a hash of the root path, not inside the root. The consolidated root may be cloud-synced storage, where flock semantics are unreliable and lock churn would replicate to every machine.

The schedule is a native unit

The installer writes a native scheduler unit per platform rather than pasting a crontab line. This is a post-mortem decision: raw cron failed differently on every OS in the dogfooding fleet — a PATH defect on one Mac, macOS TCC silently denying cron any access to a cloud-synced path, a moved repo freezing another machine’s cron line — and the archiving system had no way to notice it had gone dark.

Platform	Unit	Missed-tick behaviour
macOS	launchd LaunchAgent `com.pibmo.recall.sync`	ticks missed while the Mac slept coalesce into one run on wake; `RunAtLoad` adds a catch-up run at login
Linux	systemd user service + timer `pibmo-recall-sync`	`Persistent=true` fires a missed tick at the next opportunity
WSL	Windows Task Scheduler (the command is printed; the unit cannot be written from inside the distro)	per Task Scheduler

Units are deterministic — the same inputs render byte-identical files — so scheduler.py --check (and the doctor) can verify the installed unit against a fresh render and catch drift, such as a repo that has moved out from under its unit. Crontab remains a documented fallback, not the default.

The unit points at the repo checkout: there is no deployed copy and no self-updater. Updating is git pull; the next tick runs the new code. Since extra runs are idempotent no-ops, catch-up runs cost nothing.

Provenance in SQLite

Every run records what it saw in {root}/{machine}/pibmo.db: the run itself (the heartbeat the doctor and the session banner read), a per-file observation log, and a schema manifest of JSONL event types collected incrementally from the lines actually read. The schema manifest is a canary: vendors add event types without notice, and a schema change on this machine is detected and named at sync time rather than discovered months later in processing.

Consolidation and replication

Multi-machine operators join machines at a consolidated root in two explicit, separate steps — neither of which is required on a single computer, where the local root is the whole product.

Consolidation (--consolidate) pushes this machine’s subtree only into the consolidated root as a grow-only mirror. Shared config (machines.json, projects.json, profiles.json) never crosses — the destination’s copies are authoritative and other machines depend on them. The copy rule is type-scoped: *.jsonl files only ever grow in the primary, so the mirror copies them only when they grew — a newer-but-smaller transcript (a primary rebuilt or restored with fresh mtimes) can never clobber a longer copy at the destination. There is no --delete anywhere: a source-side regression must not propagate a deletion into the consolidated root.

Replication to cloud storage is delegated, permanently, by design. Recall never speaks a cloud API and never bundles a cloud client; it ships recipes (rclone is the reference tool — API-direct, no mount, works headless) and the doctor measures the outcome. Every fragility in the fleet’s failure ledger — TCC fences, dataless-file eviction, clients not running — lives in the mechanism of writing through a vendor mount, none of it in the cloud service itself. Using a vendor mount (Dropbox, OneDrive) as the consolidated root is permitted with eyes open: that configuration simply means the vendor’s own client is the replication tool. See topologies for the shapes this takes across machines and satellites.

The doctor

The 2026-07 fleet outage proved the one question nobody was asking: is this machine actually archiving? The doctor answers it, with an exact remediation per failure: is the machine registered; is the root reachable and writable (verified by a write-and-delete canary — the only write it ever makes); does the scheduler unit exist, match a fresh render, and point at an interpreter and script that still exist; how old is the last sync heartbeat; and, when a consolidated root is named, how stale is each machine’s replicated subtree there. It exits 0 when everything passes (warnings allowed), 1 on any failure.

The output below is a real run on the author’s Mac, reproduced as printed with home-directory paths shortened and four more fleet rows trimmed:

$ python3 scripts/sync/doctor.py --root "$HOME/Library/Application Support/pibmo/recall" \
    --python /usr/local/bin/python3 \
    --consolidate "$HOME/Library/CloudStorage/Dropbox/Datasets/pibmo"
Pibmo Recall doctor -- machine 'moraba', root ~/Library/Application Support/pibmo/recall
  OK    root         ~/Library/Application Support/pibmo/recall/machines.json readable (1 machine(s) registered)
  OK    identity     381CB91B...
  OK    registered   'moraba'
  OK    writable     canary ok under moraba/logs/
  OK    unit         1 unit file(s) current
  OK    unit-cmd     embedded interpreter and script exist
  OK    heartbeat    last completed sync 2026-07-21T19:40:15.586026+00:00 (0.3h ago)
  OK    log          ~/Library/Application Support/pibmo/recall/moraba/logs/cron.log (modified 0.2h ago)
  WARN  consolidated this machine's replicated heartbeat 2026-06-12T13:06:45.775023+00:00 (942.6h behind local)
        fix: transport is lagging -- check the tool that moves this subtree (vendor client / rclone / pull job)
  WARN  fleet        seneca: heartbeat 659.0h old
        fix: that machine (or its transport) has gone dark -- run the doctor there
Summary: 8 ok, 6 warn, 0 fail

Note what the warnings are doing: replication is delegated, so the doctor cannot see the replication mechanism — but each machine’s heartbeat travels with its consolidated subtree, so staleness per hop is still observable. It measures outcomes, not mechanisms. The same watch-the-watchers principle runs in the session banner, which warns at session start when this machine’s archive is stale.

One rule backs all of this: the preservation path never swallows a diagnostic. A bare except once hid a five-week outage; the invariant now is that code on the preservation path must surface the underlying error before degrading — refusing loudly is the only acceptable failure mode.

From archive to Markdown

Reading raw JSONL is nobody’s idea of recall, so a separate processing layer projects the archive into per-project Markdown under processed/<key>/<profile>/. It is deliberately boring: deterministic Python, no LLM calls, safe to delete and re-run since raw is never touched.

Grouping happens here, at process time. The sync auto-appends newly seen projects to projects.json with process: false; you edit a row’s key (or consolidate_into) to merge the same project’s transcripts across machines and checkout paths into one output tree. Editing that file changes only how the converter groups Markdown — the raw archive is never rerouted or renamed.

Profiles are denylists: faithful excludes nothing (lossless by construction), and dialogue excludes machinery — tool calls, thinking blocks, harness text — by name. The asymmetry is the point: when a vendor ships a new event type, a denylist profile includes it visibly — the reader notices noise, the schema canary names the type, the config gets a one-line update, and nothing was lost in the meantime. An allowlist would have dropped it silently, which is the one failure mode this project exists to prevent.

Performance envelope

Be candid about what this is for. The sync is an hourly file copy with a (size, mtime_ns) fast path — on an idle machine a tick reads almost nothing, and on a busy one it moves kilobytes of appended transcript lines. It is designed against the session tools’ rotation windows, which are measured in days, not against real-time capture: between ticks, a transcript’s only copy is the tool’s own, for up to an hour. And because nothing is ever deleted, the archive only grows — a few GB per machine, stored twice under consolidation. That is the price of preservation nothing can fence off, paid in the cheapest currency available.

What this does not give you

Trust also means being clear about the boundary.

Exclusion is not redaction. The denylist keeps the tools’ credential files out of the archive. It cannot redact the transcripts themselves: a secret pasted into a session is part of the session and is archived faithfully — that fidelity is the product. Treat the archive with the same sensitivity as the sessions it preserves.
Archiving is not access control. The archive is plain files under your home directory. Filesystem permissions are the access control; there is no encryption layer.
One local copy is not a backup. On a single machine, the archive survives the tools’ rotation but not the disk. Durability beyond one machine is the consolidation and replication tiers — and replication is performed by a tool you choose, not by Recall, which can only measure that it happened.
Only on-disk sources are captured. claude.ai web chats and cloud-hosted coding environments never touch your disk; they are an import problem, not a sync problem, and no sync schedule will ever see them.
Nothing resurrects the past. The sync archives what exists on disk at tick time. Sessions a tool rotated away before Recall was installed are gone; install day is day zero.

Getting started — install, first sync, first doctor run.
Concepts — roots, machines, sources, and profiles.
Configuration — the root, the unit line, and the shared config files.
CLI reference — every script and flag.
Topologies — single machine, consolidation, satellites.
Hooks — the session banner that warns when the archive is stale.
Troubleshooting and the FAQ.