On-Disk Contract Spec (v1)

This specification defines the versioned on-disk data format and directory layout for Pibmo Recall dataset roots. Any implementation agreeing with this contract can safely share a dataset root: each machine writes its own subtree, a converter on any machine reads them all, and nothing is ever lost. The contract is verified against the reference implementation (scripts/sync/sync-transcripts.py, scripts/sync/preserve.py) and the architecture decisions behind it.

One scoping rule up front: below the per-tool directories, the layout is the source tool’s own, mirrored faithfully. This contract constrains what Recall adds to the archive — forks, versions, snapshots, pointers, provenance — not what Claude Code, Gemini, or Codex emit.

1. Directory layout

A Recall dataset root contains shared config at the top level, one subtree per machine, and a derived output tree:

<root>/
├── machines.json                       # Machine registry (installer-owned)
├── projects.json                       # Process-time grouping map (never sync routing)
├── profiles.json                       # Processing-profile overrides (denylist semantics)
├── <machine>/                          # One subtree per machine; written by that machine only
│   ├── machine.json                    # This machine's identity record
│   ├── pibmo.db                        # Provenance store (SQLite)
│   ├── logs/                           # Operational logs
│   ├── claude/
│   │   └── projects/
│   │       └── <encoded-path>/         # Faithful mirror of ~/.claude/projects/<encoded-path>/
│   │           ├── <uuid>.jsonl        # One session transcript (lineage file)
│   │           ├── <uuid>.fork-<sha8>.jsonl   # Divergent rewrite, preserved as a sibling
│   │           └── <uuid>/
│   │               └── subagents/      # Subagent transcripts (tool-native layout)
│   ├── gemini/                         # Faithful mirror of ~/.gemini (Gemini CLI + Antigravity)
│   └── codex/                          # Faithful mirror of ~/.codex
└── processed/
    └── <key>/
        └── <profile>/                  # Derived Markdown -- regenerable, never authoritative

<machine>: the machine’s friendly name from machines.json, lowercased (e.g. moraba).
<encoded-path>: the source tool’s own project identifier, copied verbatim (e.g. -Users-you-code-myproject). Never renamed, never remapped.
<uuid>: the session UUID assigned by the source tool; it is the transcript’s filename stem.
<sha8>: the first 8 lowercase hex characters of a SHA-256 digest. Which bytes are hashed depends on the preservation primitive (section 5).
<key> / <profile>: a logical project key from projects.json and a profile name from profiles.json.

Everything under <machine>/claude/, <machine>/gemini/, and <machine>/codex/ is raw and immutable. Everything under processed/ is derived: deleting it loses nothing, because the converter regenerates it from raw.

2. Machine identity and write scope

Identity: a machine is identified by its stable hardware UUID — IOPlatformUUID from ioreg -d2 -c IOPlatformExpertDevice on macOS, /etc/machine-id (fallback /var/lib/dbus/machine-id) on Linux.
Registration: machines.json maps hardware UUID to friendly name. Registration is the installer’s job; an implementation that finds its UUID unregistered must refuse to sync, not invent a folder.
Folder naming: the per-machine directory is the lowercased friendly name.
Write scope: a machine writes only under its own <machine>/ subtree. The one shared file the sync touches is projects.json, which it extends by atomic append of newly seen rows (section 6.2). Cross-machine writes are forbidden, so two machines sharing one root never contend for the same file.

3. Timestamp format

All timestamps Recall writes (machine.json, pibmo.db) use ISO 8601 UTC with an explicit +00:00 offset and microsecond resolution, as produced by Python’s datetime.now(timezone.utc).isoformat(): YYYY-MM-DDTHH:MM:SS.ffffff+00:00 (e.g., 2026-07-22T03:14:07.123456+00:00). Timestamps embedded inside mirrored source files are the vendors’ own and are out of scope.

4. Source trees and exclusions

Each source tool’s tree is mirrored under the machine subtree, minus a denylist of named exclusions. Everything not named flows into the archive by default — the same default-include model the processing profiles use (section 6.3).

Tool	Source	Destination	Excluded by name
Claude Code	`~/.claude/projects/`	`<machine>/claude/projects/`	the `<encoded-path>/memory` directory (auto-memory: mutable and derived)
Gemini CLI + Antigravity	`~/.gemini/`	`<machine>/gemini/`	`oauth_creds.json`, `google_accounts.json`, `extensions/`, `antigravity-browser-profile/`; inside any `antigravity*` data dir: `browser_recordings/`, `bin/`, `cache/`, `builtin/`, `antigravity-oauth-token`
Codex CLI	`~/.codex/`	`<machine>/codex/`	`auth.json`, `.tmp/`, `tmp/`, `cache/`

Two rules the table encodes:

Credentials never enter the archive. Every excluded credential file is excluded by name, at sync time, on every machine.
Exclusion lists are per-tool, not generic. Gemini CLI keeps session data under tmp/ (tmp/<project>/logs.json, chats/), so Gemini’s tmp/ is included while Codex’s tmp/ is ephemera and excluded.

The archive is therefore deliberately not a full backup of the source trees: what is absent is absent by named rule, and the rule is the contract.

Layout note (Gemini, pre-2026-07-21): archives synced before 2026-07-21 hold Antigravity’s contents flattened directly at <machine>/gemini/. Those files are frozen in place — the archive is append-only — and newer runs write the faithful layout beside them. Readers must tolerate both.

5. Preservation primitives

An archived file is never overwritten with different content and never shrunk. Every write is atomic: temp file in the destination directory -> fsync -> rename, so a crash mid-write cannot corrupt the archive. Which archived state is “the real transcript” is decided at process time; the sync’s only job is to lose nothing.

Three primitives, routed by file type.

5.1 Lineage — append-only line logs (`*.jsonl`)

Source and archived content are compared on complete lines only; a trailing line without a newline is an in-progress write and is ignored. Four outcomes:

Identical -> no write.
Extend (archived content is an exact line-prefix of source) -> the lineage file is rewritten with the longer content. This is the only permitted in-place change to a raw file, and it is a strict superset.
Regression (source is a line-prefix of archived) -> skipped. A lineage file never shrinks.
Divergence (the lines differ before either ends — the compaction/rewrite case) -> the existing lineage is left untouched, and the divergent content is written as a sibling: <name>.fork-<sha8>.jsonl, where <sha8> hashes the first divergent line. That key is stable as the fork keeps growing, so re-syncing a growing fork extends it (rule 2) instead of spawning duplicates. A fork that itself diverges chains the suffix: <name>.fork-<sha8a>.fork-<sha8b>.jsonl. The reference implementation bounds fork chains at depth 32 and refuses beyond it.

5.2 Content-addressed versions — opaque rewritten files (everything else)

Files rewritten wholesale (Gemini .pb, metadata JSON, …) have no line lineage. Every distinct content hash is preserved as an immutable version beside a pointer:

Version file: <stem>.<sha8><suffix> (e.g. logs.json -> logs.a1b2c3d4.json), where <sha8> hashes the full file content. Write-once.
Pointer file: the bare <name>, carrying a copy of the latest content.

Re-syncing unchanged content is a no-op. If the pointer’s bytes ever match no recorded version (out-of-band edit), that content is versioned first — nothing is overwritten-and-lost.

5.3 SQLite logical snapshots (`.sqlite`, `.db`)

Raw databases churn physically (WAL, vacuum, page reorder) without logical change, so file hashes are noisy. Instead, a deterministic logical dump (schema sorted, every table’s rows ordered by primary key) is hashed, and only a changed logical hash triggers preservation:

Raw snapshot: <stem>.<sha8><suffix> — the actual database file, content-addressed by the first 8 hex of the logical dump’s SHA-256. Write-once.
Dump: <stem>.<sha8>.sql beside it, for diffing and grep.
Pointer: the bare <name>, updated to the latest snapshot.

Raw is preserved faithfully (real DB files, not just dumps); physical churn produces no version noise.

6. Shared config formats

6.1 machines.json

{
  "machines": {
    "<hardware-uuid>": "<FriendlyName>"
  }
}

Installer-owned. The sync reads it and never writes it. Folder names under the root are the lowercased values.

6.2 projects.json

Schema version 3. One row per encoded path:

{
  "schema_version": 3,
  "projects": [
    {
      "key": "myproject",
      "encoded_path": "-Users-you-code-myproject",
      "git_repo": "/Users/you/code/myproject",
      "process": true
    },
    {
      "key": "myproject-worktree",
      "encoded_path": "-Users-you-code-myproject--worktrees-feature-x",
      "git_repo": "/Users/you/code/myproject/.worktrees/feature-x",
      "process": false,
      "consolidate_into": "myproject"
    }
  ]
}

key: unique folder name for processed output (a primary key).
encoded_path: the source tool’s identifier, verbatim.
git_repo: the working repo’s path, informational.
process: whether the converter emits output for this row.
consolidate_into (optional): the canonical row’s key, folding several paths (clones, worktrees, machines) into one logical project at process time.

This file is a process-time grouping map, never a sync route: editing any row changes only how the converter groups output; the raw archive is never rerouted or renamed. The sync auto-appends a row for each newly seen encoded path — key equal to encoded_path, git_repo empty, process: false — via atomic write, so a crash can never truncate the hand-curated map.

6.3 profiles.json

Processing profiles are denylists by design: a profile declares what to exclude; everything else flows through by default. Two profiles ship:

faithful: excludes nothing. The lossless rendering of the raw transcript, subagents included.
dialogue: excludes, by name, the machinery — non-conversation event types, thinking/redacted_thinking/tool_use/tool_result/image blocks, meta-flagged and sidechain lines, slash-command echoes, embedded system-reminder text. Subagents excluded. Lossy by design, for wiki/LLM ingestion.

Each profile carries a version; bumping it re-emits that profile’s outputs. The asymmetry is the point: when a vendor ships a new event type, a denylist includes it visibly until the config learns its name — visible noise over silent loss.

7. Per-machine state

7.1 machine.json

Written only by the machine itself:

{
  "uuid": "<hardware-uuid>",
  "name": "<machine>",
  "folder": "<machine>",
  "first_sync": "<ISO8601 +00:00>",
  "last_sync": "<ISO8601 +00:00>"
}

first_sync is preserved once set; last_sync advances on every run.

7.2 pibmo.db

The per-machine provenance store, SQLite. It is the audit trail for “what did we see, and when”:

Table	Holds
`sync_runs`	One row per run: start/completion timestamps, machine, per-tool action counts
`observations`	One row per preserved-file event: timestamp, tool, relative path, primitive (`lineage` \| `blob` \| `sqlite`), action, line/byte counts
`file_state`	Fast-path cache per file: size, mtime, line count, tail hash — unchanged files are skipped unread
`db_logical_state`	Last logical hash and schema SQL per source database
`schema_events`	Every event type ever seen per tool, with first-seen timestamp
`schema_snapshots`	Per-run JSON snapshot of each tool’s schema state
`schema_changes`	Unified diff against the previous snapshot — the schema-drift canary

Lineage actions recorded in observations are new, identical, extend, skip-regression, fork, and their fork- prefixed variants when resolved against a fork sibling; the blob primitive records identical, new-version, and pointer-update; the SQLite primitive records identical and new-version. file_state is a cache: deleting it forces a full recomparison on the next run and loses nothing.

8. Mutation invariants

Accumulate-only: nothing under a raw tree is ever deleted, overwritten with divergent content, or shrunk. The only in-place rewrites are a lineage extension (strict line-superset, 5.1) and a pointer update whose displaced content is already versioned (5.2, 5.3).
Raw authority: raw is immutable and everything else is derived. processed/ can always be rebuilt from raw; raw can never be rebuilt from processed/.
Faithful mirror: no renames, no remapping, no consolidation inside raw trees. Grouping lives in projects.json and is applied at process time only.
Per-machine scope: all raw and state writes for a machine occur exclusively under <machine>/.
Canonical-state deferral: which of a lineage and its forks is “the real transcript” is a process-time decision. The sync records states; it never adjudicates them.
Fail loudly: an implementation that cannot uphold a rule must refuse and surface the diagnostic. Refusing is the only acceptable failure mode; silent degradation is a contract violation.