Status: Design (hands-off)
Module: sourcecraft.dev/bigbes/lethe
Depends on: lethe-server.md (#1) — FTS5 tables and triggers were created in #1; this task only adds query code. lethe-collector-claude-code.md (#2) — the collector framework and Parser interface this task extends.
Sibling tasks (deferred): per-tool parsers (lethe-collector-crush.md, lethe-collector-pi.md, lethe-collector-kimi.md); RFC backlog items (cost rollups for tools that report it, tagging, JSON/Markdown export).
Make the archive answer "what did I ask any robot last Tuesday about X" in under five seconds, and add the second tool to prove the parser interface scales. End state: a search box on the timeline that returns ranked turn snippets with click-through to session, a stats page showing token rollups, and a working opencode parser registered in the collector.
A successful end state for this task: I can type "tt bundle lockfile" into the search box, see ranked snippets across both Claude Code and opencode sessions on either machine, click into any result and land on the matching turn. The stats page shows a per-tool, per-day token bar for the last month.
In:
GET /api/v1/search?q=&tool=&host=&since=&until=&include_tool_outputs=&limit=&cursor= — FTS5 query against turns_fts. With include_tool_outputs=1, also union tool_outputs_fts and merge by BM25 rank, deduped on (tool, host, session_id, turn_id).GET /api/v1/stats?group_by=tool|host|day&since=&until= — token rollups (sum tokens_in, tokens_out, count of turns, count of sessions). Cost is summed only where non-NULL.GET /search?q=... — server-rendered results page with snippet/highlight, filters in URL.GET /stats — table view of the stats endpoint, simple and readable.#turn-<turn_id> and a "find on page" hint.internal/collector/parser/opencode/ — new parser implementing the same Parser interface from #2.docs/spikes/opencode-format.md before the parser is written; spike output is checked in.testdata/opencode/).Out:
Search query (default — include_tool_outputs=0). One FTS5 MATCH against turns_fts, filtered by the optional tool, host, since, until UNINDEXED columns:
SELECT
f.tool, f.host, f.session_id, f.turn_id, f.timestamp, f.role,
snippet(turns_fts, 0, '<mark>', '</mark>', '…', 32) AS snippet,
bm25(turns_fts) AS rank
FROM turns_fts AS f
WHERE turns_fts MATCH ?
AND (? IS NULL OR f.tool = ?)
AND (? IS NULL OR f.host = ?)
AND (? IS NULL OR f.timestamp >= ?)
AND (? IS NULL OR f.timestamp < ?)
ORDER BY rank LIMIT ?;
Pagination: cursor encodes (rank, turn_id) of the last row; next page applies (rank, turn_id) > cursor in lexicographic order. Offset-based pagination over FTS5 is correct but gets pathological at depth; a tuple cursor is barely more code. For the personal scale, offset+limit would also be fine — defaulting to cursor anyway because changing pagination later is an API break.
Search query (include_tool_outputs=1). Two MATCH queries (one per FTS table), UNION ALL, then a window dedupe on (tool, host, session_id, turn_id) keeping the better-ranked match (a turn might appear in both indexes). The frontend sets include_tool_outputs=1 only when the user toggles the "include tool output" checkbox; the default URL stays clean and fast.
Stats query. Standard SQL aggregation over turns, parameterized by group_by:
group_by=tool → GROUP BY tool returning per-tool totals.group_by=host → GROUP BY host.group_by=day → GROUP BY date(timestamp, 'unixepoch') returning a daily series.The idx_turns_timestamp index from #1 covers the date-range filter; for the small data scale this stays sub-second without further tuning.
Search HTML. A new search.html template sharing base.html with the timeline and session views. The result list shows: tool icon, host, working_dir, timestamp, snippet (with <mark>), session-link to /session/{tool}/{host}/{session_id}#turn-<turn_id>. Snippets pass through bluemonday UGC policy before insertion.
JS (vanilla, no framework). One small script (internal/ui/static/lethe.js):
/search?q=...&format=fragment, replace the results region's innerHTML. The format=fragment query param tells the server to return only the results partial template, no <html> chrome.#turn-<id> → scroll, briefly add a .highlighted CSS class).<details> from #1; no JS needed.Total JS budget: under 50 lines. CSP-safe; no eval.
opencode parser — discovery first.
The format is unknown until the spike runs. Step 0 of this task is cmd/lethe-spike-opencode/main.go (deleted before the task closes), which:
~/.local/share/opencode/ and ~/.config/opencode/ and ~/.cache/opencode/.Spike output goes to docs/spikes/opencode-format.md. Until that spike runs, the parser implementation below is provisional. Two likely shapes:
Parse() reads from a byte offset, returns wire.TurnEvents, persists offset.Parse() opens the source DB read-only with _busy_timeout set, queries rows newer than a stored marker (rowid or (session, sequence)), maps to wire.TurnEvent. The ingestion_state.last_offset column gets reused for the marker (it's an INTEGER — works for either rowid or byte offset).In both cases the Parser interface from #2 is unchanged. The opencode-specific code is contained in internal/collector/parser/opencode/.
If the spike reveals an opaque or encrypted format, opencode is dropped from this task and an issue is filed against opencode upstream. The task still ships search.
Tradeoffs that settled it.
Unknowns that remain.
k1=1.2, b=0.75) is fine for prose. If empirical search quality is bad against real data, tune via bm25(turns_fts, 5.0, 1.0, ...) per-column weights — that's a #3-and-a-half follow-up, not a blocker./api/v1/sessions/... and the timeline/session HTML are unchanged. The internal/shared/wire/ types are untouched. No schema changes.sources:. The Parser interface is unchanged. Existing claude-code config keeps working./api/v1/search rather than offset — costs one helper, prevents an API break later. Stats keeps since/until only (no pagination — the result set is intrinsically small).include_tool_outputs defaults to false — keeps default search latency low and result lists uncluttered. UI surfaces a checkbox; URL param drives behavior.?format=fragment returns the partial.cmd/lethe-spike-opencode/ deleted before task closes; spike findings preserved in docs/spikes/opencode-format.md. Spike code is throwaway; the writeup is the artifact.#turn-<turn_id> rather than scroll-into-view-via-JS — anchors are stable across renders, work with browser back/forward, and survive the page being reloaded from a bookmark.TDD: yes (reason: FTS query result shape is exactly the kind of thing that silently regresses on schema or trigger changes; stats aggregation has off-by-one risks around timezone/date boundaries; opencode parser determinism mirrors the claude-code parser's TDD justification.)
migrations/. This task adds no new migration files.internal/shared/wire/ types are not modified by this task. If a need arises, it's flagged and split into a separate amendment task.GET /api/v1/search and GET /api/v1/stats are read-only: they execute SELECT only, with no transaction lower than BEGIN DEFERRED./api/v1/search; time-ordered results are explicitly opt-in via a future ?order=time parameter (not implemented in this task; reserved).Tailscale-User-Login allowlist middleware as #1's existing routes. No exemptions.bluemonday before insertion.parser.Parser from #2 verbatim — no special interface, no opencode-specific hooks in the ingestion loop.docs/spikes/opencode-format.md and committed before the parser code lands.tool_outputs_fts is queried only when include_tool_outputs=1; default search path never touches it.?format=fragment partials, not a parallel JS render path.internal/collector/parser/<tool>/ plus a single register.go import — opencode's job is to prove this and leave the path obvious for crush/pi/kimi.