~bigbes/lethe

e6173cab1e231b1f4128025ba53075b3b186e564 — Eugene Blikh a month ago dd2e2ea
docs: add roadmap TODO and lethe-web-ui-aggregates design

TODO.md indexes all 9 task specs (4 done/in-progress, 5 deferred or
blocked) plus the carry-overs from the foundation Conclusion. New task
file lethe-web-ui-aggregates covers the backend /projects + /stats
endpoints and the three frontend screens (Projects index, Project
detail, Stats), replacing three of the foundation's stub routes.
2 files changed, 134 insertions(+), 0 deletions(-)

A docs/TODO.md
A docs/tasks/lethe-web-ui-aggregates.md
A docs/TODO.md => docs/TODO.md +34 -0
@@ 0,0 1,34 @@
# TODO — lethe roadmap

Index of task specs and their state. Each row points at a `docs/tasks/<slug>.md` task file (when one exists) or describes what needs to be created.

## Tasks

| # | Slug | Status | Description |
|---|---|---|---|
| 1 | [`lethe-server`](tasks/lethe-server.md) | **Verified** | Backend skeleton: SQLite ingest, sessions list/detail, forward-auth, RFC 7807, deployable on phoebe behind Authelia. Shipped over 9 phases. |
| 2 | [`lethe-collector-claude-code`](tasks/lethe-collector-claude-code.md) | Designed (deferred) | Per-host systemd-user collector that tails `~/.claude/projects/*.jsonl` and POSTs normalized turns to ingest. Blocks #8 and #9. |
| 3 | [`lethe-search-and-opencode`](tasks/lethe-search-and-opencode.md) | Designed (deferred) | Adds `GET /api/v1/search` (FTS5) and an `opencode` collector. Blocks #7. |
| 4 | [`lethe-web-ui-foundation`](tasks/lethe-web-ui-foundation.md) | **Reviewed** | Vite/React/TS SPA, embed pipeline, shell + Home + Session views, palette skeleton, 5 stub routes. Plus `/sessions` aggregate fields. |
| 5 | [`lethe-web-ui-aggregates`](tasks/lethe-web-ui-aggregates.md) | **In progress** | Backend `/projects` + `/stats` endpoints, Projects index + Project detail + Stats screen. Replaces 3 of #4's stubs. |
| 6 | `lethe-web-ui-palette-savedsearch` | Not started | Full ⌘K palette (PROJECT/SESSION items), saved-searches table + UI, palette pulls from it. Small backend (1 table) + Settings → Saved searches. |
| 7 | `lethe-web-ui-search` | Blocked on #3 | Search route: turn-level results, FTS `<mark>` highlighting, save-search action. |
| 8 | `lethe-web-ui-settings-display` | Not started | Settings → Display: theme toggle (light/dark/system), density toggle, "show tool calls" toggle, persisted to localStorage. |
| 9 | `lethe-web-ui-health-sources` | Blocked on #2 (and ideally #3) | Settings → Sources (per-host-per-tool config table) + `/health` route (collector ingestion table, status pills, footer strip with backfill progress and last-error). |

## Carry-overs from `lethe-web-ui-foundation`

These were captured in `lethe-web-ui-foundation.md` Conclusion → Future work / Deferred. Pulled here so they're visible across tasks.

- **Composite-id-in-URL** (cosmetic, 1-line fix): `SessionsTable` Link passes the full composite as `params.id`, producing URLs like `/session/T/H/T%2FH%2FID`. Fix in `web/src/features/home/SessionsTable.tsx` to pass the bare `session_id`.
- **Aggregates absent on `GET /sessions/{tool}/{host}/{id}`**: the new `summary`/`turn_count`/`tokens_*_total` fields appear as zero values on the Get path because the Get SQL is unchanged (Plan 1.2 said so). UI doesn't read them, but a future API consumer would. Fix is either extending the Get SQL or adding `,omitempty` to the four numeric tags.
- **Turn meta-line lacks timestamp**: the `Turn` TS interface from foundation Plan 6.1 omitted `timestamp`; meta line shows `# seq · model · tokens-in→tokens-out` only. Add `timestamp` to the interface + render.
- **`go test ./...` walks `web/node_modules/flatted/golang/pkg/flatted`**: stray Go package shipped inside an npm dep. Switch CI to `go test ./internal/...` or gitignore the path.
- **CI configuration**: neither `.github/workflows/` nor `.sourcecraft/ci.yml` exists. Pick one (sourcecraft.dev native preferred for personal repos) and wire `npm ci && npm run lint && npm run typecheck && npm test && npm run build` plus the existing Go job.

## Notes

- Task numbering is arbitrary order-of-appearance, not priority.
- "Not started" means a slot in this index without a `docs/tasks/<slug>.md` yet — invoking `/up:udesign <slug>` creates one.
- All commits across the project are GPG-signed (`commit.gpgsign=true`).
- Sourcecraft.dev is the primary git host; mirror to GitHub is not currently configured.

A docs/tasks/lethe-web-ui-aggregates.md => docs/tasks/lethe-web-ui-aggregates.md +100 -0
@@ 0,0 1,100 @@
# lethe-web-ui-aggregates

**Status:** Design
**Module:** `sourcecraft.dev/bigbes/lethe`
**Branch:** master
**Worktree:** none
**Parent RFC:** Personal AI Assistant Log Aggregator (2026-04-25)
**Design source:** `docs/design_handoff_assistant_log/` (Direction 4 — Dense Data UI)
**Sibling tasks:**
- `lethe-server.md` (#1, ✓ Verified)
- `lethe-web-ui-foundation.md` (#4, ✓ Reviewed)
- `lethe-collector-claude-code.md` (#2, deferred)
- `lethe-search-and-opencode.md` (#3, deferred)
- `lethe-web-ui-palette-savedsearch.md` (#6, deferred)
- `lethe-web-ui-search.md` (#7, blocked on #3)
- `lethe-web-ui-settings-display.md` (#8, deferred)
- `lethe-web-ui-health-sources.md` (#9, blocked on #2)

## Design

### Purpose

Add the two backend aggregation endpoints (`/api/v1/projects`, `/api/v1/stats`) the foundation deferred, plus the three frontend screens that consume them: Projects index, Project detail, Stats. Replaces three of the foundation's stub routes (`/projects`, `/project/$`, `/stats`) with real content.

**In scope**: Go aggregation queries; the new project URL pattern; chart primitives (stacked-bar, heatmap, horizontal-bar, hour-bar) inline in SVG; range/group-by sub-bar controls.
**Out of scope**: Search route, Health route, Settings → Display, palette PROJECT/SESSION items, saved searches.

### Chosen approach

**Backend — two endpoints, both additive**:

- `GET /api/v1/projects?since=<unix>&limit=&offset=` returns rows of `{ cwd, sessions, turn_count, tokens_in_total, tokens_out_total, last_active, hosts[], tools[], top_tool }`. SQL groups by `working_dir` over the existing sessions+turns join, ordered by `MAX(ended_at) DESC`. Owner-scoped via existing auth middleware.
- `GET /api/v1/stats?range=7d|30d|90d|all` returns a single fat JSON response with six cards in one round-trip:
  - `per_tool`: `{ tool, sessions, turns, tokens_in, tokens_out, daily_sparkline: number[N] }[]` for the requested range
  - `daily`: `{ date_unix, per_tool: { [tool: string]: number /*turns*/ } }[]` over the last 60 days
  - `heatmap`: `{ date_unix, count }[]` — 84 cells (12 weeks × 7 days), fixed window
  - `top_cwd`: `{ cwd, count }[]` — top 20 cwds by turn count in the requested range
  - `hour_of_day`: `{ hour, count }[]` — 24 buckets over the range
  - `host_split`: `{ host, count }[]` over the range
- Both endpoints reuse the existing `apierror.Render` for error paths; both honor admin `?owner=` override exactly like sessions list.

**Backend — one tiny extension to sessions list**: `GET /api/v1/sessions?cwd=<exact>` accepts a new optional filter on `working_dir`. Project detail page uses this to pull a cwd's sessions; no separate project-detail endpoint. Wire/schema unchanged.

**Frontend**:
- `useProjects(filters)` — TanStack Query hook for `/api/v1/projects`.
- `useStats(range)` — TanStack Query hook for `/api/v1/stats`.
- Routes:
  - `web/src/routes/projects.tsx` (replaces stub) — projects table.
  - `web/src/routes/project.$.tsx` (new, `$` catch-all so cwd-with-slashes works as a single URL-encoded path segment) — header card + sessions table scoped to that cwd via the new `?cwd=` filter.
  - `web/src/routes/stats.tsx` (replaces stub) — sub-bar with range pills (7d / 30d / 90d / all), 2-column card grid.
- New chart primitives (`web/src/primitives/`): `StackedBars` (60×N stacked-bar SVG), `Heatmap` (12×7 cell grid), `HorizontalBars` (top-cwd list), `HourBars` (24-bin column chart). `Spark` from the foundation is reused for the per-tool sparkline cards. All SVGs inline; no chart library.

**Project identity**: exact `working_dir` string, no canonicalization. URL is `/project/$` with a single segment carrying the URL-encoded cwd. Tradeoff: `/Users/x/proj` and `~/proj` resolved-to-the-same-place show as different projects. Acceptable for v1; canonicalization is a known limitation.

### Hands-off decisions

- **udesign — stats endpoint = single fat JSON**. Six small queries server-side beat six round-trips; payloads are a few KB.
- **udesign — project identity = exact `working_dir`**. Simpler, matches prototype; canonicalization needs OS-level path resolution that's out of scope.
- **udesign — project detail uses `GET /api/v1/sessions?cwd=`** rather than a dedicated endpoint. Avoids duplicating the sessions DTO server-side.
- **udesign — range-pill semantics are unix-seconds offsets**. `7d` = `now - 7*86400`; `all` omits the `range` param. Mirrors the existing `since` pattern on sessions list.
- **udesign — `top_cwd` capped at 20**. Card has fixed height; prototype shows ~10 entries.
- **udesign — heatmap window is fixed at 12 weeks (84 cells), not range-dependent**. Matches the prototype.

### Backwards-compatibility check

All changes additive:
- Two new endpoints; no existing consumers.
- One new optional query param (`?cwd=`) on `/api/v1/sessions`; old clients ignore it and get the existing behavior.
- No schema migrations.
- No wire-type changes.
- Frontend stub routes are replaced; no deployed user is affected.

### TDD: yes (scoped)

- **Yes** for: each backend aggregate query (Go repository tests with seed data per card); the date-bucketing logic that fills missing days/hours with zeros (small Go helper, deterministic); the TS `adaptStats` adapter that pipes Go DTOs into the chart-friendly shapes.
- **No** for: SVG chart primitives — visual, no logic worth testing beyond what the rendered output shows.

### Unknowns (resolve during execute)

- **Catch-all `$` route compatibility with TanStack Router**: needs a quick spike — TanStack supports `$param` (segment) and `$` (full splat). Verify the splat captures slash-bearing cwds correctly and `Link` round-trips them.
- **Aggregate query plan at scale**: at 10–100k turns the GROUP BY's are fine on existing PKs. If `EXPLAIN QUERY PLAN` shows a scan on `turns` for the `top_cwd` query, add a covering index `(owner, timestamp)` in the same execute. Not blocking the phase.

### Invariants

- `/api/v1/projects` and `/api/v1/stats` reuse the existing auth middleware. No bypass, no special-casing.
- `?owner=` admin override behaves identically across `/sessions`, `/projects`, `/stats` (all-owners when set to `*`, specific owner when set to a username, default = caller's identity).
- No schema migrations; no `internal/shared/wire/` changes; no removed/renamed JSON fields.
- All new HTTP error paths render `application/problem+json` via the existing `apierror.Render`.
- Stats response is computed fresh per request (no server-side cache). Range parameter is the only knob; no per-tool/host filters at the endpoint level.
- Project identity is exact `working_dir`; the server never normalizes paths.
- All API calls in the SPA flow through TanStack Query + the existing `apiFetch` wrapper. No raw `fetch` outside `api/client.ts`.
- All chart primitives are inline `<svg>` in TSX; no chart library dependency added.

### Principles

- One round-trip per stats screen render. The frontend doesn't fan out to six endpoints; the server's GROUP BYs are cheap enough to bundle.
- Server returns numbers; the SPA formats and colors. No HTML, no pre-rendered SVG, no localized strings on the wire.
- Project detail = scoped sessions list. Resist building a parallel "project entity" model on either side.
- Reuse `Spark`, `Tag`, `ToolDot`, `EmptyState`, `Sub`, `SessionsTable` from the foundation. Add chart primitives only where none of these fits.
- No silent fallbacks. If a stats card's query returns zero rows for the requested range, the response includes that card as an empty array; the SPA renders an `EmptyState` per-card. No "0%" placeholders, no synthesized data.