From e6173cab1e231b1f4128025ba53075b3b186e564 Mon Sep 17 00:00:00 2001 From: Eugene Blikh Date: Sun, 26 Apr 2026 07:53:41 +0300 Subject: [PATCH] docs: add roadmap TODO and lethe-web-ui-aggregates design TODO.md indexes all 9 task specs (4 done/in-progress, 5 deferred or blocked) plus the carry-overs from the foundation Conclusion. New task file lethe-web-ui-aggregates covers the backend /projects + /stats endpoints and the three frontend screens (Projects index, Project detail, Stats), replacing three of the foundation's stub routes. --- docs/TODO.md | 34 +++++++++ docs/tasks/lethe-web-ui-aggregates.md | 100 ++++++++++++++++++++++++++ 2 files changed, 134 insertions(+) create mode 100644 docs/TODO.md create mode 100644 docs/tasks/lethe-web-ui-aggregates.md diff --git a/docs/TODO.md b/docs/TODO.md new file mode 100644 index 0000000000000000000000000000000000000000..c75e8936b48bc7eb9f04ba61c35630edc62cb789 --- /dev/null +++ b/docs/TODO.md @@ -0,0 +1,34 @@ +# TODO — lethe roadmap + +Index of task specs and their state. Each row points at a `docs/tasks/.md` task file (when one exists) or describes what needs to be created. + +## Tasks + +| # | Slug | Status | Description | +|---|---|---|---| +| 1 | [`lethe-server`](tasks/lethe-server.md) | **Verified** | Backend skeleton: SQLite ingest, sessions list/detail, forward-auth, RFC 7807, deployable on phoebe behind Authelia. Shipped over 9 phases. | +| 2 | [`lethe-collector-claude-code`](tasks/lethe-collector-claude-code.md) | Designed (deferred) | Per-host systemd-user collector that tails `~/.claude/projects/*.jsonl` and POSTs normalized turns to ingest. Blocks #8 and #9. | +| 3 | [`lethe-search-and-opencode`](tasks/lethe-search-and-opencode.md) | Designed (deferred) | Adds `GET /api/v1/search` (FTS5) and an `opencode` collector. Blocks #7. | +| 4 | [`lethe-web-ui-foundation`](tasks/lethe-web-ui-foundation.md) | **Reviewed** | Vite/React/TS SPA, embed pipeline, shell + Home + Session views, palette skeleton, 5 stub routes. Plus `/sessions` aggregate fields. | +| 5 | [`lethe-web-ui-aggregates`](tasks/lethe-web-ui-aggregates.md) | **In progress** | Backend `/projects` + `/stats` endpoints, Projects index + Project detail + Stats screen. Replaces 3 of #4's stubs. | +| 6 | `lethe-web-ui-palette-savedsearch` | Not started | Full ⌘K palette (PROJECT/SESSION items), saved-searches table + UI, palette pulls from it. Small backend (1 table) + Settings → Saved searches. | +| 7 | `lethe-web-ui-search` | Blocked on #3 | Search route: turn-level results, FTS `` highlighting, save-search action. | +| 8 | `lethe-web-ui-settings-display` | Not started | Settings → Display: theme toggle (light/dark/system), density toggle, "show tool calls" toggle, persisted to localStorage. | +| 9 | `lethe-web-ui-health-sources` | Blocked on #2 (and ideally #3) | Settings → Sources (per-host-per-tool config table) + `/health` route (collector ingestion table, status pills, footer strip with backfill progress and last-error). | + +## Carry-overs from `lethe-web-ui-foundation` + +These were captured in `lethe-web-ui-foundation.md` Conclusion → Future work / Deferred. Pulled here so they're visible across tasks. + +- **Composite-id-in-URL** (cosmetic, 1-line fix): `SessionsTable` Link passes the full composite as `params.id`, producing URLs like `/session/T/H/T%2FH%2FID`. Fix in `web/src/features/home/SessionsTable.tsx` to pass the bare `session_id`. +- **Aggregates absent on `GET /sessions/{tool}/{host}/{id}`**: the new `summary`/`turn_count`/`tokens_*_total` fields appear as zero values on the Get path because the Get SQL is unchanged (Plan 1.2 said so). UI doesn't read them, but a future API consumer would. Fix is either extending the Get SQL or adding `,omitempty` to the four numeric tags. +- **Turn meta-line lacks timestamp**: the `Turn` TS interface from foundation Plan 6.1 omitted `timestamp`; meta line shows `# seq · model · tokens-in→tokens-out` only. Add `timestamp` to the interface + render. +- **`go test ./...` walks `web/node_modules/flatted/golang/pkg/flatted`**: stray Go package shipped inside an npm dep. Switch CI to `go test ./internal/...` or gitignore the path. +- **CI configuration**: neither `.github/workflows/` nor `.sourcecraft/ci.yml` exists. Pick one (sourcecraft.dev native preferred for personal repos) and wire `npm ci && npm run lint && npm run typecheck && npm test && npm run build` plus the existing Go job. + +## Notes + +- Task numbering is arbitrary order-of-appearance, not priority. +- "Not started" means a slot in this index without a `docs/tasks/.md` yet — invoking `/up:udesign ` creates one. +- All commits across the project are GPG-signed (`commit.gpgsign=true`). +- Sourcecraft.dev is the primary git host; mirror to GitHub is not currently configured. diff --git a/docs/tasks/lethe-web-ui-aggregates.md b/docs/tasks/lethe-web-ui-aggregates.md new file mode 100644 index 0000000000000000000000000000000000000000..5285704eac13514a75a8a9faf947eef3cd637862 --- /dev/null +++ b/docs/tasks/lethe-web-ui-aggregates.md @@ -0,0 +1,100 @@ +# lethe-web-ui-aggregates + +**Status:** Design +**Module:** `sourcecraft.dev/bigbes/lethe` +**Branch:** master +**Worktree:** none +**Parent RFC:** Personal AI Assistant Log Aggregator (2026-04-25) +**Design source:** `docs/design_handoff_assistant_log/` (Direction 4 — Dense Data UI) +**Sibling tasks:** +- `lethe-server.md` (#1, ✓ Verified) +- `lethe-web-ui-foundation.md` (#4, ✓ Reviewed) +- `lethe-collector-claude-code.md` (#2, deferred) +- `lethe-search-and-opencode.md` (#3, deferred) +- `lethe-web-ui-palette-savedsearch.md` (#6, deferred) +- `lethe-web-ui-search.md` (#7, blocked on #3) +- `lethe-web-ui-settings-display.md` (#8, deferred) +- `lethe-web-ui-health-sources.md` (#9, blocked on #2) + +## Design + +### Purpose + +Add the two backend aggregation endpoints (`/api/v1/projects`, `/api/v1/stats`) the foundation deferred, plus the three frontend screens that consume them: Projects index, Project detail, Stats. Replaces three of the foundation's stub routes (`/projects`, `/project/$`, `/stats`) with real content. + +**In scope**: Go aggregation queries; the new project URL pattern; chart primitives (stacked-bar, heatmap, horizontal-bar, hour-bar) inline in SVG; range/group-by sub-bar controls. +**Out of scope**: Search route, Health route, Settings → Display, palette PROJECT/SESSION items, saved searches. + +### Chosen approach + +**Backend — two endpoints, both additive**: + +- `GET /api/v1/projects?since=&limit=&offset=` returns rows of `{ cwd, sessions, turn_count, tokens_in_total, tokens_out_total, last_active, hosts[], tools[], top_tool }`. SQL groups by `working_dir` over the existing sessions+turns join, ordered by `MAX(ended_at) DESC`. Owner-scoped via existing auth middleware. +- `GET /api/v1/stats?range=7d|30d|90d|all` returns a single fat JSON response with six cards in one round-trip: + - `per_tool`: `{ tool, sessions, turns, tokens_in, tokens_out, daily_sparkline: number[N] }[]` for the requested range + - `daily`: `{ date_unix, per_tool: { [tool: string]: number /*turns*/ } }[]` over the last 60 days + - `heatmap`: `{ date_unix, count }[]` — 84 cells (12 weeks × 7 days), fixed window + - `top_cwd`: `{ cwd, count }[]` — top 20 cwds by turn count in the requested range + - `hour_of_day`: `{ hour, count }[]` — 24 buckets over the range + - `host_split`: `{ host, count }[]` over the range +- Both endpoints reuse the existing `apierror.Render` for error paths; both honor admin `?owner=` override exactly like sessions list. + +**Backend — one tiny extension to sessions list**: `GET /api/v1/sessions?cwd=` accepts a new optional filter on `working_dir`. Project detail page uses this to pull a cwd's sessions; no separate project-detail endpoint. Wire/schema unchanged. + +**Frontend**: +- `useProjects(filters)` — TanStack Query hook for `/api/v1/projects`. +- `useStats(range)` — TanStack Query hook for `/api/v1/stats`. +- Routes: + - `web/src/routes/projects.tsx` (replaces stub) — projects table. + - `web/src/routes/project.$.tsx` (new, `$` catch-all so cwd-with-slashes works as a single URL-encoded path segment) — header card + sessions table scoped to that cwd via the new `?cwd=` filter. + - `web/src/routes/stats.tsx` (replaces stub) — sub-bar with range pills (7d / 30d / 90d / all), 2-column card grid. +- New chart primitives (`web/src/primitives/`): `StackedBars` (60×N stacked-bar SVG), `Heatmap` (12×7 cell grid), `HorizontalBars` (top-cwd list), `HourBars` (24-bin column chart). `Spark` from the foundation is reused for the per-tool sparkline cards. All SVGs inline; no chart library. + +**Project identity**: exact `working_dir` string, no canonicalization. URL is `/project/$` with a single segment carrying the URL-encoded cwd. Tradeoff: `/Users/x/proj` and `~/proj` resolved-to-the-same-place show as different projects. Acceptable for v1; canonicalization is a known limitation. + +### Hands-off decisions + +- **udesign — stats endpoint = single fat JSON**. Six small queries server-side beat six round-trips; payloads are a few KB. +- **udesign — project identity = exact `working_dir`**. Simpler, matches prototype; canonicalization needs OS-level path resolution that's out of scope. +- **udesign — project detail uses `GET /api/v1/sessions?cwd=`** rather than a dedicated endpoint. Avoids duplicating the sessions DTO server-side. +- **udesign — range-pill semantics are unix-seconds offsets**. `7d` = `now - 7*86400`; `all` omits the `range` param. Mirrors the existing `since` pattern on sessions list. +- **udesign — `top_cwd` capped at 20**. Card has fixed height; prototype shows ~10 entries. +- **udesign — heatmap window is fixed at 12 weeks (84 cells), not range-dependent**. Matches the prototype. + +### Backwards-compatibility check + +All changes additive: +- Two new endpoints; no existing consumers. +- One new optional query param (`?cwd=`) on `/api/v1/sessions`; old clients ignore it and get the existing behavior. +- No schema migrations. +- No wire-type changes. +- Frontend stub routes are replaced; no deployed user is affected. + +### TDD: yes (scoped) + +- **Yes** for: each backend aggregate query (Go repository tests with seed data per card); the date-bucketing logic that fills missing days/hours with zeros (small Go helper, deterministic); the TS `adaptStats` adapter that pipes Go DTOs into the chart-friendly shapes. +- **No** for: SVG chart primitives — visual, no logic worth testing beyond what the rendered output shows. + +### Unknowns (resolve during execute) + +- **Catch-all `$` route compatibility with TanStack Router**: needs a quick spike — TanStack supports `$param` (segment) and `$` (full splat). Verify the splat captures slash-bearing cwds correctly and `Link` round-trips them. +- **Aggregate query plan at scale**: at 10–100k turns the GROUP BY's are fine on existing PKs. If `EXPLAIN QUERY PLAN` shows a scan on `turns` for the `top_cwd` query, add a covering index `(owner, timestamp)` in the same execute. Not blocking the phase. + +### Invariants + +- `/api/v1/projects` and `/api/v1/stats` reuse the existing auth middleware. No bypass, no special-casing. +- `?owner=` admin override behaves identically across `/sessions`, `/projects`, `/stats` (all-owners when set to `*`, specific owner when set to a username, default = caller's identity). +- No schema migrations; no `internal/shared/wire/` changes; no removed/renamed JSON fields. +- All new HTTP error paths render `application/problem+json` via the existing `apierror.Render`. +- Stats response is computed fresh per request (no server-side cache). Range parameter is the only knob; no per-tool/host filters at the endpoint level. +- Project identity is exact `working_dir`; the server never normalizes paths. +- All API calls in the SPA flow through TanStack Query + the existing `apiFetch` wrapper. No raw `fetch` outside `api/client.ts`. +- All chart primitives are inline `` in TSX; no chart library dependency added. + +### Principles + +- One round-trip per stats screen render. The frontend doesn't fan out to six endpoints; the server's GROUP BYs are cheap enough to bundle. +- Server returns numbers; the SPA formats and colors. No HTML, no pre-rendered SVG, no localized strings on the wire. +- Project detail = scoped sessions list. Resist building a parallel "project entity" model on either side. +- Reuse `Spark`, `Tag`, `ToolDot`, `EmptyState`, `Sub`, `SessionsTable` from the foundation. Add chart primitives only where none of these fits. +- No silent fallbacks. If a stats card's query returns zero rows for the requested range, the response includes that card as an empty array; the SPA renders an `EmptyState` per-card. No "0%" placeholders, no synthesized data.