1 files changed, 61 insertions(+), 9 deletions(-)

M docs/DESIGN.md

@@ 263,7 263,11 @@ ignored. Rejection must therefore happen earlier:
   find weeks later. The escape hatch exists so that a hook bug or a bad schema
   can never lock you out of your own repository.
 - **`post-receive`** (after the fact, cannot reject) — notify the daemon so it
-  reindexes the changed documents and updates the space's index rev stamp.
+  rebuilds that **space** at its new revision and updates the index rev stamp.
+  Note the unit: a space at a revision, not a set of changed documents. An
+  earlier draft said "reindexes the changed documents", which contradicts the
+  decision to absorb the batch rebuilder — and nothing tracks per-document change,
+  because `index_stamp` records exactly a space and a rev.
 
 #### The hook is service code, not a shell script
 


@@ 514,6 518,59 @@ covers the approved branch and `proposals/*` and was silent on tags and other
 branches. Silence would mean "allowed", and a ref nothing ever reads is a place
 for content to rot unnoticed. Loosening this later is one branch in a switch.
 
+## Bilingual search: per-line, not per-document
+
+Specs here are written in Russian and English, often within one document (cf. the
+`ru-spec-style` skill). The design's suggested fix — detect each document's
+language and route it to a `ru`- or `en`-analyzed field — was implemented,
+measured, and **found insufficient**. The resolution is per-*line* routing into
+`title_ru`/`title_en` and `body_ru`/`body_en`, queried across all four. Each
+document still carries one dominant-language label, used for reporting and
+snippet selection, never for placing text.
+
+The measurements that settled it, each a committed test:
+
+1. **A single analyzer does not *break* the other language — it silently stops
+   stemming it.** The `ru` analyzer leaves `"indexes rebuild all the documents"`
+   entirely unstemmed (and keeps English stop words); the `en` analyzer does the
+   same to Russian. Foreign-script terms still match *literally*, which is why
+   this looks fine in casual testing. The failure is subtler and worse: singular
+   stops finding plural.
+2. **Per-document routing loses the minority language outright.** A dominantly
+   Russian spec quoting English requirements verbatim, indexed whole into
+   `body_ru`: searching `attachments` finds it, searching `attachment` does not.
+   That is exactly the document shape this corpus has.
+3. **Per-line routing finds both halves**, with neighbouring single-language
+   documents unaffected.
+
+**Dual-writing the whole text into both fields was rejected**, although it also
+fixes stemming: it doubles the index and makes every mixed document match in two
+fields at once, summing both scores in the disjunction — a systematic ranking
+bias toward mixed documents that has nothing to do with relevance.
+
+**The threshold is 35% Cyrillic, not 50%**, requiring at least 12 letters to
+decide at all, falling back to the document's dominant language otherwise. The
+error directions are not symmetric: English prose has essentially no Cyrillic,
+whereas Russian technical prose here routinely runs a third Latin characters —
+identifiers, product names, untranslated terms.
+
+Residual limitation, stated plainly: a single *line* mixing both languages goes
+wholly to one field. Fixing that needs token-level routing and is not worth it.
+
+**Filters must be exact-match, and warren's were not.** warren filtered by
+section using a match query on an *analyzed* field. At this scale that is a
+correctness bug rather than a wart: a project **is** a space filter, so an
+analyzed `~bigbes/home-ops` tokenizes and a filter for one space can return
+another's documents. `space`, `section` and `lang` are keyword-analyzed and
+matched as terms.
+
+**Measured rebuild cost** (1000 synthetic specs across 5 spaces, ~3 KB each):
+full rebuild **284 ms**, single space of 200 documents **58 ms**, queries
+**0.4 ms**. At roughly 30× the confirmed daily volume, the "absorb the batch
+rebuild, do not build incremental indexing" decision holds by a wide margin —
+and rebuild duration is instrumented, so the decision to revisit it will be
+measured rather than guessed.
+
 ## Consistency and recovery
 
 Three systems are touched by a merge — git refs, the bleve index, and Postgres —


@@ 1210,14 1267,9 @@ degrading, which is the point of listing them.
 - **MCP transport.** Streamable HTTP on the same chi router (`/mcp`) keeps it to
   one listener and one nginx block; a second port is only needed if MCP ends up
   wanting different timeouts than the web UI.
-- **Mixed Russian/English search — partially solved, not finished.** warren's
-  `search/keyword.go` already wires both the `lang/en` and `lang/ru` analyzers
-  with per-index selection, so the starting point is better than an earlier draft
-  claimed. What is still missing is **per-document** language routing: specs here
-  are written in both (cf. the `ru-spec-style` skill), and one analyzer per index
-  mangles whichever language it was not chosen for. Options: detect language at
-  index time and write to `ru`/`en` fields, querying both; or accept degraded
-  stemming on the minority language.
+- ~~**Mixed Russian/English search.**~~ **Resolved — see "Bilingual search"
+  below.** Per-*document* routing was measured and found insufficient; the answer
+  is per-*line* routing.
 - **Attachments and binaries — now a concrete gap, not a hypothetical.** The
   Space section says a space holds "markdown documents, plus attachments", but
   the git layer enumerates and accepts **`.md` only**, so `![[diagram.png]]` has