From 54fac8ddcf7a3232e2335300db465a6244a1b4be Mon Sep 17 00:00:00 2001 From: bigbes Date: Wed, 22 Jul 2026 14:24:05 +0300 Subject: [PATCH] docs: resolve bilingual search with per-line routing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per-document language routing, the option this document suggested, was implemented and measured as insufficient: a dominantly Russian spec quoting English verbatim matches attachments but not attachment. A single analyzer does not break the other language, it silently stops stemming it, which is why the flaw survives casual testing. Per-line routing into ru and en title/body fields, queried across all four, finds both halves. Dual-writing was rejected despite also fixing stemming, because it biases ranking toward mixed documents by summing two field scores. The threshold is 35 percent Cyrillic rather than 50, since Russian technical prose here routinely runs a third Latin. Also records that filters must be exact-match — warren filtered on an analyzed field, which at our scale means a filter for one space can return another's documents, and a project is defined as a space filter — and corrects post-receive's unit of freshness to a space at a revision. --- docs/DESIGN.md | 70 +++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 61 insertions(+), 9 deletions(-) diff --git a/docs/DESIGN.md b/docs/DESIGN.md index 7f4b98fb4b908b987a353387a95efe5292b926af..61fc06628bd36462c53daeb80738c1f7b1c7d408 100644 --- a/docs/DESIGN.md +++ b/docs/DESIGN.md @@ -263,7 +263,11 @@ ignored. Rejection must therefore happen earlier: find weeks later. The escape hatch exists so that a hook bug or a bad schema can never lock you out of your own repository. - **`post-receive`** (after the fact, cannot reject) — notify the daemon so it - reindexes the changed documents and updates the space's index rev stamp. + rebuilds that **space** at its new revision and updates the index rev stamp. + Note the unit: a space at a revision, not a set of changed documents. An + earlier draft said "reindexes the changed documents", which contradicts the + decision to absorb the batch rebuilder — and nothing tracks per-document change, + because `index_stamp` records exactly a space and a rev. #### The hook is service code, not a shell script @@ -514,6 +518,59 @@ covers the approved branch and `proposals/*` and was silent on tags and other branches. Silence would mean "allowed", and a ref nothing ever reads is a place for content to rot unnoticed. Loosening this later is one branch in a switch. +## Bilingual search: per-line, not per-document + +Specs here are written in Russian and English, often within one document (cf. the +`ru-spec-style` skill). The design's suggested fix — detect each document's +language and route it to a `ru`- or `en`-analyzed field — was implemented, +measured, and **found insufficient**. The resolution is per-*line* routing into +`title_ru`/`title_en` and `body_ru`/`body_en`, queried across all four. Each +document still carries one dominant-language label, used for reporting and +snippet selection, never for placing text. + +The measurements that settled it, each a committed test: + +1. **A single analyzer does not *break* the other language — it silently stops + stemming it.** The `ru` analyzer leaves `"indexes rebuild all the documents"` + entirely unstemmed (and keeps English stop words); the `en` analyzer does the + same to Russian. Foreign-script terms still match *literally*, which is why + this looks fine in casual testing. The failure is subtler and worse: singular + stops finding plural. +2. **Per-document routing loses the minority language outright.** A dominantly + Russian spec quoting English requirements verbatim, indexed whole into + `body_ru`: searching `attachments` finds it, searching `attachment` does not. + That is exactly the document shape this corpus has. +3. **Per-line routing finds both halves**, with neighbouring single-language + documents unaffected. + +**Dual-writing the whole text into both fields was rejected**, although it also +fixes stemming: it doubles the index and makes every mixed document match in two +fields at once, summing both scores in the disjunction — a systematic ranking +bias toward mixed documents that has nothing to do with relevance. + +**The threshold is 35% Cyrillic, not 50%**, requiring at least 12 letters to +decide at all, falling back to the document's dominant language otherwise. The +error directions are not symmetric: English prose has essentially no Cyrillic, +whereas Russian technical prose here routinely runs a third Latin characters — +identifiers, product names, untranslated terms. + +Residual limitation, stated plainly: a single *line* mixing both languages goes +wholly to one field. Fixing that needs token-level routing and is not worth it. + +**Filters must be exact-match, and warren's were not.** warren filtered by +section using a match query on an *analyzed* field. At this scale that is a +correctness bug rather than a wart: a project **is** a space filter, so an +analyzed `~bigbes/home-ops` tokenizes and a filter for one space can return +another's documents. `space`, `section` and `lang` are keyword-analyzed and +matched as terms. + +**Measured rebuild cost** (1000 synthetic specs across 5 spaces, ~3 KB each): +full rebuild **284 ms**, single space of 200 documents **58 ms**, queries +**0.4 ms**. At roughly 30× the confirmed daily volume, the "absorb the batch +rebuild, do not build incremental indexing" decision holds by a wide margin — +and rebuild duration is instrumented, so the decision to revisit it will be +measured rather than guessed. + ## Consistency and recovery Three systems are touched by a merge — git refs, the bleve index, and Postgres — @@ -1210,14 +1267,9 @@ degrading, which is the point of listing them. - **MCP transport.** Streamable HTTP on the same chi router (`/mcp`) keeps it to one listener and one nginx block; a second port is only needed if MCP ends up wanting different timeouts than the web UI. -- **Mixed Russian/English search — partially solved, not finished.** warren's - `search/keyword.go` already wires both the `lang/en` and `lang/ru` analyzers - with per-index selection, so the starting point is better than an earlier draft - claimed. What is still missing is **per-document** language routing: specs here - are written in both (cf. the `ru-spec-style` skill), and one analyzer per index - mangles whichever language it was not chosen for. Options: detect language at - index time and write to `ru`/`en` fields, querying both; or accept degraded - stemming on the minority language. +- ~~**Mixed Russian/English search.**~~ **Resolved — see "Bilingual search" + below.** Per-*document* routing was measured and found insufficient; the answer + is per-*line* routing. - **Attachments and binaries — now a concrete gap, not a hypothetical.** The Space section says a space holds "markdown documents, plus attachments", but the git layer enumerates and accepts **`.md` only**, so `![[diagram.png]]` has