package search

import (
	"strings"
	"unicode"
)

// Lang is a language the index has a stemming analyzer for. Every indexed
// document is labelled with exactly one — its dominant language — and its text
// is routed block by block into the matching analyzed field.
type Lang string

const (
	LangEN Lang = "en"
	LangRU Lang = "ru"
)

// DefaultLang is the label a document with too little text to classify gets,
// and the language a block falls back to when it is too short to classify on
// its own. English rather than Russian because the machine-generated half of
// this corpus — frontmatter keys, paths, identifiers, fenced code — is English
// whatever language the prose around it is written in.
const DefaultLang = LangEN

// ruLetterRatio is the share of a text's letters that must be Cyrillic for it
// to count as Russian.
//
// The threshold is deliberately far below one half. The two error directions
// are not symmetric in likelihood: an English block essentially never contains
// Cyrillic at all, while a Russian block in this corpus routinely carries a
// third or more Latin letters — identifiers, product names, and untranslated
// technical terms are written in Latin inside Russian prose. A 0.5 threshold
// would therefore misfile real Russian paragraphs as English, and misfile
// almost no English ones as Russian.
const ruLetterRatio = 0.35

// minDetectLetters is the least number of letters a text needs before its
// script mix is treated as evidence. Below it, a single Latin acronym in a
// Russian heading (or one Russian word in an English one) would decide the
// whole block, so the caller's fallback is used instead.
const minDetectLetters = 12

// Detect classifies a text by script. ok is false when the text carries too
// few letters to classify, in which case the caller supplies the fallback —
// Detect never guesses.
func Detect(text string) (lang Lang, ok bool) {
	var cyrillic, letters int
	for _, r := range text {
		if !unicode.IsLetter(r) {
			continue
		}
		letters++
		if unicode.Is(unicode.Cyrillic, r) {
			cyrillic++
		}
	}
	if letters < minDetectLetters {
		return "", false
	}
	if float64(cyrillic)/float64(letters) >= ruLetterRatio {
		return LangRU, true
	}
	return LangEN, true
}

// DetectIn classifies a text, falling back to a language when it is too short
// to classify on its own.
func DetectIn(text string, fallback Lang) Lang {
	if l, ok := Detect(text); ok {
		return l
	}
	return fallback
}

// route splits a text into its Russian and its English part, block by block,
// so that each half is stemmed by the analyzer that understands it.
//
// This is the part the design left open, and per-*document* routing — the
// obvious reading of "detect the language and write the matching field" — is
// not sufficient. The two analyzers pass each other's script through
// untouched: bleve's `ru` analyzer leaves "indexes" as "indexes" and its `en`
// analyzer leaves "индексы" as "индексы". Foreign-script terms therefore still
// match literally (which is why a single-analyzer index is not catastrophic),
// but they match *unstemmed*, so "index" does not find "indexes" and
// "документы" does not find "документ". A specification whose prose is Russian
// and whose examples, headings and quoted requirements are English is one
// document, and one label for it necessarily mangles one of its two halves.
//
// Routing per block costs nothing extra — the document is being walked anyway —
// and removes the failure entirely: each block lands in the field whose
// analyzer stems it, and a query is run against both fields. Duplicating the
// whole text into both fields would also fix the stemming, but it doubles the
// index and double-counts every document that matches in both fields, which
// biases ranking toward mixed documents for no reason related to relevance.
//
// The unit is a line, because that is the unit the text arrives in: doc's
// plain-text projection emits a newline after every block-level node and at
// every soft line break, and frontmatter search text is one "key: value" line
// per key. So a line here is a paragraph, a wrapped fragment of one, a heading,
// a table row, a list item or a line of code. A line too short to classify
// takes fallback, which is the document's dominant language — the language of
// the prose it sits inside.
func route(text string, fallback Lang) (ru, en string) {
	var ruB, enB strings.Builder
	for _, line := range strings.Split(text, "\n") {
		if strings.TrimSpace(line) == "" {
			continue
		}
		b := &enB
		if DetectIn(line, fallback) == LangRU {
			b = &ruB
		}
		if b.Len() > 0 {
			b.WriteByte('\n')
		}
		b.WriteString(line)
	}
	return ruB.String(), enB.String()
}