2026-09-05

A rendering bug that only real English input could expose

A record of a real discover → locate → fix → verify cycle, not a retrospective written after the fact — every before/after output in this post comes from the same real fix.

Discovery: exposed while writing the English README demo

retrieval/ranker.py's _relation_to_sentence() renders a (subject, predicate, object) relation into a natural-language sentence, assembled into context for the LLM. This code was written assuming Chinese input by default:

return f"{subject_name}{relation.predicate}{object_name}（记录于{recorded}）。"

In Chinese, words aren't separated by spaces, so concatenating directly reads as a normal sentence. This code ran fine in Chinese-language use for a long time — until writing an English demo for the README and running the full pipeline against real English input for the first time, which produced this:

Idoes AI research atCAS（记录于2026-09-05 10:34:29）。

"I" and "does" ran together into "Idoes," "at" and "CAS" ran together into "atCAS," and a Chinese-language timestamp label was still tacked onto the end — regardless of the input language, the template always concatenated the Chinese way. This wasn't an occasional display glitch; it was the inevitable result of a template that only ever considered CJK input from the moment it was written — Chinese-language use just never exercised the English path, so it never surfaced.

Locating it

Following _relation_to_sentence() upward showed the problem wasn't confined to rendering: the triple-extraction system prompt in llm/openai_compatible.py was also written entirely in Chinese, with no explicit instruction to "preserve the original language, don't translate." Even with rendering fixed, extraction itself had no language-agnostic guarantee for English input. Rendering and extraction were two symptoms of the same root cause — the entire pipeline was Chinese-first by design, with no explicit language check anywhere.

The Entity/Relation models also had no language field to query — extraction never recorded which language a triple was in, so rendering had no such information available either. The only fix was to detect language directly from the assembled text at render time, rather than relying on a language tag that didn't exist.

The fix

Added a Unicode-codepoint-range-based CJK detector in ranker.py:

_CJK_RANGES = (
    (0x4E00, 0x9FFF),  # CJK Unified Ideographs
    (0x3400, 0x4DBF),  # CJK Unified Ideographs Extension A
    (0x3000, 0x303F),  # CJK punctuation
    (0xFF00, 0xFFEF),  # halfwidth/fullwidth forms
)

def _is_cjk_text(text: str) -> bool:
    return any(
        any(start <= ord(ch) <= end for start, end in _CJK_RANGES) for ch in text
    )

_relation_to_sentence() now checks this before assembling the sentence: Chinese takes the original no-space concatenation with a Chinese timestamp label, anything else takes space-separated concatenation with an English timestamp label:

if _is_cjk_text(f"{subject_name}{relation.predicate}{object_name}"):
    return f"{subject_name}{relation.predicate}{object_name}（记录于{recorded}）。"
return f"{subject_name} {relation.predicate} {object_name} (recorded at {recorded})."

The extraction system prompt in openai_compatible.py was also updated to explicitly require that extracted subject/predicate/object "preserve the same language as the original text, don't translate," and the existing "fold date into predicate" technique got an English example added alongside the existing Chinese one.

Verification

After the fix, a real DeepSeek call against the same English input turned the README demo's output from:

Idoes AI research atCAS（记录于2026-09-05 10:34:29）。Idoes AI research inPython（记录于2026-09-05 10:34:29）。

into:

I do AI research at CAS (recorded at 2026-09-05 10:40:00).I do AI research mostly in Python (recorded at 2026-09-05 10:40:00).

At the unit-test level, tests/test_ranker.py gained a dedicated regression test covering English concatenation (test_build_context_spaces_english_sentences_instead_of_running_words_together), and the existing dedup/boundary test that used single-letter Latin names (a/b/c) was switched to CJK names, so that test keeps testing only dedup/boundaries rather than getting entangled with the new language-aware concatenation logic. The README's prior "known limitations" note admitting that "retrieval context rendering and triple extraction currently favor Chinese" was also removed after this fix — it's no longer a known defect that needs disclosing.

The more general lesson

This bug survived undetected for as long as it did because tests and demos only ever used Chinese input — under Chinese input, no-space concatenation happens to be correct behavior, which masked the template's underlying assumption that input is always CJK. Only actually running the full end-to-end pipeline in a different language exposed this kind of overfitting to one input distribution — which is also why this fix didn't stop at "the code logic now makes sense": it insisted on verifying the output against a real DeepSeek API call, rather than confirming the fix by code review alone.