Proposal: Standardize OCR Language Codes Using BCP-47

[nikos-livathinos]

Updated after feedback from @Ahmetcanyvz

Docling supports multiple OCR engines (RapidOCR, nemotron-ocr, EasyOCR, Tesseract, ocrmac), each using a different notation for the same languages. Users must learn which notation applies to each engine/backend combination, code cannot be written portably across engines, and it silently breaks when engine or model versions change.

Current Issues

1. The same language, five notations. Simplified Chinese:

Engine	Code	Note
Tesseract	chi_sim	ISO 639-2/B chi plus a suffix
EasyOCR	ch_sim
RapidOCR	ch or chinese	varies by PP-OCR version
ocrmac	zh-Hans	already BCP-47 language-REGION
nemotron-ocr	multilingual	no per-language codes at all

Any standard we adopt has to accommodate both ends of that range. Docling's current normalization layer is two entries (rapid_ocr_model.py):

_DOCLING_LANG_NORMALIZE: dict[str, str] = {"chinese": "ch", "english": "en"}

2. Notation varies within an engine, by backend and model version. RapidOCR alone:

• PP-OCRv6 (onnxruntime/openvino/paddle/torch) — ~52 codes: ch, en, de, fr, japan, …
• PP-OCRv5 (onnxruntime/openvino/paddle) — 12: arabic, ch, cyrillic, devanagari, el, en, eslav, korean, latin, ta, te, th
• PP-OCRv4 (torch only) — 8: arabic, cyrillic, devanagari, ka, korean, latin, ta, te

3. Script-based vs. language-based systems. PP-OCRv5's cyrillic covers 34 languages and latin covers many more, while PP-OCRv6 has per-language codes (de, fr). The user must know which kind of code an engine expects.

4. Codes change with model versions. v4→v5 added el, eslav, th; v5→v6 went from ~12 to ~52 codes and introduced aliases (zh→ch, ja→japan, ko→korean). Users track this across dependencies.

5. Language alone does not identify an OCR model. OCR consumes a language rendered in a writing system, and the two vary independently: Serbian is written in both Cyrillic and Latin; Chinese Simplified vs. Traditional is not a language distinction at all, so no ISO 639 code expresses it; Punjabi is Gurmukhi in India and Shahmukhi in Pakistan (likewise Uzbek, Kazakh, Kurdish). Plain ISO 639 cannot say "Serbian, Cyrillic", so it is not sufficient on its own.

Proposed Solution

Decouple Docling's language notation from engine-specific implementations by adopting BCP-47 as the standard.

BCP-47 (RFC 5646) composes the ISO registries into one tag — language[-script][-region], e.g. en, zh-Hans, sr-Cyrl, yue. The primary subtag is ISO 639-1 where a two-letter code exists and ISO 639-3 otherwise, giving full 639-3 coverage while avoiding ISO 639-2's bibliographic/terminological duplicates (ger/deu, chi/zho).

1. User-facing standardization. The CLI and API accept BCP-47 tags. Script variants use the script subtag (zh-Hans, sr-Latn), which states what is meant; region tags (zh-CN, zh-TW) are also accepted as lossy-but-valid, since the region implies the script (zh-SG → zh-Hans too) — so ocrmac's existing ["en-US", "fr-FR"] style keeps working unchanged.

2. Internal mapping layer. Per-engine adapters translate to native notation, versioned per model release. Incoming tags are canonicalized (deu, ger and de are all registry-valid, so only canonicalization collapses them) and keyed on (language, script) rather than the tag string — the region is dropped once it has inferred the script:

de, de-DE, deu, ger  ->  (de, Latn)   # one key, one model
zh, zh-CN, zh-Hans   ->  (zh, Hans)
zh-TW, zh-Hant       ->  (zh, Hant)
sr                   ->  (sr, Cyrl)   # same language, different key
sr-Latn              ->  (sr, Latn)

BCP-47 also reserves three non-language subtags we can adopt directly: und (undetermined) for script auto-detection, replacing today's "auto" sentinel; mul (multiple languages) for an engine's broadest multilingual model, which is what nemotron-ocr's "multilingual" already means; and zxx (no linguistic content) to skip OCR.

3. Script handling. Engines that expose script families rather than languages (RapidOCR PP-OCRv5's cyrillic, Tesseract's script/Cyrillic) need no separate vocabulary from the user: the script is derived from the tag, so "ru" resolves to the Cyrillic model on its own. Macrolanguages fall out the same way — cmn and yue resolve under zh, so zh means "any Chinese language in this script", which is the right granularity for OCR.

4. Implementation. langcodes provides parsing, canonicalization, likely-subtags and RFC 4647 matching in 1.0 MB with zero dependencies (the 16 MB langcodes[data] extra is only for human-readable names, which we don't need). Legacy engine-native codes (ch_sim, chi_sim, chinese) keep working with a DeprecationWarning through a transition period.

Example Usage

# Current - one language, three spellings
RapidOcrOptions(lang=["ch"])
EasyOcrOptions(lang=["ch_sim"])
TesseractOcrOptions(lang=["chi_sim"])

# Proposed - the same tag, whichever engine is selected
RapidOcrOptions(lang=["zh-Hans"])      # -> "ch"      (PP-OCRv6)
EasyOcrOptions(lang=["zh-Hans"])       # -> "ch_sim"
TesseractOcrOptions(lang=["zh-Hans"])  # -> "chi_sim"

Only the options class changes. The language tag no longer does.

Question for Discussion

Multi-language behavior: should we standardize how engines handle multi-language requests — warn when an engine uses only the first language, error when it has no multi-language support, or auto-select the best engine for the combination?

BCP-47 gives us mul for "explicitly multilingual" but does not decide the policy, and engines diverge sharply: ocrmac takes an ordered preference list, Tesseract joins with +, nemotron-ocr collapses everything into one "multilingual" model. This is the main open question — community input most valuable here.

---

Looking forward to community feedback on this proposal! 🚀

References

• BCP 47 / RFC 5646 — Tags for Identifying Languages
• RFC 4647 — Matching of Language Tags
• IANA Language Subtag Registry
• CLDR — Likely Subtags and Language Matching (UTS #35)
• langcodes (Python implementation)
• ISO 639 Language Codes
• Docling OCR Documentation
• RapidOCR Language Support
• EasyOCR Language List
• Tesseract Language Data