Image OCR text extraction for HTML(and other?) files

[Vit-Mrkvica]

Hi - my team uses docling for one project and we would like to use it for another one - there is a feature that would help us greatly which is if the extraction process instead of image description would have an option to pass OCR extracted text (from the particular image) into the output instead.

We are thinking of contributing, wanted to discuss if similar features are already planned?

[wittjeff]

Hi @Vit-Mrkvica — not a maintainer, but I've been contributing in this area and looked into what this would take, since the same capability is being asked for in #1985 (DOCX) and #1888 (Markdown). Short version: the architecture has evolved since June in a way that makes this a well-scoped contribution, and I'd suggest framing it slightly differently than "OCR text instead of the description."

Where things stand today

• ConvertPipeline now runs a common enrichment pipe on all backends — HTML, DOCX, PPTX, Markdown, not just PDF. Picture classification, picture description, and chart extraction already work this way.
• BaseItemAndImageEnrichmentModel.prepare_element already handles documents without page images by reading the embedded image straight off the PictureItem (docling/models/base_model.py, "Allow the case of documents without page images but embedded images (e.g. Word and HTML docs)").
• What's missing is exactly one thing: none of the existing OCR engines (docling/models/stages/ocr/) is wrapped as an enrichment model — they're all page-level PDF-pipeline stages.

So the feature is: a new enrichment stage that runs a standard OCR engine on each PictureItem image, on every backend.

Suggested shape: sibling field, not replacement

Rather than replacing the description, I'd propose storing OCR text as a sibling of it. PictureMeta in docling-core is already a struct of independent optional fields (description, classification, molecule, tabular_chart, code, …), so a picture can carry both a transcription and a description. This mirrors a long-standing distinction in ISO 32000 / PDF/UA: /ActualText (what the content is — character-for-character replacement text, the right home for OCR output) versus /Alt (what the content depicts — a description). They legitimately coexist on the same element, and accessibility workflows need both. "OCR instead of description" then becomes a serializer preference, not a data-model constraint — which is both semantically cleaner and easier to land.

Concrete plan

docling-core:

1. Add an ocr_text field to PictureMeta (e.g. OcrMetaField carrying the text plus provenance: engine name/version, optionally language and confidence), alongside description.
2. Serializer support so it reaches the output: the meta-serializer hook (serialize_meta / meta_serializer) already exists; add markdown/HTML/DocTags handling plus a serializer param controlling precedence when both ocr_text and description are present (emit both, OCR-only, description-only).

docling:
3. New PictureOcrEnrichmentModel(BaseItemAndImageEnrichmentModel) under docling/models/stages/picture_ocr/, instantiating an engine through the existing OCR factory so tesseract/easyocr/rapidocr/ocrmac/etc. all work unchanged. Shape mirrors PictureDescriptionBaseModel (batching, scale, min-size threshold to skip icons/decorations).
4. do_picture_ocr: bool = False plus a picture_ocr_options block on ConvertPipelineOptions, wired into the common enrichment pipe — so it works for HTML, DOCX, PPTX, Markdown, and image inputs with one switch. CLI flag to match.
5. PDF nuance: the PDF pipeline's page-level OCR can already extract text inside picture regions, so for PDFs this enrichment should either default off or be documented as potentially overlapping (see also PR #3686 about OCR text in picture clusters during page assembly). For declarative backends there's no overlap — nothing OCRs those images today.
6. Tests: HTML and DOCX fixtures containing images-of-text, asserting meta.ocr_text is populated and appears in markdown output; plus a docs example.

Happy to help review, or to collaborate on the docling-core piece if the maintainers agree with the field addition. Maintainers — does this direction fit your plans for the meta/serializer model?