Add PaddleOCR-VL external parser support - #3376
Draft
ouyangfeng2022 wants to merge 39 commits into
Draft
Conversation
added 15 commits
July 8, 2026 10:45
- Add PaddleOCR-VL external parser implementation - Update environment configuration with PaddleOCR related settings - Add PaddleOCR-VL to parser registry - Add tests for PaddleOCR-VL parser - Update documentation for FileProcessingPipeline
…oise - Add _nearest_neighbor helper to skip over page numbers/headers/footers when matching figure/table titles to their media elements - Improve caption heuristic with Chinese prefix recognition (表, 图, etc.) - Fix missing default values for show_formula_number, prettify_markdown, visualize - Change USE_OCR_FOR_IMAGE_BLOCK default to False - Refactor _coerce_text → _item_content and improve _item_label - Use regex for bbox parsing from image paths - Add docstrings and type improvements - Add comprehensive tests for caption matching logic
…ions - Split PaddleOCRVLParserOptions into nested DocParsingOptions for optionalPayload - Add generic _build_payload helper using dataclass fields - Add new options: merge_layout_blocks, format_block_content, vlm_extra_args - Change DEFAULT_PADDLEOCR_VL_API_MODE from 'local' to 'official' - Support PADDLEOCR_VL_ENDPOINT fallback for official endpoint - Reorder fields in DocParsingOptions for logical grouping - Update client to use nested optional_payload in request_payload
- Merge _coerce_* functions into a single generic _coerce_value - Add _resolve_option and _override_env helpers for consistent override/env handling - Simplify DocParsingOptions and PaddleOCRVLParserOptions construction - Add tests for None rejection and type coercion behavior
- Remove 20+ DEFAULT_* constants that are only used internally within DocParsingOptions.from_env() - Remove test-only helpers (current_endpoint_signature, current_options_signature, current_engine_version, snapshot_tunable_env) - Only 8 items remain in __all__, all actually used by production code - Update tests to access internal helpers via module object
…on and streamline endpoint handling
…add new parsing parameters
…ntation in env.example and processing pipeline
… and add IR validation method
- Added detailed instructions for using the PaddleOCR-VL file parsing engine in FileProcessingPipeline.md, including configuration for both official and local modes, deployment options, and troubleshooting tips. - Updated ParserDebugCLI-zh.md and ParserDebugCLI.md to include PaddleOCR-VL as a supported engine, with relevant command-line options and output directory structure. - Revised ThirdPartyParser-zh.md and ThirdPartyParser.md to reflect the inclusion of PaddleOCR-VL in the list of built-in engines and provided guidance for third-party parser development.
…ngs, and validation logic
… cache and client tests
Collaborator
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f0dedc49e9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Collaborator
|
@codex review |
Contributor
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b2f11d3510
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
added 10 commits
July 9, 2026 20:11
…ove unused sample file check
…ndling - Relax BOS URL host validation from *.bj.bcebos.com to *.bcebos.com for broader region support - Add bbox normalization to handle polygon coordinates and convert to enclosing rectangle - Extract _bbox_key helper for consistent bbox tuple conversion - Update documentation and env.example to reflect the relaxed URL validation - Add tests for polygon bbox normalization, poly bbox image asset matching, and page range cache signature without manifest extras
…cr_vl_options_signature for clarity
…ard on undecodable markdown images Replace the hardcoded `.bcebos.com` BOS host check with an env-driven allowlist (`PADDLEOCR_VL_ALLOWED_ASSET_HOSTS`, default `.bcebos.com`) so self-hosted asset domains can be admitted without code changes; the SSRF guard is preserved since only HTTPS URLs on a listed suffix are fetched. Split image materialization into mandatory vs. soft paths: - `markdown.images` are referenced from the parsed body/IR, so a missing or undecodable entry now raises instead of silently dropping the asset. - `outputImages` are diagnostic, so failures stay soft-skipped with a warning. Also refine the 6+ even-count bbox normalization in the IR builder to document the polygon-vertex heuristic, export `snapshot_tunable_env` from the cache module, and add an autouse conftest fixture that strips all PaddleOCR-VL env vars (including the new host allowlist) so a leaked `.env` can't silently flip option/signature defaults.
…odels Add model recommendation notes across the user-facing docs, env.example, and the cache module default to clarify that PaddleOCR-VL-1.6 is the preferred model, with PaddleOCR-VL-1.5 as an alternative.
…nd streamline engine version retrieval
…l API mode handling
…image download, and _safe_name
- Make options_signature validation strict (MinerU-style: missing
signature = hard miss, prevents silently reusing stale output)
- Refactor _detect_heading to return int level with ("", 0) sentinel,
matching MinerU's call-site pattern; remove text_level logic
(PaddleOCR-VL has no such field)
- Make _download_referenced_images concurrent via asyncio.gather,
separating mandatory (markdown.images) and optional (outputImages)
task groups
- Enhance _safe_name to match sanitize_filename pattern (strip .., /,
\, \x00, \x7f, leading/trailing dots and whitespace)
- Document N--M relative-end page range semantics and
relevel_titles=True heading-level behaviour
- Fill test gaps: strict signature, error paths (httpx RequestError,
HTTP 503, poll failed state), concurrent error surfacing, HTTPS
asset-url gate, _safe_name sanitization
… test coverage for image naming
…nce test coverage for bundle validity
…e order consistency
…andling for JSONL results
…ce tests for allowed hosts
# Conflicts: # lightrag/parser/param_schema.py
…r_external_paddle # Conflicts: # lightrag/parser/param_schema.py
danielaskdd
marked this pull request as draft
August 16, 2026 04:28
dianatofficial
left a comment
dianatofficial
left a comment
There was a problem hiding this comment.
Nice improvement. Refactoring this module enhances readability and code maintainability.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This pull request adds PaddleOCR-VL as a first-class external parser engine for LightRAG. It wires the parser into the registry and pipeline configuration, implements the official API client/cache layer, converts PaddleOCR-VL layout results into the sidecar IR model, and documents configuration and debugging usage.
Related Issues
No linked issue.
Changes Made
paddleocr_vlexternal parser package with cache option normalization, official API client support, parser adapter, and sidecar IR builder.optionalPayloadhandling, environment configuration, and per-file hint validation.env.exampleconfiguration.Checklist
Additional Notes
Local verification completed:
uv run pytest tests/parser/external/paddleocr_vl tests/parser/test_registry.py tests/parser/test_engine_params.py tests/parser/external/test_engine_param_thread.py -q(85 passed, 1 skipped)uv run ruff check .(All checks passed!)git diff --check