Fix five small converter correctness bugs (data URIs, RSS, zip, plaintext, CLI) - #2329
Open
5awtooth wants to merge 1 commit into
Open
Fix five small converter correctness bugs (data URIs, RSS, zip, plaintext, CLI)#23295awtooth wants to merge 1 commit into
5awtooth wants to merge 1 commit into
Conversation
- parse_data_uri: accept unpadded base64 data URIs (was binascii.Error) - RssConverter: initialize md_text so channels without title/description don't raise UnboundLocalError - ZipConverter: fall back to "unknown.zip" instead of printing `None` - PlainTextConverter: guard charset_normalizer .best() returning None so content is never the literal "None" - CLI: fall back to UTF-8 when sys.stdout.encoding is None Each fix has a regression test in tests/test_converter_bugfixes.py. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Author
|
@microsoft-github-policy-service agree |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Five small, independent correctness fixes in the core converters, each with a regression test:
_uri_utils.parse_data_uri— accept unpadded base64.base64.b64decoderaisesbinascii.Error: Incorrect paddingwhen adata:URI's base64 payload length isn't a multiple of 4. Many encoders omit the=padding, so e.g.convert_uri("data:text/plain;base64,SGVsbG8")crashed. Re-pad before decoding.RssConverter—UnboundLocalErroron a channel with no title/description.md_textwas only bound insideif channel_title:, so an RSS<channel>lacking both<title>and<description>raisedUnboundLocalErrorin the item loop. Initializemd_text = ""first.ZipConverter— don't print`None`as the archive name. When converting raw zip bytes viaconvert_stream(no url/local_path/filename), the header readContent from the zip file `None`:. Fall back to"unknown.zip".PlainTextConverter— don't emit the literal"None".charset_normalizer's.best()returnsNonewhen nothing matches;str(None)made the document content"None". Guard against it._handle_output— toleratesys.stdout.encoding is None.sys.stdout.encodingcan beNonefor some redirected/embedded streams, which turned the output step intoAttributeError. Fall back to UTF-8.How these were found
I did a whole-repository read of
markitdownwith an LLM to surface candidate issues, then verified each one by hand — reading the surrounding code and running a minimal repro. Only fixes I could confirm are included, and each is deliberately minimal and idiomatic to the surrounding code. Happy to split this into separate PRs if you'd prefer one change per PR.Test plan
packages/markitdown/tests/test_converter_bugfixes.py; each fails before its fix and passes after.340 passed, 4 skippedlocally on Python 3.13 (the one unrelated failure,test_speech_transcription, is a network/service-dependent test that also fails on a clean checkout in this environment).🤖 Generated with Claude Code