doctools.py writes every generated artifact in text mode with no encoding=, so Python falls back to locale.getpreferredencoding(False). On Windows that is the ANSI code page, not UTF-8.
The decision point descriptions carry U+2019 and U+201C/U+201D. Those code points exist in cp950, cp1252 and the other legacy pages, so the write succeeds: no exception, no warning, exit 0. The file is simply no longer UTF-8.
Running the hook exactly as .pre-commit-config.yaml invokes it:
PYTHONPATH=src python src/ssvc/doctools.py --datadir=./data --overwrite # exit 0
on a cp950 host:
|
|
files under data/ modified |
185 |
| no longer valid UTF-8 |
34 |
data/csv/** containing \r\r\n |
16 |
The CSV writer has a second, independent problem: it lacks newline="", and to_csv output already ends lines with \r\n, so text-mode translation makes it \r\r\n.
This is not a private annoyance, because two places instruct a contributor to do exactly this and push the result: the doctools-regenerate pre-commit hook runs on every src/ssvc/**.py edit, and run_doctools.yml tells fork contributors in its own failure message to regenerate locally and push the resulting data/ changes.
Representative line: src/ssvc/doctools.py:184. The same shape is at :197, :247, :273 and :345, and off the hook path at md_gen.py:144 and decision_tables/helpers.py:65.
Happy to send the fix.
doctools.pywrites every generated artifact in text mode with noencoding=, so Python falls back tolocale.getpreferredencoding(False). On Windows that is the ANSI code page, not UTF-8.The decision point descriptions carry U+2019 and U+201C/U+201D. Those code points exist in cp950, cp1252 and the other legacy pages, so the write succeeds: no exception, no warning, exit 0. The file is simply no longer UTF-8.
Running the hook exactly as
.pre-commit-config.yamlinvokes it:on a cp950 host:
data/modifieddata/csv/**containing\r\r\nThe CSV writer has a second, independent problem: it lacks
newline="", andto_csvoutput already ends lines with\r\n, so text-mode translation makes it\r\r\n.This is not a private annoyance, because two places instruct a contributor to do exactly this and push the result: the
doctools-regeneratepre-commit hook runs on everysrc/ssvc/**.pyedit, andrun_doctools.ymltells fork contributors in its own failure message to regenerate locally and push the resultingdata/changes.Representative line:
src/ssvc/doctools.py:184. The same shape is at :197, :247, :273 and :345, and off the hook path atmd_gen.py:144anddecision_tables/helpers.py:65.Happy to send the fix.