From 8443c3643aa69df004eebcb539a8ee5b45684a78 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 10:10:00 -0500 Subject: [PATCH 01/31] docs(adrs): Add the architecture decision record scaffold why: Design decisions for the doctest engine have lived in commit bodies and pull request threads, where a later reader cannot tell a deliberate constraint from an accident. The project needs one place that records what was decided, what forced it, and what it rules out. what: - Add docs/adrs/index.md stating what a record is for - Fix numbering as sequential and permanent, and name the four statuses a record can carry - Require pinned source links, since a blob/master anchor rots silently onto unrelated code while still resolving - Wire the section into the docs toctree --- docs/adrs/index.md | 27 +++++++++++++++++++++++++++ docs/index.md | 1 + 2 files changed, 28 insertions(+) create mode 100644 docs/adrs/index.md diff --git a/docs/adrs/index.md b/docs/adrs/index.md new file mode 100644 index 0000000..273990e --- /dev/null +++ b/docs/adrs/index.md @@ -0,0 +1,27 @@ +(adrs)= + +# Architecture Decision Records + +Significant design decisions for `doctest_docutils` and +`pytest_doctest_docutils`, and their rationale. + +These records govern the shape of the doctest engine: what it produces, what it +may reach into, and what vocabulary it speaks. A record states the context that +forced a decision, the decision itself, what it costs, and what it rules out — +so a later reader can tell a deliberate constraint from an accident. + +Supporting structural research lives in `notes/analyses/`. It decides nothing and +is cited by these records as evidence. + +## Conventions + +**Numbering** is sequential and permanent. A record is never renumbered, and a +superseded one is marked rather than deleted. + +**Status** is one of `Draft`, `Proposed`, `Accepted`, `Superseded by NNNN`. + +**Source links are pinned.** Every citation of an external project names a git +tag, or a commit reachable from that project's trunk where it publishes no tags. +Line anchors are only used on a pinned ref, because they are meaningless without +one, and a `blob/master` link rots silently — the file moves, lines shift, and +the anchor lands on unrelated code while still resolving. diff --git a/docs/index.md b/docs/index.md index aba9e4c..006ef63 100644 --- a/docs/index.md +++ b/docs/index.md @@ -74,6 +74,7 @@ modules/doctest_docutils/index modules/pytest_doctest_docutils/index modules/linkify_issues/index project/index +adrs/index history GitHub ``` From a857a1c90f6d20977930e7ad20689b93a890b1b1 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 10:11:04 -0500 Subject: [PATCH 02/31] docs(adrs[0001]): Record the doctest core architecture why: The finder is one 320-line method spanning parsing, grouping and naming; the runner reaches CPython's name-mangled loop through a code-object clone; the plugin blocks pytest's doctest plugin and then imports its privates; and a fork of xdist guards state xdist cannot see. All four follow from one conflation: test identity and shared state are separate axes, coupled here into a single setting. what: - Decide the item is the group and the DocTest is the block, so N DocTests report N locations under one unsplittable node id - Record the three facts it rests on, each verified by running it: repr_failure reads locations per failure, _DocTestRunner__run is an ordinary attribute override, and an item-scoped mapping never crosses a process - Fix the vocabulary where doctest, pytest and Sphinx collide on globs, namespace, group, scope, skip and name - Tabulate the upstream constraints, each anchored at a pinned tag - Record what was rejected and why, including prefix replay, IntFlag option surfaces and a whole-file docstring - Stub the five deferred decisions as 0002 through 0006 --- docs/adrs/0001-typed-vanilla-doctest-core.md | 551 ++++++++++++++++++ .../0002-runner-conformance-across-cpython.md | 71 +++ ...0003-retiring-per-block-namespace-items.md | 54 ++ docs/adrs/0004-diagnostics-as-data.md | 63 ++ .../0005-line-recovery-for-nested-blocks.md | 59 ++ .../0006-pytest-private-api-compatibility.md | 81 +++ docs/adrs/index.md | 11 + 7 files changed, 890 insertions(+) create mode 100644 docs/adrs/0001-typed-vanilla-doctest-core.md create mode 100644 docs/adrs/0002-runner-conformance-across-cpython.md create mode 100644 docs/adrs/0003-retiring-per-block-namespace-items.md create mode 100644 docs/adrs/0004-diagnostics-as-data.md create mode 100644 docs/adrs/0005-line-recovery-for-nested-blocks.md create mode 100644 docs/adrs/0006-pytest-private-api-compatibility.md diff --git a/docs/adrs/0001-typed-vanilla-doctest-core.md b/docs/adrs/0001-typed-vanilla-doctest-core.md new file mode 100644 index 0000000..51a4c9f --- /dev/null +++ b/docs/adrs/0001-typed-vanilla-doctest-core.md @@ -0,0 +1,551 @@ +(adr-0001-typed-vanilla-doctest-core)= + +# ADR 0001: A typed, vanilla-compatible doctest core + +Status: Proposed +Date: 2026-08-02 + +## Context + +`doctest_docutils` re-implements the *finding* half of {func}`doctest.testfile` +over a docutils or MyST doctree, keeps CPython's *running* half, and adds a +grouping layer — namespace, phase, merge, lift — that exists in neither parent. +`pytest_doctest_docutils` wraps that in a pytest plugin. + +The two modules work. They are also the join point of three separate +vocabularies and three separate extension models, and the seams between them +were never named. Four structural costs follow. + +**The finder is one method.** `DocutilsDocTestFinder._find` performs format +dispatch, node filtering, group resolution, wildcard expansion, name-collision +detection, `:skipif:` evaluation, `testoutput` pairing, test construction, +merging, splitting and naming, in one body with two nested closures, with +configuration threaded through as positional parameters. There is no boundary +between *document → blocks*, *blocks → tests*, and *tests → items*, so a change +to any one of them is a change to all three. + +**It reaches through name-mangled CPython privates.** Sphinx's prompt-free +`{testcode}` form needs `compile()` in `"exec"` mode, but the per-example loop +[`DocTestRunner.__run`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1344) +hard-codes `"single"` +([`Lib/doctest.py:1400`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1400)). +`sphinx.ext.doctest` solves this by rebinding `doctest.compile` process-wide and +never restoring it +([`sphinx/ext/doctest.py:310`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L310)) — +unavailable to a library that loads into every pytest session that installed it. +`_exec_mode_run` therefore clones the mangled method's code object into a fresh +{class}`types.FunctionType` whose globals map `compile` to a local helper, guarded +by probing `co_freevars` and `co_names` and degrading with a logged error. That +mechanism also carries a latent defect: its globals are a snapshot of +`vars(doctest)` taken at import, so a later rebind of a module-level name in +`doctest` is invisible to the clone while remaining visible to the stock runner, +and two runners in one process disagree. + +**It blocks the plugin whose internals it imports.** `pytest_configure` calls +`config.pluginmanager.set_blocked("doctest")`, and the same module then imports +that plugin's private helpers — `_is_setup_py`, `_is_main_py`, `_get_checker`, +`_get_continue_on_failure` — and reads four ini and CLI options the blocked +plugin declared. This survives only because `_pytest/fixtures.py` has no +`pytest_plugin_unregistered` handler, so the already-parsed `doctest_namespace` +fixture outlives unregistration. + +**It forks pytest-xdist to protect state xdist cannot see.** Because a live +`globs` mapping is a Python object and only execnet-serializable builtins cross a +worker boundary, a shared namespace must either be merged into one +{class}`doctest.DocTest` or kept on one worker. Keeping it there means a +scheduler, and the only affinity primitive in all of xdist is +[`LoadScopeScheduling._split_scope`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L284), +a pure function on node-id strings. The controller never collects; it learns the +suite only as node ids arriving from workers. So `_shared_page` and `_is_page` +re-derive "these ids share state" from strings, and `_worker_count` re-implements +[`parse_tx_spec_config`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/workermanage.py#L26) +including its quirks. + +Underneath all four is one conflation. **The granularity of test identity and the +granularity of shared state are different axes, and the current design couples +them.** `namespace_items = merged` gives one `DocTest` and one node id per group; +`per-block` gives N `DocTest`s, N node ids, and one live mapping — a node id that +raises `NameError` when selected alone. Every surveyed project makes the same +mistake or a worse one (see [](#prior-art)). + +## Decision + +**The pytest item is the group. The {class}`doctest.DocTest` is the block.** + +One {class}`pytest.Item` per (document, group). Inside it, a tuple of per-block +`DocTest`s run in phase order against one live `globs` mapping that never leaves +the item. + +This occupies the cell no surveyed project occupies: N `DocTest`s, one node id. +Per-block failure locations, per-block gutters, per-block `SKIPPED`, per-block +"location unknown" — while `-k`, `--lf`, `-x`, `--reruns` and every `--dist` mode +are structurally incapable of splitting the shared state, because there is only +one item to schedule. + +### The three facts this rests on + +Each was verified by executing it, not by reading it. + +**1. pytest reads failure locations per failure, not per item.** +[`DoctestItem.repr_failure`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L317) +iterates the failure list and reads `failure.test.filename`, `failure.test.lineno` +and `example.lineno` inside the loop +([`_pytest/doctest.py:337-344`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L337-L344)): + +```python +for failure in failures: + example = failure.example + test = failure.test + filename = test.filename + if test.lineno is None: + lineno = None + else: + lineno = test.lineno + example.lineno + 1 +``` + +With one `DocTest` per block, every failure therefore carries its own `filename` +and `lineno` for free. A block reached through `.. include::` reports the +*included* file. A block docutils could not locate carries `lineno=None` and +takes pytest's honest `EXAMPLE LOCATION UNKNOWN` branch **without poisoning its +siblings**. No override of `repr_failure` or `reportinfo` is required. + +This is what retires `_merge_blocks`: the synthetic merged page, its blank-line +padding and its `max(..., len(lines))` clamp exist only to reconstruct locations +from a single spliced docstring. + +**2. `_DocTestRunner__run` is an ordinary attribute override.** Name mangling +rewrites the *call site* at compile time, so the `self.__run(...)` lookup inside +`run()` +([`Lib/doctest.py:1571`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1571)) +resolves through the instance's MRO like any other attribute. A subclass that +defines `_DocTestRunner__run` takes over the loop, with `run()` untouched. + +This page is itself a doctest, so the claim is checked on every run: + +```{doctest} +>>> import doctest +>>> import io +>>> fired = [] +>>> class Runner(doctest.DocTestRunner): +... def _DocTestRunner__run(self, test, compileflags, out): +... fired.append(test.name) +... return super()._DocTestRunner__run(test, compileflags, out) +>>> example = doctest.Example("1 + 1\n", "2\n") +>>> test = doctest.DocTest([example], {}, "demo", "demo.py", 0, None) +>>> Runner().run(test, out=io.StringIO().write) +TestResults(failed=0, attempted=1) + +The subclass method ran, and `run()` was never overridden: + +>>> fired +['demo'] +``` + +Keeping `run()` as stdlib's matters: it owns the save-and-restore of +`sys.stdout`, `pdb.set_trace`, `linecache.getlines`, `sys.displayhook`, +`_colorize.can_colorize` and the `PYTHON_COLORS`/`FORCE_COLOR` environment +variables, all in its own `finally` +([`Lib/doctest.py:1534-1573`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1534-L1573)). +That contract is not reproduced; it is inherited. + +**3. Making the item the sharing unit dissolves distribution.** A live mapping +never crosses a process boundary, so there is nothing for xdist to split, under +any `--dist` mode. No affinity primitive, no scheduler substitution, no +`parse_tx_spec_config` fork, no node-id string sniffing. + +### Layers + +Dependencies flow one way, from the leaf toward the hosts. No layer may import a +layer above it. + +```text +blocks inert data: Block, BlockKind, Phase, Diagnostic, Example + | (stdlib imports only) +markup/ _rst, _myst, _text, _python -> (blocks, diagnostics) + | +project grouping, pairing, phase order, naming -> GroupTest + | +runner owns the per-example loop; phase sequencing + | +settings one frozen Settings, resolved once + | +pytest_doctest_docutils collection, items, globs lifetime, reporting +``` + +| Layer | Owns | Must not know | +|---|---|---| +| `blocks` | `Block`, `BlockKind` registry, `Phase`, `Diagnostic`, `BlockAttributes`, `Example(doctest.Example)` carrying compile mode | docutils, MyST, Sphinx, pytest, xdist, the filesystem. Stdlib imports only, enforced by an `import-linter` contract | +| `markup/` | Text → `(blocks, diagnostics)`. Line-number recovery and its per-front-end meaning, `.. include::` attribution, `nodes.comment` traversal, reporter capture, idempotent directive registration | Groups as a runtime concept, `DocTest`, pytest, pairing | +| `project` | The **only** place grouping exists: `*` expansion, anonymous naming, phase order, `testcode`/`testoutput` pairing, option defaults, name minting. A pure function | docutils, pytest, the filesystem, whether anything will run. Evaluates no user code | +| `runner` | `_DocTestRunner__run`; option merge, `SKIP`-after-merge, `FAIL_FAST`, `report_*` dispatch, version shims. `run_group()` owns phase sequencing and the `try`/`finally` guaranteeing `testcleanup` | docutils, markup, pytest. Never overrides `run()` | +| `settings` | One frozen `Settings` resolved once, with `None` sentinels at the resolve boundary so a future default change is announceable | pytest's `Config`, argparse, ini format, Sphinx's `app` | +| `pytest_doctest_docutils` | Options, `Document(pytest.Module)`, `DocutilsItem`, group `globs` lifetime, built-in-plugin dedup, surfacing diagnostics | docutils node classes, MyST configuration, grouping rules | + +`pytest_doctest_docutils._compat` is the only module that imports +`_pytest.doctest`, behind a pinned support matrix. See +{doc}`0006-pytest-private-api-compatibility`. + +### Vocabulary + +Goal (e) — speaking doctest's, pytest's *and* Sphinx's idioms — is mostly a +naming problem, because the three overload the same nouns with different +referents. Each term below is decided once and used only that way. + +| Term | doctest | pytest | Sphinx | Decision | +|---|---|---|---|---| +| `globs` | the dict examples exec in; [`DocTest.__init__` stores a **copy**](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L565) | — | assigned to `test.globs` after construction, run with `clear_globs=False` | Keep `globs` for the mapping | +| namespace | — | [`doctest_namespace`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L721) means *injected names* | — | **Not** used for the sharing unit; pytest owns the word | +| group | — | `xdist_group` is a *scheduling* affinity marker ([`remote.py:245-254`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/remote.py#L245-L254)) | the author-facing bucket: `.. doctest:: intro`, `default`, `*` | Adopt `group` for the sharing unit. The xdist affinity key is *derived*, never the group name | +| scope | — | the fixture-lifetime ladder | — | Reserved for pytest. The block-vs-document axis becomes `share` | +| test / item / block | `DocTest`, `Example` | `Item` | [`TestCode`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L235) is the parsed unit | Three nouns: `Example` (stdlib), `Block` (parsed), `DocTest` (runnable) | +| skip | the `SKIP` flag, short-circuiting before `report_start` | a reported outcome with a reason | [drops the node entirely](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L443-L444) | pytest's meaning, doctest's mechanism. Sphinx's drop is deliberately rejected | +| directive | inline `# doctest: +FLAG` | — | a docutils directive with an `option_spec` | Reserved for the docutils meaning. doctest's form is "inline flags" | +| optionflags | an int bitmask; `register_optionflag` | `doctest_optionflags` ini | `:options:` plus `doctest_default_flags` | Keep verbatim. `register_optionflag` is the one genuinely cross-library extension point | +| setup / cleanup | `setUp`/`tearDown` on the suite builders | fixtures | `testsetup`/`testcleanup` directives | Author-facing names stay Sphinx's; `phase` is the internal ordering axis; a fixture is never "setup" | +| name | a dotted path; `__lt__` compares it as **text** | node id is `parent.nodeid + "::" + name` | the *group* name, shared by every block in it | `DocTest.name` is a unique machine-independent id. The absolute path lives only in `filename` | + +### Data model + +```python +class Phase(enum.IntEnum): + SETUP = 0 + TEST = 1 + CLEANUP = 2 + + +class Block(t.NamedTuple): + kind: str # registered BlockKind name + source: str # dedented body, verbatim + want: str | None # from a paired testoutput + path: pathlib.Path # the file the text lives in, not the collected document + line: int | None # None when docutils could not recover one + position: int # 0-based document pre-order index; the naming key + groups: tuple[str, ...] # declared verbatim; () and ("*",) unresolved here + options: t.Mapping[int, bool] # plain int keys, exactly as doctest produces + skipif: str | None # UNEVALUATED + hidden: bool + + +class BlockKind(t.NamedTuple): + name: str + phase: Phase + mode: t.Literal["single", "exec"] + pairs_with: str | None + grouped: bool + node_types: tuple[str, ...] # docutils tagnames; "comment" for testsetup/:hide: + option_spec: t.Mapping[str, t.Callable[[str], object]] + + +class GroupTest(t.NamedTuple): + group: str + tests: tuple[doctest.DocTest, ...] # one per block, in phase order + globs: dict[str, t.Any] # the live mapping, shared by every test above +``` + +`Block.line` being nullable is load-bearing, not defensive. A bare `>>>` block +nested in a `.. note::`, a list item or a block quote reports `line=None, +source=None` from docutils, and an `.. include::`-ed block numbers against the +*included* file. Both propagate to `DocTest.lineno=None` and pytest's honest +"location unknown", rather than to a fabricated number. + +`BlockKind.node_types` is likewise load-bearing: `testsetup`, `testcleanup` and +any `:hide:` block are emitted as {class}`docutils.nodes.comment`, not +`literal_block` +([`sphinx/ext/doctest.py:92-93`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L92-L93)). +A walker restricted to `literal_block` silently loses every one of them while the +page still renders. + +### Typing + +The runtime objects are stdlib's, unconditionally. Precision lives in a parallel +layer that never changes what is constructed. + +- **`Protocol` for the seams, nominal subclassing for the concrete classes.** + `Frontend` is a `Protocol` so a third party can supply one structurally; the + shipped parser also subclasses {class}`doctest.DocTestParser` so it stays + passable to `doctest.testfile(parser=...)` and + {func}`doctest.DocFileSuite`. Typeshed's signatures demand the class, not the + shape — `DocTestFinder.__init__(parser: DocTestParser = ...)` — so a compatible + `find()` alone is not enough, which is why `DocutilsDocTestFinder` cannot be + passed to `DocTestSuite(test_finder=...)` today. +- **`TypedDict` at the docutils boundary.** `BlockAttributes` types what a + directive stamps on a node, with one narrowing accessor that validates once. + This is where `Any` currently enters: `str(node.get("test") or ...)` and + `dict(node.get("options") or {})` are runtime coercions paid for a static hole, + and typeshed's docutils stub makes it worse — its `get(key, failobj: _T) -> _T` + claims `_T` even when the key is present holding something else. +- **`t.Literal` for closed vocabularies**, derived from one source of truth so a + public signature and a config field cannot diverge. +- **Plain `int` keys for optionflags.** {class}`enum.IntFlag` was considered and + rejected; see [](#alternatives-rejected). +- **`py.typed` ships.** The project already runs mypy strict over `src` and + `tests`, and the marker file does not exist, so every consumer sees the package + as untyped. This is a packaging defect independent of the rest of this ADR. + +## Constraints + +The design is pinned by facts about three upstreams. Each was verified at the tag +cited. The full derivation is in `notes/analyses/`. + +### CPython `doctest` (v3.14.2) + +| Constraint | Anchor | +|---|---| +| A failure's file line is `test.lineno + example.lineno + 1`; `Example.lineno` is 0-based within the containing string | [`doctest.py:1344`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1344) | +| `DocTest.__init__` **copies** the globs mapping, so a shared mapping must be assigned after construction and run with `clear_globs=False` | [`doctest.py:565`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L565) | +| Never sort collected tests by name: `__lt__` compares names as text, and a name carries its position as text, so `page.md[10]` sorts before `page.md[1]` | [`doctest.py:596`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L596) | +| The per-example loop is name-mangled; the supported in-loop seams are the four `report_*` methods and the injected checker | [`doctest.py:1286-1314`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1286-L1314) | +| `run()` mutates global interpreter state for its duration and restores in `finally`; it is neither reentrant nor thread-safe | [`doctest.py:1534-1573`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1534-L1573) | +| Each example compiles under `""` in `"single"` mode with `dont_inherit=True`; `"exec"` suppresses expression echo, emptying every `want` | [`doctest.py:1400`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1400) | +| `TestResults` is a 2-field namedtuple carrying `skipped` as an extra instance attribute; a third tuple field breaks every `failures, tries = runner.run(...)` unpack | [`doctest.py:114`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L114) | +| Custom flag names must be registered at import; ints are `1 << len(OPTIONFLAGS_BY_NAME)` and an unregistered name makes a page fail to **parse** | [`doctest.py:153`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L153) | + +`report_skip` does not exist at v3.14.2 — the runner has only `report_start`, +`report_success`, `report_failure` and `report_unexpected_exception`. A runner +that owns the loop must probe for it rather than assume it. + +### pytest (9.1.1) + +| Constraint | Anchor | +|---|---| +| `repr_failure` reads each failure's own `test` — the fact this design is built on | [`doctest.py:317-344`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L317-L344) | +| `DoctestItem.setup()` does `self.dtest.globs.update(globs)`, so the mapping must be mutable and survive collection → setup → run | [`doctest.py:288-293`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L288-L293) | +| `runtest()` calls `run(self.dtest, out=failures)` with `clear_globs` defaulting to `True` — which would empty a shared mapping after the first block | [`doctest.py:295-303`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L295-L303) | +| `PytestDoctestRunner` is defined *inside* `_init_runner_class()` and is not importable, so the `OutcomeException` re-raise, `BdbQuit` → `outcomes.exit` and `continue_on_failure` handling do **not** come for free | [`doctest.py:178-181`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L178-L181) | +| A page must be a `pytest.Module` with `obj = None`, and the collector must call `parsefactories` | [`doctest.py:421`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L421), [`:556`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L556) | +| `_is_doctest` claims any `.txt`/`.rst` **initial path before consulting `--doctest-glob`**, so `pytest docs/page.rst` is claimed by the built-in plugin regardless of glob | [`doctest.py:148-152`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L148-L152) | +| An empty `DocTest` must not be yielded as an item | [`doctest.py:451`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L451) | + +### pytest-xdist (v3.8.0) + +| Constraint | Anchor | +|---|---| +| Every worker must collect identical node ids in identical order. Violation is not an exception — the scheduler logs `**Different tests collected, aborting run**` and the session executes zero tests | [`load.py:259`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/load.py#L259), [`loadscope.py:359`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L359) | +| The only affinity primitive is `_split_scope(nodeid) -> str`; `loadfile` and `loadgroup` are two-line overrides of it, and `load`/`worksteal` have no scope concept at any layer | [`loadscope.py:284`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L284), [`loadfile.py:35`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadfile.py#L35) | +| `xdist_group` is honoured only when the *worker's own* `--dist` is `loadgroup`, and works by appending `@name` to `item._nodeid` | [`remote.py:245-254`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/remote.py#L245-L254) | +| `parse_tx_spec_config` builds a list, so a negative multiplier contributes zero specs — `xspeclist.extend([spec] * num)`, not a sum | [`workermanage.py:26-37`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/workermanage.py#L26-L37) | + +Making the item the sharing unit removes the need to satisfy the first three at +all. The fourth is why `_worker_count` is deleted rather than fixed. + +### Sphinx (v9.1.0) + +| Constraint | Anchor | +|---|---| +| `testsetup`, `testcleanup` and `:hide:` blocks are emitted as `nodes.comment` | [`doctest.py:92-93`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L92-L93) | +| A `:skipif:`-gated node is dropped during collection, with no outcome, id or count | [`doctest.py:443-444`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L443-L444) | +| `:options:` is accepted only on `doctest` and `testoutput`; a `testcode`'s own options are discarded in favour of its output block's | [`doctest.py:111`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L111), [`:207`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L207) | +| `is_allowed_version(spec, version)` takes the specifier **first** — the reverse of this project's local helper | [`doctest.py:45`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L45) | +| `DocTestBuilder` flips a mutable `self.type` between `"single"` and `"exec"` and reads it through a process-global `doctest.compile` patch | [`doctest.py:310`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L310), [`:548`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L548) | + +## Tensions + +Each is a genuine conflict where satisfying one goal costs another. "Both" is not +an answer; the position taken and its price are recorded. + +**A vanilla `DocTest` cannot carry the front-end's metadata.** It has exactly +`(examples, globs, name, filename, lineno, docstring)`. *Position:* metadata that +the runner needs rides on a nominal {class}`doctest.Example` subclass — +`doctest.Example` has no `__slots__`, so attributes survive +{func}`copy.copy`, {mod}`pickle`, and a third party's naive +`DocTest(examples, globs, name, filename, lineno, docstring)` rebuild, because +that rebuild reuses the same `Example` objects. Metadata the runner does *not* +need — groups, wildcards, pairing — never touches a stdlib object and dies in the +projection layer. *Price:* one subclass to explain, and a rule that nothing may +smuggle through `DocTest.name`. + +**Node-id granularity versus shared state.** *Position:* decouple them — N +`DocTest`s under one node id. *Price:* selecting a group runs all its blocks; +there is no id that names block three alone. That is honest: no surveyed +implementation makes a node id a promise of independent runnability, and the +current `per-block` mode ships ids that raise `NameError` when selected. + +**Sphinx's skip versus pytest's skip.** *Position:* pytest's meaning, doctest's +mechanism — set `SKIP`, never drop the node. *Price:* a page carrying a gated +block gains an item relative to `sphinx-build -b doctest`, and `--collect-only` +must not evaluate the gate, which is why `:skipif:` is carried through collection +unevaluated and run in `runtest()`. + +**Collection purity versus `--collect-only` fidelity.** *Position:* collection +evaluates no user Python, so it is a pure function of (bytes, argv, ini). *Price:* +`--collect-only` no longer shows which blocks will skip. + +**Sphinx compatibility versus silent-loss behaviours.** Sphinx silently discards +an orphan `testoutput`, silently overwrites a duplicate one, and silently drops a +`testcode`'s own `:options:`. *Position:* keep the behaviour, add a diagnostic +with a stable code. *Price:* a page warns under pytest and is silent under +`sphinx-build`; the results still match. + +**Owning the loop versus tracking CPython.** *Position:* own it, because +constraint 4 makes it the only way to control compile mode without a +process-global patch or a code-object clone. *Price:* a version shim and a +conformance harness. See {doc}`0002-runner-conformance-across-cpython`. + +## What this deletes + +`_find`'s monolith · `_merge_blocks` and its padding and clamp · +`_split_skipped_blocks` and `_lifted_name` · `_exec_mode_run`, `_compile_source` +and `_ExecSource` · `_worker_count` · `_shared_page` and `_is_page` · +`_splitting_scheduler` and both xdist hooks · the `_init_runner_class` fork · +`set_blocked("doctest")` and its unblock path. + +**The result is not smaller.** It lands roughly flat against the current source. +Under this project's rule that every function carries a NumPy docstring with a +working doctest, splitting one long method into five functions costs prose it did +not previously pay. The value is fewer hazards, not fewer lines: a code-object +clone with a stale globals snapshot, a fork of an upstream parser, node-id string +sniffing, and a method that branches on string literals to decide what a block is +all disappear. Gate on shape instead — a function-length ceiling, a module-length +ceiling, and the `import-linter` contract on the leaf. + +(prior-art)= + +## Prior art + +| Project | Bet | Outcome | +|---|---|---| +| [Sybil 10.0.1](https://github.com/simplistix/sybil/tree/10.0.1) | A document is a flat sequence of non-overlapping character spans; every format is a regex lexer; zero runtime dependencies | Format independence at no dependency cost, and a non-overlap invariant that raises on double collection. But one mutable namespace per document with [one independently selectable item per span](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/integration/pytest.py), so `-k` on a later example raises `NameError`. [Node ids are positional](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/sybil.py#L155-L157) (`line:4,column:1`), so adding a paragraph renames every downstream test. No group support at all — a regex cannot see directive options | +| [xdoctest v1.3.2](https://github.com/Erotemic/xdoctest/tree/v1.3.2) | Abandon stdlib compatibility; own the parser via `ast`/`tokenize`; make directives structured objects | `ast` parsing and structured directives are real advances — [`REQUIRES`](https://github.com/Erotemic/xdoctest/blob/v1.3.2/src/xdoctest/directive.py#L58) carries *why* a block skipped, which a bool cannot. But it is now building compatibility back, and its permissive got/want defaults silently change tests users wrote for stdlib. It unregisters pytest's doctest plugin outright | +| [pytest-examples v0.0.18](https://github.com/pydantic/pytest-examples/tree/v0.0.18) | Emit the canonical form rather than parse it; rewrite expected output in place | Check-mode and update-mode collapse into one path, and byte offsets plus an invertible dedent scalar are what a data model needs to rewrite source. Composes with pytest by contributing no collector at all — the cheapest correct integration in the survey | +| [typeshed](https://github.com/python/typeshed/blob/8c7256c/stdlib/doctest.pyi) | Annotate the 2001 API faithfully | Hands back `Any` at exactly the three extensible points — `globs`, `**options`, `optionflags: int`. Declares `DocTestRunner.test: DocTest` unconditionally although runtime assigns it only inside `run()`, so the stub type-checks a crash | + +The collective lesson: **namespace scope is not test identity, markup parsing is +not Python parsing, and runtime compatibility is not static precision.** Every +project conflated at least two, and the first conflation is the one that produces +silent wrong answers rather than inconvenience. + +(alternatives-rejected)= + +## Alternatives rejected + +**Prefix replay** — re-executing a group's predecessors so any block can be +selected standalone. It rebinds `getfixture` to the *replaying* item's request, +so a replayed predecessor resolves different fixture instances: a results +difference, not a performance one. It re-executes gated and deselected blocks +with no node id, in exactly the environment the gate says they must not run in. +It is superlinear precisely where shared groups exist. And it assumes +idempotence, while downstream setup blocks spawn servers and create +repositories. + +**One `DocTest` per group whose `docstring` is the whole file.** Attractive — +`lineno=0` and true file lines with no padding — but broken at the two shapes +docutils cannot locate. A nested block reports `line=None`, and an included file +numbers against itself, so one group-wide docstring maps both to confidently +wrong lines, and pytest's per-`DocTest` "location unknown" signal becomes +structurally inexpressible. Per-block `DocTest`s deliver the same benefits with +none of this. + +**{class}`enum.IntFlag` as the optionflag surface.** The runtime hash equality is +real and irrelevant to the claims made from it. With a third-party flag +registered — and pytest lazily registers `ALLOW_UNICODE`, `ALLOW_BYTES` and +`NUMBER` in [`_get_flag_lookup`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L385) — +iteration and `repr` silently under-report members outside the enum. And +`Mapping` is invariant in its key type, so `Mapping[Flag, bool]` fails mypy +strict against `dict[int, bool]` in both directions, requiring casts at exactly +the boundaries the change was supposed to clean. + +**A registered `EXEC` optionflag to carry compile mode.** It would put compile +mode in the same user-writable namespace as `ELLIPSIS`. A `doctest_optionflags = +EXEC` in any ini, or a stray `# doctest: +EXEC`, compiles in exec mode, which +suppresses expression echo so every `want` compares against empty output and the +suite passes vacuously. Compile mode is an attribute on the `Example` subclass, +unreachable from user configuration. + +**Sybil's non-overlap-raises invariant.** Two `.. include::` directives naming +the same file produce blocks over identical source spans. That is a legitimate +page, and an invariant that raises would reject it. Double collection becomes a +diagnostic carrying both provenances. + +**Reimplementing `_pytest.doctest`'s helpers instead of importing them.** +Mis-costed by roughly threefold: `_get_checker` alone returns a checker +implementing `ALLOW_UNICODE`, `ALLOW_BYTES` and `NUMBER` with float-precision +handling. Since the plugin still reads `doctest_optionflags` from the built-in +plugin, a project setting `NUMBER` would either raise or — worse — have the bit +accepted and silently do nothing. + +**A Sphinx builder.** New scope in a rewrite that must not grow, and both drafts +that proposed one had it deliberately diverging from `sphinx-build -b doctest` on +`:skipif:` and the silent-loss cases. Shipping a builder that disagrees with the +tool it replaces is worse than shipping none. + +**An import-time guard that raises.** A `pytest11` plugin raising at import +aborts the whole session, taking down suites whose majority of tests never touch +a doctest. It also checks the wrong property: `hasattr(DocTestRunner, +"_DocTestRunner__run")` is true in exactly the scenario it claims to prevent, +while the thing that actually changed within the supported range — +`__record_outcome`'s arity — is invisible to it. This becomes a differential +conformance test in CI. + +## Consequences + +### Positive + +- Failure locations are correct by construction, including through `.. include::`, + with no `repr_failure` or `reportinfo` override. +- A block docutils cannot locate degrades to an honest disclaimer instead of a + fabricated line, and does not affect its siblings. +- Every `--dist` mode works, because there is no shared state to split. +- No CPython code-object clone, and no process-global rebinding of anything. +- Collection is a pure function of (bytes, argv, ini), so worker divergence is + structurally impossible and `--collect-only` runs no user code. +- Grouping is one pure function with no docutils, pytest or filesystem dependency, + and is testable without any of them. +- A new block kind is a registration, not an edit to a method branching on string + literals. +- Parse diagnostics become values with stable codes rather than stderr writes and + mid-parse aborts. + +### Tradeoffs + +- The per-example loop is this project's to maintain across supported + interpreters, including two version-shaped divergences. +- A page containing Sphinx `{testcode}` blocks produces `DocTest`s a stock runner + cannot run, because `compile("a = 1\nb = 2\n", "", "single")` raises. + Prompt-form blocks — the overwhelming majority — run perfectly on an unmodified + runner, and a test asserts it. +- The plugin now *requires* the built-in doctest plugin rather than blocking it, + so `-p no:doctest` is an error rather than a degraded mode. +- Retiring `namespace_items = per-block` is a real feature removal. +- The line count does not fall. + +### Risks + +**Runner drift.** A CPython refactor that inlines the loop into `run()` would +silently route execution back to stdlib — invisible for prompt-form blocks, +immediately broken for `{testcode}`. Mitigated by the conformance harness in +{doc}`0002-runner-conformance-across-cpython`, gating on capability probes rather +than a `sys.version_info` ladder. + +**pytest private API.** Four private helpers and a subclassed item. Mitigated by +quarantining them in one module behind a pinned matrix; see +{doc}`0006-pytest-private-api-compatibility`. + +**Foreign directive registration.** `Sphinx.add_directive` overrides existing +registrations unconditionally, so `sphinx.ext.doctest` loaded in the same +interpreter can replace these directive classes. Mitigated by reading +`BlockAttributes` off the node — byte-compatible with what Sphinx stamps — rather +than depending on this project's own classes having run. + +**Over-suppressed diagnostics.** Suppressing one code too many turns a broken page +into a silent zero-test page, which is worse than a mid-parse abort. Mitigated by +the narrow default set in {doc}`0004-diagnostics-as-data`. + +## Relationship to other ADRs + +This ADR fixes the architecture. Five decisions it defers get their own records: +{doc}`0002-runner-conformance-across-cpython` (how the owned loop is proven +equivalent), {doc}`0003-retiring-per-block-namespace-items` (the deprecation +path), {doc}`0004-diagnostics-as-data` (what is reported and what is suppressed), +{doc}`0005-line-recovery-for-nested-blocks` (the optional last step), and +{doc}`0006-pytest-private-api-compatibility` (the quarantine and its matrix). + +## Final position + +The core produces real {class}`doctest.DocTest` objects holding real +{class}`doctest.Example` objects, and `Example.source` is the author's verbatim +text. Everything else — groups, phases, pairing, diagnostics, distribution — is a +layer above that fact, and no layer reaches around another. + +The unit that shares a `globs` mapping is the unit pytest schedules. That is the +one invariant every other property in this document follows from, and it is not +negotiable for a convenience elsewhere. diff --git a/docs/adrs/0002-runner-conformance-across-cpython.md b/docs/adrs/0002-runner-conformance-across-cpython.md new file mode 100644 index 0000000..bd7882e --- /dev/null +++ b/docs/adrs/0002-runner-conformance-across-cpython.md @@ -0,0 +1,71 @@ +(adr-0002-runner-conformance-across-cpython)= + +# ADR 0002: Runner conformance across CPython versions + +Status: Draft +Date: 2026-08-02 + +## Context + +{doc}`0001-typed-vanilla-doctest-core` decides that the runner owns the +per-example loop by defining `_DocTestRunner__run` in a subclass, rather than +cloning CPython's code object or rebinding `doctest.compile` process-wide. + +Owning the loop means owning the private state it writes into, and that state has +changed shape inside this project's supported interpreter range. Three +divergences are known: + +**The outcome accumulator changed name and arity.** On 3.10 through 3.12 it is +`__record_outcome(self, test, f, t)` writing into `self._name2ft`; on 3.13 and +later it is `__record_outcome(self, test, failures, tries, skips)` writing into +`self._stats` +([`Lib/doctest.py:1485`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1485)). +A loop that calls the wrong one leaves `summarize()` reporting zeros for a +passing file — a silent, total failure of the reporting path. + +**`TestResults` gained a third value that is not a tuple field.** It carries +`skipped` as an extra instance attribute +([`Lib/doctest.py:114`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L114)), +so `TestResults(f, a, skipped=s)` works on 3.13+ and raises on earlier versions. + +**`report_skip` does not exist at v3.14.2.** The runner has only `report_start`, +`report_success`, `report_failure` and `report_unexpected_exception` +([`Lib/doctest.py:1286-1314`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1286-L1314)). +It appears in later prereleases, so a loop must probe rather than assume in +either direction. + +A fourth risk has no current instance but would be silent: a CPython refactor +that inlines the loop into `run()` would route execution back to stdlib. That is +invisible for prompt-form blocks and immediately broken for `{testcode}`. + +## Question + +How is an owned per-example loop proven equivalent to the interpreter's own, +continuously, without a `sys.version_info` ladder? + +## Direction + +A differential conformance harness, run in CI on every supported interpreter, +gating the build step that lands the runner. + +A fixed case matrix — pass, fail, unexpected exception, `SyntaxError`, all +examples skipped, partially skipped, `FAIL_FAST`, `REPORT_ONLY_FIRST_FAILURE`, +`IGNORE_EXCEPTION_DETAIL` — is run through both this runner and a stock +{class}`doctest.DocTestRunner`, asserting equality of `TestResults`, the captured +`report_*` text, `summarize()` output, and the accumulator contents. + +Version handling is by capability probe, never by version comparison, so a +backport, a vendored interpreter or a fork behaves correctly rather than by +coincidence. {doc}`0001-typed-vanilla-doctest-core` rejects an import-time guard +that raises: a `pytest11` plugin that aborts at import takes down suites whose +majority of tests never touch a doctest. + +## Open + +- Whether the harness asserts on `report_*` text verbatim, or on a normalized + form — verbatim is stricter and will churn when CPython adjusts wording. +- Whether a probe failure degrades to stdlib's loop with a diagnostic, or fails + the affected items loudly. Degrading is silent for prompt-form blocks, which is + the argument against it. +- The floor: whether supporting 3.10's `_name2ft` shape is worth its shim once + that version reaches end of life. diff --git a/docs/adrs/0003-retiring-per-block-namespace-items.md b/docs/adrs/0003-retiring-per-block-namespace-items.md new file mode 100644 index 0000000..e9c7787 --- /dev/null +++ b/docs/adrs/0003-retiring-per-block-namespace-items.md @@ -0,0 +1,54 @@ +(adr-0003-retiring-per-block-namespace-items)= + +# ADR 0003: Retiring `namespace_items = per-block` + +Status: Draft +Date: 2026-08-02 + +## Context + +`doctest_docutils_namespace_items` selects between `merged` (the default: a group +collects as one test) and `per-block` (each block keeps its own node id while the +blocks share one live `globs` mapping). + +{doc}`0001-typed-vanilla-doctest-core` makes the setting unnecessary by +decoupling the two axes it was invented to trade between: one pytest item per +group, with one {class}`doctest.DocTest` per block inside it. That gives +`per-block`'s per-block failure locations, gutters and `SKIPPED` reporting +without `per-block`'s shared live mapping. + +It also removes `per-block`'s cost. A live mapping is a Python object, so it +neither crosses a worker process nor survives an item running twice. Guarding it +is the sole reason `_worker_count`, `_shared_page`, `_is_page`, +`_splitting_scheduler` and both xdist hooks exist. + +The setting's remaining distinguishing feature is a node id per block — and those +ids raise `NameError` when selected alone, because selecting one block does not +run the blocks that bound the names it reads. A node id that cannot be selected +is not a node id. + +## Question + +`per-block` is a shipped CLI option with an ini twin and its own how-to sections. +How does it get retired without breaking a user who set it? + +## Direction + +Announce it as a breaking change with the migration path stated plainly, and keep +the option accepting `per-block` for one minor release, mapped to `merged` with a +{class}`pytest.PytestDeprecationWarning`. + +Before landing, confirm no downstream consumer sets it. +`--doctest-docutils-modules` and `--no-doctest-docutils-modules` keep their exact +current spelling and `dest`, because downstream projects carry them in `addopts`. + +## Open + +- Whether the deprecation shim warns once per session or once per page. +- Whether `--doctest-docutils-namespace-scope` is renamed to `--doctest-docutils-share` + in the same release or a later one. {doc}`0001-typed-vanilla-doctest-core` + decides the vocabulary — `scope` is reserved for pytest's fixture-lifetime + ladder — but the rename is independently schedulable. +- Whether the `pytest11` entry point rename (from `sphinx`, which is what + `-p no:sphinx` disables today) ships in the same release as this removal or its + own. diff --git a/docs/adrs/0004-diagnostics-as-data.md b/docs/adrs/0004-diagnostics-as-data.md new file mode 100644 index 0000000..0eb82a1 --- /dev/null +++ b/docs/adrs/0004-diagnostics-as-data.md @@ -0,0 +1,63 @@ +(adr-0004-diagnostics-as-data)= + +# ADR 0004: Diagnostics as data + +Status: Draft +Date: 2026-08-02 + +## Context + +Parsing a page currently writes docutils reporter output straight to stderr, +interleaved with pytest's own output and attributable to nothing. Two failure +modes follow from the default settings. + +A level-4 message raises `SystemMessage` mid-parse and aborts collection of the +file, so one malformed construct takes down a page whose other blocks are fine. + +More quietly, a `.. doctest::` block carrying an unknown option collects **zero** +tests and the session exits green. A page that checks nothing reports the same +way as a page that passes. + +{doc}`0001-typed-vanilla-doctest-core` gives the front-end layer a second return +value for this: `Diagnostic(level, code, message, path, line)`, produced by +setting `halt_level` past the abort threshold and attaching a reporter observer. +Suppression and promotion key on the stable `code`, never on message text. + +## Question + +Which diagnostics are shown by default? + +The naive answer — show everything — was measured against this project's own +`docs/` and produces well over a hundred messages per run, almost all of them +`Unknown interpreted text role` and `Unknown directive type` for roles and +directives that Sphinx supplies and a bare-docutils parse structurally cannot +resolve. Those are false positives. Emitting them is noise-as-policy, and users +would learn to ignore the channel that also carries real errors. + +The opposite error is worse: suppressing one code too many turns a broken page +into a silent zero-test page, which is the exact condition this ADR exists to +surface. + +## Direction + +Suppress by code, narrowly, and only for the two classes a bare-docutils parse +cannot judge: unknown roles and unknown directives. + +Every diagnostic raised by this project's own layers defaults to visible, and +`level="error"` from those layers fails collection with the file and line named. +A page whose only block fails to parse, and a page with a malformed `:options:` +value, must both produce a collection error rather than collecting nothing and +passing. + +Expose promotion and suppression by code so a project can tune the set without a +global on/off switch. + +## Open + +- Whether diagnostics surface as {class}`pytest.PytestWarning` subclasses, giving + `-W error::` control for free, or as a dedicated report section. +- Whether an unknown-directive diagnostic should be promoted when the directive + name is one this project registers — that case is not Sphinx supplying it, it + is a registration that did not happen, which is the GH-48 failure mode. +- Whether the CLI (`python -m doctest_docutils`) and the pytest plugin share one + formatter or two. diff --git a/docs/adrs/0005-line-recovery-for-nested-blocks.md b/docs/adrs/0005-line-recovery-for-nested-blocks.md new file mode 100644 index 0000000..f24616b --- /dev/null +++ b/docs/adrs/0005-line-recovery-for-nested-blocks.md @@ -0,0 +1,59 @@ +(adr-0005-line-recovery-for-nested-blocks)= + +# ADR 0005: Exact line recovery for nested blocks + +Status: Draft +Date: 2026-08-02 + +## Context + +docutils does not report a usable line for every node, and what it does report +differs by front-end. + +A bare `>>>` block nested in a `.. note::`, a list item, a block quote or a +`{tab}` directive reports `line=None, source=None`. A top-level reStructuredText +`doctest_block` reports its **last** line; a MyST fence reports its **first**. An +`.. include::`-ed block numbers against the *included* file. + +{doc}`0001-typed-vanilla-doctest-core` handles all three honestly rather than +approximately. `Block.line` is nullable, the per-front-end meaning is normalized +inside the front-end that knows it, `Block.path` carries the file the text +actually lives in, and a block with no recoverable line propagates +`DocTest.lineno=None` into pytest's `EXAMPLE LOCATION UNKNOWN` branch. + +That is correct but not maximal. A nested block's failure says the location is +unknown when the information exists in the parser and was discarded. + +## Question + +Can the exact line be recovered for a nested block without a fragile dependency? + +The mechanism identified is per-instance substitution of +`docutils.parsers.rst.Parser.state_classes`. `state_classes` is an instance +attribute, so substitution is fully scoped to one parse with no process-global +mutation — unlike the directive registry, which has no such scoping. + +It also depends on undocumented docutils structure, and a refactor upstream would +break it as `None` spans rather than as an exception. + +## Direction + +Land it last, as a strictly optional improvement behind a feature probe. + +When the expected structure is absent, fall back to `node.line` normalization and +emit `line=None`. Degrading to an honest disclaimer is acceptable; degrading to a +fabricated number is not, and that is the whole reason this is separable from +{doc}`0001-typed-vanilla-doctest-core` rather than part of it. + +Pin with tests asserting exact `(path, line)` for a bare block nested in each of +the four constructs, plus a test that forces the probe to fail and asserts the +fallback yields `lineno=None`. + +## Open + +- Whether the same technique gives MyST nested blocks anything, or whether + markdown-it's token stream already carries enough. +- Whether `.. include::` line attribution needs the same treatment or is already + correct once `Block.path` is respected. +- Whether the probe result is reported anywhere, so a user can tell which mode + produced a given report. diff --git a/docs/adrs/0006-pytest-private-api-compatibility.md b/docs/adrs/0006-pytest-private-api-compatibility.md new file mode 100644 index 0000000..943e9d9 --- /dev/null +++ b/docs/adrs/0006-pytest-private-api-compatibility.md @@ -0,0 +1,81 @@ +(adr-0006-pytest-private-api-compatibility)= + +# ADR 0006: pytest private API compatibility + +Status: Draft +Date: 2026-08-02 + +## Context + +The plugin currently calls `config.pluginmanager.set_blocked("doctest")` and then +imports that same blocked plugin's private helpers, while continuing to read four +ini and CLI options the blocked plugin declared. It works only because +`_pytest/fixtures.py` has no `pytest_plugin_unregistered` handler, so the +already-parsed `doctest_namespace` fixture outlives unregistration. That is an +undocumented behaviour bet on for every collected page. + +{doc}`0001-typed-vanilla-doctest-core` inverts the relationship: the built-in +doctest plugin is not blocked, it is *required*. Its checker, its failure repr, +its `--doctest-report` formatting and `doctest_namespace` are all worth keeping, +and reimplementing them was measured as costing roughly three times what it was +budgeted at — `_get_checker` alone returns a checker implementing +`ALLOW_UNICODE`, `ALLOW_BYTES` and `NUMBER` with float-precision handling +([`_pytest/doctest.py:662`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L662)). + +Not blocking it exposes a live defect the block was masking. +[`_is_doctest`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L148-L152) +claims any `.txt` or `.rst` **initial path before consulting `--doctest-glob`**: + +```python +def _is_doctest(config: Config, path: Path, parent: Collector) -> bool: + if path.suffix in (".txt", ".rst") and parent.session.isinitpath(path): + return True + globs = config.getoption("doctestglob") or ["test*.txt"] + return any(fnmatch_ex(glob, path) for glob in globs) +``` + +So `pytest docs/page.rst` is claimed by the built-in plugin regardless of glob +configuration, and `pytest_collect_file` is not `firstresult` — the directory +collector yields the results of *every* implementation for a path. Declining the +path is therefore not sufficient; the duplicate has to be removed. + +## Question + +What private surface is depended on, and how does a pytest release that changes +it fail? + +The current surface is `_get_checker`, `get_optionflags`, +`_get_continue_on_failure`, `_get_report_choice`, `MultipleDoctestFailures`, and +a `pytest.DoctestItem` subclass. `_init_runner_class` is explicitly *not* usable: +`PytestDoctestRunner` is defined inside it +([`_pytest/doctest.py:178-181`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L178-L181)) +and is unreachable by name, which is why the `OutcomeException` re-raise, +`BdbQuit` → `outcomes.exit` and `continue_on_failure` handling must be +reimplemented rather than inherited. + +## Direction + +Quarantine every private import in one module, `pytest_doctest_docutils._compat`, +with a pinned support matrix and an import-time probe that raises a named error at +plugin registration — naming the pytest version and the missing symbol — rather +than failing somewhere in the middle of collection. + +Claim `.rst` and `.md` paths and deselect the built-in plugin's duplicate items in +a `pytest_collection_modifyitems` hookwrapper, firing `pytest_deselected` so the +reported counts stay consistent. + +CI carries a job pinned to the minimum supported pytest and one tracking its +prerelease. + +## Open + +- Whether requiring the built-in plugin should be stated as a hard dependency. + `-p no:doctest` already fails today with a raw `ValueError` about an unknown + option, so this is not a regression — but the message should become actionable. +- Whether deselecting is preferable to narrowing the built-in's globs. Deselecting + is explicit and reportable; narrowing is quieter but mutates another plugin's + configuration. +- Whether the probe should accept a *newer* pytest it has not been tested against, + or refuse it. Refusing is safer and more annoying. +- Whether any of these helpers can be promoted upstream, which would delete the + quarantine entirely. diff --git a/docs/adrs/index.md b/docs/adrs/index.md index 273990e..ed763f1 100644 --- a/docs/adrs/index.md +++ b/docs/adrs/index.md @@ -25,3 +25,14 @@ tag, or a commit reachable from that project's trunk where it publishes no tags. Line anchors are only used on a pinned ref, because they are meaningless without one, and a `blob/master` link rots silently — the file moves, lines shift, and the anchor lands on unrelated code while still resolving. + +```{toctree} +:maxdepth: 1 + +0001-typed-vanilla-doctest-core +0002-runner-conformance-across-cpython +0003-retiring-per-block-namespace-items +0004-diagnostics-as-data +0005-line-recovery-for-nested-blocks +0006-pytest-private-api-compatibility +``` From d88f1de9785116c3d38f5c51e8341d742145d9be Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 10:11:17 -0500 Subject: [PATCH 03/31] notes(analyses): Add the structural research behind ADR 0001 why: ADR 0001 asserts constraints about six upstream projects. Those assertions need a reviewable derivation, or the next person to question one has to redo the reading. Keeping it beside the record rather than inside it also keeps the record about the decision. what: - Add a per-system structural doc for CPython doctest, _pytest.doctest, pytest-xdist, pytest-asyncio, sphinx.ext.doctest, and docutils with myst-parser, each in the same section order so they compare - Add asyncio as the stdlib's own worked example of a pluggable architecture, for contrast with doctest's four seams - Add prior art on Sybil, xdoctest, pytest-examples and typeshed - Add cross-cutting docs on data structures, data flows, extension seams, and the namespace-scope versus test-identity axis - Add a bibliography collecting every anchor in one place - Pin every citation to a tag, or to a trunk-reachable commit where a project publishes none --- notes/analyses/00-taxonomy.md | 79 +++++++++ notes/analyses/10-cpython-doctest.md | 151 ++++++++++++++++ notes/analyses/11-pytest-doctest.md | 127 +++++++++++++ notes/analyses/12-pytest-xdist.md | 136 ++++++++++++++ notes/analyses/13-pytest-asyncio.md | 112 ++++++++++++ notes/analyses/14-asyncio.md | 109 ++++++++++++ notes/analyses/15-sphinx-ext-doctest.md | 127 +++++++++++++ notes/analyses/16-docutils-myst.md | 130 ++++++++++++++ notes/analyses/17-prior-art.md | 167 ++++++++++++++++++ notes/analyses/20-data-structures.md | 86 +++++++++ notes/analyses/21-data-flows.md | 113 ++++++++++++ notes/analyses/22-extension-seams.md | 106 +++++++++++ .../23-namespace-scope-and-test-identity.md | 127 +++++++++++++ notes/analyses/90-bibliography.md | 162 +++++++++++++++++ notes/analyses/README.md | 69 ++++++++ 15 files changed, 1801 insertions(+) create mode 100644 notes/analyses/00-taxonomy.md create mode 100644 notes/analyses/10-cpython-doctest.md create mode 100644 notes/analyses/11-pytest-doctest.md create mode 100644 notes/analyses/12-pytest-xdist.md create mode 100644 notes/analyses/13-pytest-asyncio.md create mode 100644 notes/analyses/14-asyncio.md create mode 100644 notes/analyses/15-sphinx-ext-doctest.md create mode 100644 notes/analyses/16-docutils-myst.md create mode 100644 notes/analyses/17-prior-art.md create mode 100644 notes/analyses/20-data-structures.md create mode 100644 notes/analyses/21-data-flows.md create mode 100644 notes/analyses/22-extension-seams.md create mode 100644 notes/analyses/23-namespace-scope-and-test-identity.md create mode 100644 notes/analyses/90-bibliography.md create mode 100644 notes/analyses/README.md diff --git a/notes/analyses/00-taxonomy.md b/notes/analyses/00-taxonomy.md new file mode 100644 index 0000000..0f6bf22 --- /dev/null +++ b/notes/analyses/00-taxonomy.md @@ -0,0 +1,79 @@ +# Taxonomy: the axes a doctest engine is classified on + +Nine axes. Every system in these notes takes a position on each, and most of the +disagreements between them reduce to a different position on one axis rather than +a different philosophy. + +## The axes + +| # | Axis | Positions | +|---|---|---| +| 1 | **Sharing unit vs. selection unit** | same object · different objects, acknowledged · different objects, unacknowledged | +| 2 | **Test identity** | author-declared name · symbol-derived · positional (line/column or byte range) | +| 3 | **Runtime object model** | stdlib `DocTest`/`Example` · own model with a bridge · own model, no bridge | +| 4 | **Document model** | real parse tree · flat character spans · regex over text · none | +| 5 | **Option representation** | `int` bitmask · structured state · enum | +| 6 | **Extension mechanism** | callable aliases · nominal subclassing · named registry · `Protocol` · none | +| 7 | **Relationship to pytest's doctest plugin** | compose · block/unregister · replace by instruction · no collector | +| 8 | **Got/want strictness** | stdlib defaults · permissive defaults · no want at all | +| 9 | **Direction of data** | read-only · read plus write-back | + +## Where each system sits + +| System | 1 sharing/selection | 2 identity | 3 object model | 4 document model | +|---|---|---|---|---| +| CPython `doctest` | same (one `DocTest` per docstring) | dotted symbol path | *is* the model | none — line regex over a string | +| `_pytest.doctest` | same (one item per `DocTest`) | `path::module.qualname` | stdlib, unchanged | none — delegates | +| `sphinx.ext.doctest` | same (one group, one runner pass) | group name, shared by every block | stdlib, constructed per group | real doctree | +| Sybil | **different, unacknowledged** | positional `line:N,column:N` | stdlib `Example`, one-line `DocTest` fork | flat character spans | +| xdoctest | same (one `DocTest` per docstring) | `Callname:N` | own, with a late bridge back | none for `.rst`/`.txt` | +| pytest-examples | none — no implicit sharing | positional `path:start-end` | none | regex over fences | +| `doctest_docutils` today | configurable; `per-block` is different-and-guarded | group name, or `page.md[k]` | stdlib | real doctree | +| ADR 0001 | **different, decoupled by construction** | group name, or `page.md[k]` | stdlib | real doctree | + +| System | 5 options | 6 extension | 7 vs. pytest doctest | 8 strictness | 9 direction | +|---|---|---|---|---|---| +| CPython `doctest` | `int` bitmask + registry | nominal subclassing | n/a | strict | read-only | +| `_pytest.doctest` | `int` + name lookup | subclass its classes | *is* it | strict | read-only | +| `sphinx.ext.doctest` | `int` via `:options:` | directive subclassing | unaware | strict | read-only | +| Sybil | `int` | callable aliases | replace by instruction (`-p no:doctest`) | strict | read-only | +| xdoctest | structured `TypedDict` + bridge | two registries, else fork | unregisters it | **permissive** | read-only | +| pytest-examples | n/a | none | no collector — composes trivially | no want at all | **write-back** | +| `doctest_docutils` today | `int` + a forked lookup | directive subclassing | blocks it, imports its privates | strict | read-only | +| ADR 0001 | `int` + registry | `Protocol` + nominal + `BlockKind` registry | **compose; require it** | strict | read-only | + +## What the matrix shows + +**Axis 1 is the only one where a wrong answer is silent.** Every other axis +produces inconvenience — a renamed test, a conversion layer, an extra knob. Axis 1 +produces a `NameError` in a test the user believed they could select, or a false +green under `--reruns`. Sybil sits in the unacknowledged column and its +documentation never mentions it. + +**Axes 1 and 2 are independent, and everyone treated them as one.** The full +product space is four cells: + +| | one node id | N node ids | +|---|---|---| +| **one `DocTest`** | Sphinx, `doctest_docutils` `merged` | — (incoherent) | +| **N `DocTest`s** | *unoccupied until ADR 0001* | Sybil, `doctest_docutils` `per-block` | + +The bottom-right cell is where the silent failure lives. The bottom-left cell — +per-block `DocTest`s under one item — gives per-block reporting *and* an +unsplittable sharing unit, and no surveyed project occupies it. + +**Axis 3 has an empirical answer.** xdoctest is the controlled experiment for +abandoning the stdlib object model, and it is now building the bridge back. The +cost of divergence is paid years later, in knobs that exist only to restore the +default that was abandoned. + +**Axis 4 is decided by axis 6.** A regex cannot see `:skipif:` or a group name, +which is why Sybil has no group concept at all and tells users to clear the +namespace instead. If directive options are part of the product, the document +model must be a parse tree. + +**Axis 7 correlates with hostility.** Two of the surveyed projects disable +pytest's doctest plugin — one in `pytest_configure`, one by telling users to pass +`-p no:doctest`. A `pytest11` plugin loads into sessions belonging to people who +never asked for it, and the plugin it disables is the one whose checker, failure +repr and `doctest_namespace` fixture it wants to keep. diff --git a/notes/analyses/10-cpython-doctest.md b/notes/analyses/10-cpython-doctest.md new file mode 100644 index 0000000..8d942d6 --- /dev/null +++ b/notes/analyses/10-cpython-doctest.md @@ -0,0 +1,151 @@ +# CPython `doctest` + +Pinned at [`v3.14.2`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py). + +## Classification + +A four-stage pipeline of unannotated classes, whose documented extension surface +is six classes plus three injection slots and four reporting hooks. The per-example +loop — the thing every extender eventually wants — is not among them. + +## Core data structures + +```text +Example source, want, exc_msg, lineno, indent, options + | lineno is 0-based, relative to the start of the containing string + v +DocTest examples, globs, name, filename, lineno, docstring + | globs is COPIED by __init__; __lt__ compares name as TEXT + v +TestResults namedtuple(failed, attempted), with `skipped` as an EXTRA attribute +``` + +Three properties of these are load-bearing for anything built on top: + +**`DocTest.__init__` copies the globs mapping** +([`:565`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L565)). +Passing a shared dict through the constructor has no effect whatsoever. A shared +mapping must be assigned to `test.globs` after construction, and the runner must +be given `clear_globs=False` or it empties the mapping in its `finally`. + +**`__lt__` compares names as text** +([`:596`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L596)). +Any name carrying a position as text sorts `[10]` before `[1]`. This fails +silently: every test passes, in the wrong order. + +**`TestResults` carries `skipped` outside the tuple** +([`:114`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L114)), +with a `repr` that falls back to the plain namedtuple form when it is zero. +Promoting it to a third field would break every `failures, tries = runner.run(...)` +unpack in the ecosystem, including `doctest._test()` itself. + +## Data flow + +```text +source string + | DocTestParser.parse -> list[str | Example], alternating, + | reconstructing the input exactly + | DocTestParser.get_doctest -> DocTest + v +DocTestFinder.find(obj) -> list[DocTest] + | recurses into __test__, tracks a seen-map by id() + v +DocTestRunner.run(test, compileflags, out, clear_globs) + | saves sys.stdout, pdb.set_trace, linecache.getlines, + | sys.displayhook, _colorize.can_colorize; pops PYTHON_COLORS + | and FORCE_COLOR from os.environ; restores all in finally + | + +--> __run(test, compileflags, out) <- name-mangled + for each example: + merge test-level and example-level optionflags + SKIP? -> continue BEFORE report_start; still counts as attempted + compile(source, "" % (test.name, n), + "single", flags, dont_inherit=True) + exec in test.globs + OutputChecker.check_output(want, got, flags) + report_success / report_failure / report_unexpected_exception + __record_outcome(...) +``` + +The `parse()` contract is not incidental. It must return alternating `str` and +`Example` reconstructing the input, because `script_from_examples()` walks the +`str` pieces to build the prose comments in a debugging script. `Example.indent` +is computed against the original string — after `expandtabs`, before dedent — so a +post-dedent indent shifts every reported column. + +## Extension seams + +| Seam | Kind | Documented | +|---|---|---| +| `DocTestParser` subclass, injected as `parser=` | nominal | yes | +| `DocTestFinder` subclass, injected as `test_finder=` | nominal | yes | +| `OutputChecker.check_output` / `output_difference`, injected as `checker=` | nominal | yes | +| `report_start`, `report_success`, `report_failure`, `report_unexpected_exception` ([`:1286-1314`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1286-L1314)) | subclass hook | yes | +| `register_optionflag` / `OPTIONFLAGS_BY_NAME` ([`:153`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L153)) | process-global registry | yes | +| `setUp` / `tearDown` on `DocTestSuite` / `DocFileSuite` | callable param | yes | +| `__run` ([`:1344`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1344)) | **name-mangled** | no | +| `__record_outcome` ([`:1485`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1485)) | **name-mangled** | no | +| `__patched_linecache_getlines` ([`:1501`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1501)) | **name-mangled** | no | +| `_load_testfile` ([`:245`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L245)), `_EXAMPLE_RE` ([`:618`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L618)) | private | no | + +The name-mangled three are the interesting entry. Mangling rewrites the *call +site* at compile time, so `self._DocTestRunner__run(...)` inside `run()` is an +ordinary attribute lookup that resolves through the subclass's MRO. Defining +`_DocTestRunner__run` in a subclass therefore takes over the loop — verified on +3.14.2 with `run()` untouched. It is not an override point by *design*, but it is +one by *mechanism*, and that distinction is what lets a downstream own the loop +without cloning a code object or patching a module global. + +`register_optionflag` is the only genuinely cross-library extension point in the +module. Its ints are `1 << len(OPTIONFLAGS_BY_NAME)`, so they are +registration-order dependent, typeshed hard-codes the builtin values, and an +unregistered flag name makes a page fail to **parse** rather than to run. + +## Configuration + +There is none, in the modern sense. Behaviour is set by optionflags, which arrive +from three places with a fixed precedence: the runner's constructor, the +`DocTest`'s per-example `options` dict, and the inline `# doctest: +FLAG` comment +parsed out of the example source. `set_unittest_reportflags` mutates a module +global. `doctest.master` accumulates results across invocations — the +documentation calls it advanced tomfoolery. + +## What it cannot do + +- **Run a multi-statement body.** `"single"` mode rejects it, and `"exec"` mode + suppresses expression echo, which empties every `want`. This one fact is the + origin of every downstream monkeypatch of `doctest.compile`. +- **Report a per-example result as a value.** Outcomes exist only as counters and + as text pushed through `out`. pytest works around this by repurposing `out` from + a write-callable into a *list*; Sphinx works around it by not producing machine- + readable results at all. +- **Share a globs mapping across `DocTest`s** without the caller assigning + `test.globs` post-construction and passing `clear_globs=False`. +- **Be reentrant or thread-safe.** `run()` mutates interpreter globals for its + duration ([`:1534-1573`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1534-L1573)). +- **Report a skip as an outcome.** `SKIP` short-circuits before `report_start`. + There is no `report_skip` at v3.14.2; it appears in later prereleases, so a + downstream loop must probe rather than assume. + +## Anchors + +- [`TestResults`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L114) · + [`register_optionflag`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L153) · + [`_load_testfile`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L245) +- [`DocTest.__init__` globs copy](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L565) · + [`DocTest.__lt__`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L596) +- [`DocTestParser`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L609) · + [`_EXAMPLE_RE`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L618) · + [`DocTestFinder`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L844) +- [`report_*` hooks](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1286-L1314) · + [`__run`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1344) · + [`compile(..., "single", ...)`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1400) +- [`__record_outcome`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1485) · + [`__patched_linecache_getlines`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1501) · + [`run()` global-state save/restore](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1534-L1573) +- [`OutputChecker`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1690) · + [`DebugRunner`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1874) · + [`testfile`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L2091) · + [`DocTestSuite`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L2467) · + [`DocFileSuite`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L2570) +- Typed contract: [`typeshed stdlib/doctest.pyi`](https://github.com/python/typeshed/blob/8c7256c/stdlib/doctest.pyi) diff --git a/notes/analyses/11-pytest-doctest.md b/notes/analyses/11-pytest-doctest.md new file mode 100644 index 0000000..5f616fa --- /dev/null +++ b/notes/analyses/11-pytest-doctest.md @@ -0,0 +1,127 @@ +# `_pytest.doctest` + +Pinned at [`9.1.1`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py). + +## Classification + +A deliberately thin adapter. It owns two collectors and one item, and delegates +every piece of domain knowledge to stdlib `DocTest`/`Example`/`DocTestFailure`. It +overrides exactly the seams it needs and invents no parallel abstraction. It is +the reference implementation of how to integrate with `doctest` rather than +replace it, and the model this project's pytest layer should resemble. + +## Core data structures + +```text +DoctestItem(Item) dtest: DocTest, runner: DocTestRunner, fixture_request + | obj = None (class attribute) +DoctestTextfile(Module) obj = None; one DocTest for the whole file +DoctestModule(Module) one DocTest per docstring; calls parsefactories +MultipleDoctestFailures carries a list; the workaround for stdlib having no + per-example result value +ReprFailDoctest (ReprFileLocation, lines) pairs +``` + +`obj = None` as a *class* attribute +([`:421`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L421)) +is what keeps `Module` from trying to import a `.txt`/`.rst` file. Subclassing +`Module` rather than `File` is what makes `scope="module"` fixtures resolve +against the page — a page collector *is* the module scope. `parsefactories` +([`:556`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L556)) +is what makes autouse fixtures from a visible `conftest.py` apply at all. + +## Data flow + +```text +pytest_collect_file(file_path, parent) [:126] + | .py -> DoctestModule (when --doctest-modules) + | else -> DoctestTextfile (when _is_doctest) + v +Collector.collect() -> DoctestItem.from_parent(...) per non-empty DocTest + | an EMPTY DocTest is never yielded [:451] + v +DoctestItem.setup() [:288] + | fixture request is filled, then: self.dtest.globs.update(globs) + | -> the mapping must be MUTABLE and must survive collection + v +DoctestItem.runtest() [:295] + | _check_all_skipped(self.dtest) -> outcomes.skip if every example is SKIP + | self.runner.run(self.dtest, out=failures) + | ^^^ `out` is a LIST, not a write-callable. + | clear_globs defaults to True. + | raise MultipleDoctestFailures(failures) + v +DoctestItem.repr_failure(excinfo) [:317] + for failure in failures: [:337] + lineno = test.lineno + example.lineno + 1 [:344] +``` + +Two details in that flow decide a great deal for anything built on it. + +**`out` is repurposed as a list.** The most important consumer of stdlib's runner +deliberately violates typeshed's `_Out = Callable[[str], object]`, marked with a +`# type: ignore[arg-type]`, so that `report_failure` can append rather than write. +Any claim that a "typed vanilla core" can narrow `out` honestly has to reckon with +the fact that the ecosystem's largest caller does not. + +**`repr_failure` reads each failure's own `test`.** Locations are computed +per failure inside the loop, not once per item. That is what makes N `DocTest`s +under one item report N correct locations with no override — the fact ADR 0001 is +built on. + +## Extension seams + +| Seam | Kind | +|---|---| +| `--doctest-modules`, `--doctest-glob`, `--doctest-continue-on-failure`, `--doctest-report`, `doctest_optionflags`, `doctest_encoding` | ini/CLI, declared by the always-loaded plugin | +| `doctest_namespace` session fixture ([`:721`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L721)) | fixture | +| `_get_flag_lookup` ([`:385`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L385)) | private; lazily registers `ALLOW_UNICODE`, `ALLOW_BYTES`, `NUMBER` | +| `_get_checker` ([`:662`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L662)) | private; returns the checker implementing those flags | +| `_get_continue_on_failure` ([`:410`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L410)), `_get_report_choice` ([`:703`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L703)) | private | +| `PytestDoctestRunner` ([`:181`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L181)) | **unreachable** — defined inside `_init_runner_class()` ([`:178`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L178)) | + +That last row matters more than it looks. `PytestDoctestRunner` is where the +`OutcomeException` re-raise, the `bdb.BdbQuit` → `outcomes.exit` conversion, and +the `continue_on_failure` buffering live. Because it is defined inside a function +and never bound at module scope, a downstream cannot import or subclass it. Any +design that assumes those behaviours "come for free" by subclassing `DoctestItem` +is wrong: the item supplies the *plumbing*, but the runner supplies the +*behaviour*, and only the plumbing is reachable. + +## Configuration + +`pytest_addoption` in this module declares six settings that the plugin reads back +through helpers. A third-party plugin may **read** them but must not re-declare +them — re-adding an existing option raises at option-parsing time. Conversely, +suppressing the built-in plugin before `pytest_configure` (`-p no:doctest`) +removes the options entirely, and any downstream read of them then fails. + +## What it cannot do + +- **Collect a page with directives.** It has no document model at all; a `.rst` + file is one string handed to `DocTestParser`. +- **Share state across items.** `runtest()` runs with `clear_globs=True`. +- **Decline a path it has claimed.** `_is_doctest` + ([`:148-152`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L148-L152)) + returns `True` for any `.txt`/`.rst` **initial path before consulting + `--doctest-glob`**, and `pytest_collect_file` is not `firstresult`, so a + third-party collector claiming the same path gets its items collected + *alongside* — not instead of — the built-in's. + +## Anchors + +- [`pytest_collect_file`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L126) · + [`_is_doctest`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L148-L152) · + [`_is_setup_py`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L141) · + [`_is_main_py`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L155) +- [`_init_runner_class`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L178) · + [`PytestDoctestRunner`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L181) · + [`MultipleDoctestFailures`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L172) +- [`DoctestItem`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L251) · + [`setup`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L288-L293) · + [`runtest`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L295-L303) · + [`repr_failure`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L317-L344) +- [`get_optionflags`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L401) · + [`_check_all_skipped`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L451) · + [`DoctestTextfile`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L420) · + [`DoctestModule`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L500) diff --git a/notes/analyses/12-pytest-xdist.md b/notes/analyses/12-pytest-xdist.md new file mode 100644 index 0000000..2f7cf74 --- /dev/null +++ b/notes/analyses/12-pytest-xdist.md @@ -0,0 +1,136 @@ +# pytest-xdist + +Pinned at [`v3.8.0`](https://github.com/pytest-dev/pytest-xdist/tree/v3.8.0). + +## Classification + +A controller/worker distribution layer over execnet. All scheduling is integer +indices into a per-worker collection list, and the controller **never collects**. +That single asymmetry is the source of every constraint xdist imposes on a plugin +that wants to keep related tests together. + +## Core data structures + +```text +controller worker (one per process) + NodeManager -> specs: list[str] session collects normally + Scheduling implementation reports node ids back as STRINGS + node2collection: dict[node, list[str]] + node2pending: dict[node, list[int]] <- integer indices, not ids + collection: list[str] <- the agreed id list +``` + +The controller's entire model of the suite is a list of node-id strings that +arrived from a worker. It has no items, no marks, no fixtures and no knowledge of +what any test does. A plugin that needs "these tests share state" therefore cannot +tell the controller so directly — it can only encode the fact *into the node id* +or infer it from string shape. + +## Data flow + +```text +pytest_cmdline_main xdist promotes -n N into --dist load (tryfirst) + | +pytest_sessionstart -> NodeManager.setup_nodes + | pytest_xdist_setupnodes(config, specs) <- specs already expanded + v +each worker collects independently + | + +-> pytest_xdist_node_collection_finished(node, ids) + | + v +Scheduling.add_node_collection(node, ids) + | every worker's list must be IDENTICAL, in the same ORDER + | mismatch -> log "**Different tests collected, aborting run**" + | and assign nothing. Zero tests execute. + v +Scheduling.schedule() -> send integer index batches to workers + | + v (on worker crash) + only the UNCOMPLETED items of the crashed work unit are re-sent + to a FRESH worker with FRESH process state +``` + +The abort path is the constraint that matters most. It is not an exception and it +is not loud in the usual sense: the scheduler logs a line, assigns nothing, and +the session ends having run nothing. Any collection-time decision that is not a +pure function of (files on disk, argv, ini) — a timestamp, a PID, a hostname, a +dict iteration order, an evaluated `:skipif:` that depends on the environment — +produces this. + +The crash path is the second. Because only uncompleted items of a work unit are +retried, a group whose blocks 1-2 ran before the crash has blocks 3..N re-run +against an empty process, producing a `NameError` cascade attributed to the wrong +cause. Worker restarts are on by default. + +## Extension seams + +| Seam | Kind | +|---|---| +| `pytest_xdist_make_scheduler(config, log)` | hook — substitute a `Scheduling` implementation | +| `pytest_xdist_node_collection_finished(node, ids)` | hook — observe the agreed id list | +| `pytest_xdist_setupnodes(config, specs)` | hook — receives the already-expanded spec list; never raises | +| `pytest_xdist_auto_num_workers(config)` | hook | +| `LoadScopeScheduling._split_scope(nodeid) -> str` ([`loadscope.py:284`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L284)) | subclass hook — **the only affinity primitive in the codebase** | +| `@pytest.mark.xdist_group(name)` | marker, honoured only under `--dist loadgroup` | + +`_split_scope` is worth stating plainly: it is a pure function from a node-id +string to a scope string, and both shipped grouping modes are two-line overrides +of it — [`loadfile.py:35`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadfile.py#L35) +returns the file part, [`loadgroup.py:24`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadgroup.py#L24) +returns the `@`-suffix. `load` and `worksteal` have **no scope concept at any +layer**: `load` slices `pending[:num]` and `worksteal` steals a raw suffix. + +So under a user-typed `--dist load`, a plugin has exactly three options: refuse +the run, substitute the scheduler, or make the group not need protecting. There +is no "declare affinity and let the chosen scheduler honour it" API. + +The `xdist_group` marker is narrower than it appears. It is applied +[worker-side](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/remote.py#L245-L254), +and only when the worker's own literal `--dist` string is `loadgroup`; it works by +appending `@` to `item._nodeid`. A controller-side scheduler substitution +never reaches a worker, so no `@` suffix is written and every item becomes its own +scope — strictly worse than plain `load`. Node ids copied from a `loadgroup` run +also do not select under `-n0`. + +## Configuration + +`-n`, `--dist`, `--tx`, `--maxprocesses`, `--max-worker-restart`. Worker counts +come from +[`parse_tx_spec_config`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/workermanage.py#L26-L37), +which builds a *list*: + +```python +xspeclist.extend([xspec[i + 1 :]] * num) +``` + +List multiplication by a negative number yields an empty list, so a negative +multiplier contributes **zero** specs. A re-implementation that sums the integer +instead contributes a negative number, and `--tx -1*popen --tx 2*popen` then +counts 1 where xdist counts 2 — a divergence whose failure direction is +permissive. + +`parse_tx_spec_config` raises `pytest.UsageError` when a run names no environment, +so it cannot be called defensively. `pytest_xdist_setupnodes(config, specs)` is +the safe source of the same information: it receives the already-expanded list, +fires during `pytest_sessionstart` — strictly before `pytest_xdist_make_scheduler` +— and never raises. + +## What it cannot do + +- **Ship a Python object between processes.** Only execnet-serializable builtins + cross. A live `globs` mapping cannot be shared, which is the whole reason a + shared doctest namespace is a distribution problem. +- **Tell the controller what an item is.** The controller sees strings. +- **Preserve process state across a worker restart** for the uncompleted tail of a + work unit. + +## Anchors + +- [`parse_tx_spec_config`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/workermanage.py#L26-L37) +- [`_split_scope`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L284) · + [`loadfile`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadfile.py#L35) · + [`loadgroup`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadgroup.py#L24) +- [`load.schedule` abort](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/load.py#L259) · + [`loadscope.schedule` abort](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L359) +- [`xdist_group` node-id append](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/remote.py#L245-L254) diff --git a/notes/analyses/13-pytest-asyncio.md b/notes/analyses/13-pytest-asyncio.md new file mode 100644 index 0000000..ecdd322 --- /dev/null +++ b/notes/analyses/13-pytest-asyncio.md @@ -0,0 +1,112 @@ +# pytest-asyncio + +Pinned at [`v1.4.0`](https://github.com/pytest-dev/pytest-asyncio/tree/v1.4.0). + +Read here as an **idiom exemplar**, not for async semantics. It is a mature, +widely-installed `pytest11` plugin that solves the same shape of problem this +project has: a per-item resource with a configurable lifetime, an opt-in mode, and +a default it needed to change without breaking anyone. + +## Classification + +A hook-driven behaviour plugin. It contributes no collector of its own for the +common case; it normalizes configuration into one question, answers it once, and +then injects behaviour through pytest's existing machinery rather than owning the +item class. + +## Core data structures + +```text +Mode(str, enum.Enum) AUTO | STRICT [:82] +PytestAsyncioSpecs its own hookspec namespace [:90] + pytest_asyncio_loop_factories(config, item) -> Mapping | None firstresult +_ScopeName reuses pytest's scope vocabulary verbatim +``` + +`Mode` inheriting `str` is a small but deliberate choice: the ini value, the CLI +value and the internal enum are the same object, so no conversion layer exists to +drift. + +Declaring its own `HookspecMarker("pytest")` namespace is the interesting one. A +third party extends pytest-asyncio by implementing a hook, not by subclassing +anything and not by mutating a registry — which sidesteps both the nominal-typing +trap and the process-global-registry trap. + +## Data flow + +```text +pytest_addoption --asyncio-mode + asyncio_mode ini [:108] + | asyncio_default_fixture_loop_scope [:137] + | asyncio_default_test_loop_scope [:143] + | every one declared with default=None + v +pytest_configure validate scopes; addinivalue_line for the marker [:295] + | an unset default is DETECTED, not silently assumed + v +_get_asyncio_mode(config) -> Mode, resolved once [:222] + | + v +in AUTO mode: item.add_marker("asyncio") + | => the rest of the plugin has exactly ONE query path + v +fixture/loop resolution by scope, then pyfunc call wrapping +``` + +## Extension seams + +| Seam | Kind | +|---|---| +| `asyncio_mode` ini + `--asyncio-mode` CLI | configuration | +| `@pytest.mark.asyncio` | marker, registered via `addinivalue_line` | +| `asyncio_default_fixture_loop_scope`, `asyncio_default_test_loop_scope` | configuration, reusing pytest's scope names | +| `pytest_asyncio_loop_factories` | its own `firstresult` hookspec | +| `@pytest_asyncio.fixture(loop_scope=...)` | decorator, stamping `_loop_scope` on the function | + +## What is worth stealing + +**The `default=None` sentinel.** Every option is declared with `default=None` +([`:114`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L114), +[`:122`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L122), +[`:140`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L140)) +rather than with its effective default. The plugin can therefore distinguish "the +user chose the current default" from "the user has not chosen", which is what +makes a future default change *announceable* — it warns only the second group. +`pytest_configure` uses exactly this to warn about an unset +`asyncio_default_fixture_loop_scope` +([`:296-301`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L296-L301)). + +This project has the same problem coming: ADR 0001's vocabulary change renames +`namespace_scope` to `share`, and any future move of the block-vs-document default +needs the same mechanism. + +**Normalize, then query once.** In `AUTO` mode the plugin literally adds the +marker it would otherwise have to special-case, so downstream code has one +question with one answer shape. The alternative — branching on mode at every read +site — is what produces the "two components disagree about the current setting" +class of bug. + +**Reuse the host's vocabulary rather than inventing a parallel one.** Loop scope +uses pytest's own `function`/`class`/`module`/`package`/`session` ladder and its +scope names verbatim. It does not invent a third word for lifetime. Compare the +current `namespace_scope`/`namespace_items` naming, which collides with two pytest +concepts at once. + +**Session-wide errors raise `pytest.UsageError`; per-item errors raise +`ValueError`.** The severity matches the blast radius. A misspelled session +setting should stop the session; a bad per-item value should fail that item. + +## What it cannot tell us + +Its resource — an event loop — is cheap to create, has no cross-process identity +problem, and never needs to be scheduled onto a particular worker. It therefore +has nothing to say about the distribution question that dominates this project's +design, and its scope model should not be copied on that axis. + +## Anchors + +- [`Mode`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L82) · + [`PytestAsyncioSpecs`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L90) +- [`pytest_addoption`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L108) · + [`_get_asyncio_mode`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L222) · + [`pytest_configure`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L295) +- [`_make_asyncio_fixture_function`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L210) diff --git a/notes/analyses/14-asyncio.md b/notes/analyses/14-asyncio.md new file mode 100644 index 0000000..a2a679c --- /dev/null +++ b/notes/analyses/14-asyncio.md @@ -0,0 +1,109 @@ +# `asyncio` — the standard library's own pluggable architecture + +Pinned at [`v3.14.2`](https://github.com/python/cpython/tree/v3.14.2/Lib/asyncio). + +`asyncio` has nothing to do with doctests. It is here because it is the standard +library's worked example of a *deliberately* pluggable subsystem, written by +roughly the same community and shipped in the same tree as `doctest`. Setting the +two side by side answers a question the other notes cannot: when CPython wants an +extension point, what does it build — and why does `doctest` have almost none? + +## Classification + +A layered subsystem with four distinct seam kinds, none of which `doctest` uses: +an abstract base class defining the contract, duck-typed callback interfaces, a +policy indirection for selecting an implementation, and a context-manager runner +that owns lifecycle. + +## Core data structures + +```text +Handle / TimerHandle a scheduled callback [events.py:34, :141] +AbstractEventLoop the CONTRACT, ~90 methods [events.py:254] +BaseEventLoop(AbstractEventLoop) the shared implementation [base_events.py:417] +Future a result slot with callbacks [futures.py:31] +Task(Future) a coroutine driven by a loop [tasks.py:56] +Runner context manager owning a loop [runners.py:21] +BaseProtocol / Protocol / BufferedProtocol / DatagramProtocol / SubprocessProtocol + what YOU implement [protocols.py:9, :66, :109, :162, :177] +BaseTransport / ReadTransport / WriteTransport / Transport / ... + what the LOOP implements [transports.py:9, :46, :72, :148] +``` + +## The four seam kinds + +**1. An abstract base as a published contract.** `AbstractEventLoop` +([`events.py:254`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/events.py#L254)) +names every method an event loop must provide, separately from +`BaseEventLoop`, which implements most of them. A third party writing uvloop +implements the *contract*, not a subclass of the shipped implementation. Compare +`doctest`, where the contract and the implementation are the same class, so +`DocTestFinder(parser=...)` demands the class rather than the shape. + +**2. Paired duck-typed roles.** `Protocol` is what the user writes; `Transport` is +what the loop provides. Neither is registered anywhere, neither is checked with +`isinstance`, and the split is by *direction of the call*: the transport is called +by you, the protocol is called by the loop. This is the cheapest possible +extension mechanism — two documented method vocabularies — and it has carried +third-party HTTP, TLS and subprocess stacks for a decade. + +**3. A policy indirection, and its retirement.** `get_event_loop_policy` and +`set_event_loop_policy` sit at +[`events.py:804`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/events.py#L804) +and [`:817`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/events.py#L817), +now delegating to private `_get_event_loop_policy` / `_set_event_loop_policy` +([`:798`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/events.py#L798), +[`:808`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/events.py#L808)) — +the public spellings are on the way out. That is the most instructive thing in +this file. A process-global, mutable indirection for "which implementation should +this program use" was shipped, was widely misused, and is being replaced by +passing the choice explicitly: `asyncio.run(main, loop_factory=...)` +([`runners.py:169`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/runners.py#L169)). + +The lesson transfers directly. A process-global mutable registry is the seam you +regret. docutils' directive table is the same shape and has the same problems — +see [`16-docutils-myst.md`](16-docutils-myst.md). + +**4. A runner that owns lifecycle.** `Runner` +([`runners.py:21`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/runners.py#L21)) +is a context manager that creates the loop, runs the work, cancels stragglers +([`:207`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/runners.py#L207)) +and shuts down cleanly. Global state that must be restored lives in one `finally` +owned by one object. + +`doctest.DocTestRunner.run` does exactly this for `sys.stdout`, `pdb.set_trace`, +`linecache.getlines`, `sys.displayhook` and `PYTHON_COLORS` +([`doctest.py:1534-1573`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1534-L1573)). +It is the one place `doctest` and `asyncio` agree on architecture, and it is +precisely why ADR 0001 takes over `__run` but leaves `run()` alone: the lifecycle +owner should keep owning the lifecycle. + +## Cross-cutting: what `doctest` would look like with `asyncio`'s seams + +| `asyncio` | `doctest` equivalent | Present? | +|---|---|---| +| `AbstractEventLoop` as a separate contract | an ABC or `Protocol` for finder/parser/checker | no — the class *is* the contract | +| `Protocol`/`Transport` duck-typed roles | `OutputChecker` is close: a documented method vocabulary, injected | partly | +| policy indirection | none | no | +| `Runner` owning lifecycle | `DocTestRunner.run`'s save/restore | yes | +| explicit `loop_factory=` replacing global policy | `parser=`, `checker=`, `test_finder=` injection | yes, and it is the healthy part | + +The gap is the first row, and it is the concrete reason +`DocutilsDocTestFinder` cannot be handed to `DocTestSuite(test_finder=...)` today +despite exposing a compatible `find()`. ADR 0001's answer — declare `Protocol`s +*and* subclass the stdlib classes nominally — is the cheap way to have both, and +it costs nothing at runtime. + +## Anchors + +- [`AbstractEventLoop`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/events.py#L254) · + [`BaseEventLoop`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/base_events.py#L417) +- [`get_event_loop_policy`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/events.py#L804) · + [`set_event_loop_policy`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/events.py#L817) +- [`BaseProtocol`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/protocols.py#L9) · + [`Transport`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/transports.py#L148) +- [`Runner`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/runners.py#L21) · + [`run(main, *, loop_factory=None)`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/runners.py#L169) · + [`_cancel_all_tasks`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/runners.py#L207) +- [`Future`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/futures.py#L31) · + [`Task`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/tasks.py#L56) diff --git a/notes/analyses/15-sphinx-ext-doctest.md b/notes/analyses/15-sphinx-ext-doctest.md new file mode 100644 index 0000000..7be7bbc --- /dev/null +++ b/notes/analyses/15-sphinx-ext-doctest.md @@ -0,0 +1,127 @@ +# `sphinx.ext.doctest` + +Pinned at [`v9.1.0`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py). + +## Classification + +A semantic fork. It invents its own document-level model — groups, phases, five +directives — and then converts all of it back into stdlib `doctest.DocTest` +objects for execution. It is the source of every author-facing spelling this +project supports, and the reason those spellings must be honoured exactly. + +It is also builder-coupled: nothing in the pipeline is callable without a built +Sphinx app, and it produces no machine-readable results at all. + +## Core data structures + +```text +TestCode code, type, filename, lineno, options [:235] + `type` in {testsetup, testcleanup, doctest, testcode, testoutput} +TestGroup name, setup: list, tests: list, cleanup: list [:200] + add_code(code, prepend=False) [:207] +DocTestBuilder(Builder) [:292] + self.type: "single" | "exec" <- mutable, read by a + process-global compile patch +SphinxDocTestRunner(doctest.DocTestRunner) [:257] +``` + +`TestGroup.tests` holds heterogeneous entries — `[code]` for a bare block, +`[code, output]` for a paired testcode/testoutput. `add_code` is where three +silent losses live: an orphan `testoutput` is discarded, a second `testoutput` +*replaces* the first, and a `testcode`'s own options are dropped in favour of its +output block's. + +## Data flow + +```text +directive run() [:66] + | parse group names from the optional argument + | trim `# doctest:` flags out of the RENDERED code, keep the original + | in node["test"] + | nodetype = nodes.comment when name in {testsetup, testcleanup} + | or "hide" in options [:92-93] + | stamp node["testnodetype"], ["groups"], ["options"], ["skipif"] + v +doctree + | +DocTestBuilder.test_doc(docname, doctree) [:428] + | for node in doctree.findall(condition): + | if self.skipped(node): continue <- GATED BLOCK IS DROPPED [:443-444] + | code = TestCode(...) + | "*" in groups -> add to every group + | else groups[name].add_code(code) + v +per group: ns = {} + | three runners: setup / test / cleanup, sharing one _fakeout + | test.globs = ns (assigned AFTER construction) + | runner.run(test, out=..., clear_globs=False) + | self.type flipped to "exec" for setup, cleanup, testcode [:548, :608] + | to "single" for ordinary doctests [:580] + v +six integer counters + text streamed to outdir/output.txt +``` + +## Extension seams + +| Seam | Kind | +|---|---| +| `TestDirective` subclassing, with `option_spec` ([`:66`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L66)) | subclass | +| The node attribute stamp — `testnodetype`, `groups`, `options`, `skipif`, `test` | implicit protocol | +| `doctest_global_setup`, `doctest_global_cleanup`, `doctest_test_doctest_blocks`, `doctest_default_flags` | Sphinx confvals | +| `is_allowed_version(spec, version)` ([`:45`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L45)) | function | + +The node attribute stamp is the important one, and it is undocumented. It is the +only decoupled interface in the module: any directive that emits a +`literal_block` or `comment` carrying `testnodetype` participates. It is also +what makes a *third-party* collector — this project's — able to read a page +Sphinx's own directives produced, and vice versa. Reading attributes off the node +rather than trusting one's own directive classes is the only defence against +`Sphinx.add_directive` overriding a registration unconditionally. + +## Semantics this project must match exactly + +| Rule | Anchor | +|---|---| +| `testsetup`, `testcleanup` and `:hide:` render as `nodes.comment` | [`:92-93`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L92-L93) | +| `:options:` is accepted only on `doctest` and `testoutput` | [`:111`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L111) | +| A gated block is dropped during collection — no outcome, id or count | [`:443-444`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L443-L444) | +| `*` means every group the document declares; `default` means no argument was given | [`:428`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L428) onward | +| Setup runs before tests, cleanup after, whatever order the page writes them | `TestGroup` [`:200-226`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L200-L226) | +| `is_allowed_version` takes the **specifier first** | [`:45`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L45) | +| `doctest.compile` is rebound process-wide and never restored | [`:310`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L310) | + +The last two are where this project deliberately diverges. The argument order was +a real defect in the local helper. The `compile` rebinding is unavailable to a +library that loads into every pytest session, which is the entire origin of +ADR 0001's decision to own the per-example loop instead. + +The gated-block rule is the one semantic this project rejects on purpose: Sphinx's +drop destroys the node id, the count and the `-rs` line, and pytest users +reasonably expect a `SKIPPED` outcome with a reason. + +## What it cannot do + +- **Run outside a Sphinx build.** `DocTestBuilder` binds an `env`, a `config`, an + `outdir`, a `sys.path` mutation and an open file handle. +- **Produce results a caller can inspect.** Six ints and text to a file. No failure + can be mapped back to its node without re-parsing prose. +- **Distinguish two blocks in one group.** Every block in a group shares + `DocTest.name`, which is why `SphinxDocTestRunner` overrides a private stdlib + method to swallow the resulting `IndexError` + ([`:257`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L257)). + +## Anchors + +- [`is_allowed_version`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L45) · + [`TestDirective`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L66) · + [`comment nodetype rule`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L92-L93) +- [`TestGroup`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L200) · + [`add_code`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L207) · + [`TestCode`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L235) +- [`SphinxDocTestRunner`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L257) · + [`DocTestBuilder`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L292) · + [`doctest.compile` patch](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L310) +- [`test_doc`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L428) · + [`skipped-node drop`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L443-L444) · + [`type = "exec"` for testcode](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L548) +- [User-facing contract](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/doc/usage/extensions/doctest.rst) diff --git a/notes/analyses/16-docutils-myst.md b/notes/analyses/16-docutils-myst.md new file mode 100644 index 0000000..e1fb1d3 --- /dev/null +++ b/notes/analyses/16-docutils-myst.md @@ -0,0 +1,130 @@ +# docutils and MyST-Parser + +docutils pinned at `docutils-0.22.2` (canonical repository is on SourceForge; the +GitHub copies are third-party mirrors, so anchors here name file and symbol rather +than a permalink). MyST-Parser pinned at +[`v5.1.0`](https://github.com/executablebooks/MyST-Parser/tree/v5.1.0). + +## Classification + +The parsing floor. Two front-ends producing one node model, with two different +line-number conventions and one shared, process-global, unscoped extension +registry. + +## Core data structures + +```text +docutils.nodes.Element attributes: dict[str, Any] + literal_block a rendered code block + comment what testsetup/testcleanup/:hide: become + doctest_block a bare >>> block in reStructuredText + .line int | None + .source the file the text lives in (None when unset) + .rawsource the pre-render source, when the node kept it + +docutils.parsers.rst.Parser + .state_classes an INSTANCE attribute, therefore substitutable + per parse with no process-global mutation + +myst_parser.parsers.docutils_.Parser(RstParser) [v5.1.0:235] + .settings_spec = (..., create_myst_settings_spec(), *RstParser.settings_spec) + [v5.1.0:241-245] +``` + +## The two line conventions + +| Front-end | Construct | `.line` reports | +|---|---|---| +| reStructuredText | top-level `doctest_block` | its **last** line | +| reStructuredText | any block nested in a directive, list item or block quote | `None`, with `.source` also `None` | +| MyST | fenced block | its **first** line | +| either | a block reached through `.. include::` | numbered against the **included** file | + +All four are real and all four have to be normalized by the front-end that knows +which it is. A collector that assumes one convention mis-anchors the other's +blocks; a collector that reads `.line` as a number crashes on the nested case. +This is why `Block.line` in ADR 0001 is nullable and `Block.path` is separate from +the collected document. + +## The directive registry + +`docutils.parsers.rst.directives` keeps one module-level dict consulted by both +the reStructuredText parser and MyST's `run_directive`. It has no per-document +scoping, no versioning, and no ownership. + +Three failure modes follow, and all three have been observed: + +1. **It can be rebound, not merely mutated.** Sphinx's `docutils_namespace()` + restores a snapshot by rebinding the module attribute, so any registration made + inside that context is discarded *and the dict's object identity changes*. A + registration guard that caches a boolean is wrong; membership must be + re-checked against the live dict. +2. **Registrations are overwritten silently.** `Sphinx.add_directive` overrides an + existing name unconditionally, with only a warning. `sphinx.ext.doctest` loaded + in the same interpreter therefore replaces a forked directive class with one + that has different option handling — including the reversed + `is_allowed_version` argument order. +3. **A missing registration is silent.** An unregistered directive parses to a + docutils error node, the page still renders, and the collector finds zero + tests. This is the GH-48 failure shape, and it is the reason ADR 0004 treats + diagnostics as data. + +The defence is not to win the registry. It is to read the *node attributes* — +`testnodetype`, `groups`, `options`, `skipif`, `test`, `hide` — which are +byte-compatible with what `sphinx.ext.doctest` stamps, so a page survives either +class having produced it. + +The registry is a smaller instance of the pattern `asyncio` is currently retiring; +see [`14-asyncio.md`](14-asyncio.md). + +## MyST configuration + +`myst_parser.parsers.docutils_.Parser` subclasses `RstParser` and composes its own +settings spec from `create_myst_settings_spec()` +([`v5.1.0:208`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/parsers/docutils_.py#L208), +[`:241-245`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/parsers/docutils_.py#L241-L245)). +Driving MyST through that `Parser` is what makes the `myst_*` docutils settings — +including `myst_enable_extensions` and `myst_fence_as_directive` — reachable, and +what makes front-matter configuration merge. + +Constructing an `MdParserConfig` by hand and calling `md_parser.render()` against +a bare `make_document()` skips all of it. Colon-fence directives do not exist, a +plain ```` ```python ```` fence is only picked up by a prompt sniff, and the +line-length guard and MyST transforms never run. Those omissions become a decision +rather than an accident once the front-end owns its own configuration. + +`myst_fence_as_directive` is also the answer to a real user request: it maps a +bare language fence onto a directive name, so a project that prefers not to write +`{testcode}` can still have its ```` ```python ```` blocks collected. + +## Reporter behaviour + +Default settings send reporter output to stderr and raise `SystemMessage` at +`halt_level`, aborting mid-parse. Both are configurable: raising `halt_level` past +the abort threshold and attaching an observer with `Reporter.attach_observer` +turns messages into values. `Reporter.system_message` notifies observers for any +level above `DEBUG` independently of `report_level`, so observation and display are +separable — which is what makes ADR 0004's code-keyed suppression possible without +losing the underlying record. + +## What it cannot do + +- **Scope a directive registration** to one parse, one document or one thread. +- **Report a line for every node.** See the table above. +- **Type its own attribute channel.** `Element.attributes` is `dict[str, Any]`, and + typeshed's stub for `get(key, failobj: _T) -> _T` is actively wrong — it claims + `_T` even when the key is present holding something else. One narrowing accessor + at the parse boundary is cheaper and safer than a coercion at every read site. + +## Anchors + +- MyST: [`docutils_.Parser`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/parsers/docutils_.py#L235) · + [`create_myst_settings_spec`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/parsers/docutils_.py#L208) · + [`settings_spec`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/parsers/docutils_.py#L241-L245) · + [`MdParserConfig`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/config/main.py) +- Sphinx's registry snapshot/rebind: [`sphinx/util/docutils.py`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/util/docutils.py) · + unconditional override: [`sphinx/application.py`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/application.py) +- docutils: `docutils/parsers/rst/directives/__init__.py` (`_directives`), + `docutils/parsers/rst/states.py` (`state_classes`, `doctest_block` line + assignment), `docutils/utils/__init__.py` (`Reporter.attach_observer`, + `system_message`), at `docutils-0.22.2`. diff --git a/notes/analyses/17-prior-art.md b/notes/analyses/17-prior-art.md new file mode 100644 index 0000000..c9da655 --- /dev/null +++ b/notes/analyses/17-prior-art.md @@ -0,0 +1,167 @@ +# Prior art: Sybil, xdoctest, pytest-examples, typeshed + +Four projects that solved some part of this problem differently. Each is read for +one specific question, and each answers it — two of them by counterexample. + +--- + +## Sybil + +Pinned at [`10.0.1`](https://github.com/simplistix/sybil/tree/10.0.1). + +**The bet:** a document is a flat sequence of non-overlapping character spans, not +a parse tree, and everything else — markup format, language, assertion semantics, +test runner — is a plugin over that primitive. It parses nothing itself: no +docutils, no myst-parser, no CommonMark implementation, and zero runtime +dependencies. Every format is a regex `Lexer` producing `Region(start, end, +lexemes)` spans over raw text. + +**Data model.** `Region` is a half-open span with three payload slots — `lexemes`, +`parsed`, `evaluator` — and lives in two undocumented-by-type states: lexed +(lexemes only) and parsed (evaluator attached). `Document` is text plus path plus +regions plus **one `namespace: dict`**. `Example` joins document and region at run +time and holds a *reference* to that namespace. `Sybil` itself is pure +configuration. + +**What is genuinely good.** `Document.add` bisect-inserts and **raises +`ValueError` on any overlap**. That single invariant gives a total order for free +and converts the silent class of parser bug — a block dropped or collected twice — +into a loud error at collection time. It is the best structural decision in the +codebase. + +Its public testing helpers (`check_lexer`, `check_parser`, `check_sybil`) are +documented *and* used by the project's own runnable documentation, so the +extension guide is under test. That is rare and worth copying. + +**The fatal flaw.** One mutable namespace per document, and +[one independently selectable pytest item per region](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/integration/pytest.py). +`pytest -k` on an example whose predecessor bound a name raises `NameError`. The +documentation never mentions this; there is no discussion of `xdist`, parallelism +or deselection anywhere in it. This is axis 1 of +[`00-taxonomy.md`](00-taxonomy.md), in the unacknowledged column. + +**The second flaw.** [Node ids are positional](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/sybil.py#L155-L157) — +`line:{line},column:{column}`. Adding a paragraph above an example renames every +downstream test, breaking `--lf`, `--nf`, deselect files, xfail lists and CI flake +history. For a *documentation* test runner, prose above examples is the thing that +changes most often. + +**What it cannot do.** Groups. A regex cannot see a directive's options, so Sybil +has no group concept at all and directs users to clear the namespace instead. This +is the clearest available argument for paying the docutils dependency: if +`:skipif:`, `:options:` and group names are part of the product, the document +model has to be a parse tree. + +**Verdict on the invariant.** The non-overlap check does not survive contact with +docutils: two `.. include::` directives naming the same file legitimately produce +blocks over identical source spans. Adopt the *idea* — detect double collection — +as a diagnostic carrying both provenances, not as an exception. + +--- + +## xdoctest + +Pinned at [`v1.3.2`](https://github.com/Erotemic/xdoctest/tree/v1.3.2). + +**The bet:** a doctest is Python source that happens to live in a docstring, so +parse it with `ast`/`tokenize` rather than a line regex, and abandon stdlib +compatibility to fix the design. + +**What is genuinely good.** + +- `ast`-based parsing really is better than `_EXAMPLE_RE` for Python. +- Directives are **structured objects** rather than an int bitmask, and + [`REQUIRES`](https://github.com/Erotemic/xdoctest/blob/v1.3.2/src/xdoctest/directive.py#L58) + carries a *set of unmet requirements* — so a skip knows it skipped because + `module:torch` was absent. A bool discards exactly the information the reader + wants, and this is the single most transferable idea in the survey. +- Per-part synthetic filenames plus a filename-to-block map, so an exception raised + inside a function that an *earlier* block defined is attributed to the defining + block. Directly applicable to grouped namespaces. + +**The counterexample it provides.** It is now building stdlib compatibility back, +in modules that did not exist at the tagged release. Getting stdlib semantics out +of its own intake seam requires setting `REQUIRE_WANT`, +`deferred_output_matching=False` **and** `compile_mode='single'` — three knobs to +undo one abandoned default, paid years later. This is the empirical answer to +"should the core be vanilla?" + +**Two things not to copy.** Its got/want defaults are permissive — +`ELLIPSIS`, `NORMALIZE_WHITESPACE` and `NORMALIZE_REPR` all default true — which +silently changes the meaning of tests users wrote for stdlib. And unknown +directives and parse errors `warnings.warn` and vanish, so a typo weakens a test +instead of failing it. A test that reports green while checking nothing is worse +than no test. + +It also unregisters pytest's doctest plugin outright, which is hostile in a +`pytest11` package. + +--- + +## pytest-examples + +Pinned at [`v0.0.18`](https://github.com/pydantic/pytest-examples/tree/v0.0.18). + +**The bet:** invert the contract — the author does not write expected output, the +runner writes it. A block is a module exec'd once with `print` captured, and the +output is rendered back into the source file. + +**What is genuinely good.** + +- **Emitting the canonical form instead of parsing it collapses check-mode and + update-mode into one code path.** That is a real structural insight. +- **Absolute byte offsets plus an invertible dedent scalar** are exactly what a + data model needs to rewrite source. Read-only tools discard the indent; keeping + it is the difference between "we could add `--update-examples` later" and "we + would have to redesign the data model first". Two int fields. +- **It composes with pytest by contributing no collector at all.** Examples are + `parametrize` params, so marks, `-k`, fixtures and xfail all work unmodified. + This is the cheapest correct integration in the entire survey. + +**What not to copy.** It hard-codes a stack depth as a magic integer — the depth of +another library's internal call stack — and forges frames through +`ctypes.pythonapi.PyFrame_New`. Its write-back splices at collection-time offsets +with no staleness check, so editing a file while it runs corrupts the file. And +its update mode is a session-scoped in-process two-phase commit, so `-x` or a crash +writes nothing and says nothing — the worst outcome for a tool whose selling point +is rewriting your files. + +**Rule extracted.** Any write-back must content-hash the region at collection and +refuse on mismatch. An unconditional splice at stale offsets is data loss, not a +race. + +--- + +## typeshed + +Pinned at [`8c7256c`](https://github.com/python/typeshed/blob/8c7256c/stdlib/doctest.pyi) +(no tags on this repository; commit reachable from trunk). + +**Read for:** what a typed `doctest` actually costs, and where the type system +gives up. + +The stub annotates every public name and then hands back `Any` at exactly the +three extensible points: `globs: dict[str, Any]` (correct for values, since they +are user objects), `**options: Any` on the three suite builders (incorrect — the +accepted keys are exactly known), and `optionflags: int` everywhere (so +`optionflags=4096` type-checks). + +Two facts matter for anything claiming to be a "typed vanilla core": + +**The de-facto contract is the stub, not the source.** Downstream projects run +mypy against it. Narrowing `parse()` to `list[Example]`, dropping the `bool` +overload on `find`, or typing `out` as `Callable[[str], None]` stops type-checking +for existing typed callers whose code runs fine. + +**Its largest consumer violates it deliberately.** `_pytest.doctest` repurposes +`out` from a write-callable into a *list*, with a `# type: ignore[arg-type]`. Any +honest typing of that parameter has to accommodate the ecosystem's actual usage. + +The stub also has a false negative worth knowing: `DocTestRunner.test: DocTest` is +declared unconditionally, while runtime assigns it only inside `run()`. A stub that +type-checks a crash is worse than no stub for that attribute. + +**Rule extracted.** Keep the runtime objects structurally identical to stdlib's and +put precision in a parallel layer — `TypedDict` over the node-attribute channel, +`Protocol`s for the seams, `Literal` for closed vocabularies. Do not narrow a +signature typeshed publishes wider. diff --git a/notes/analyses/20-data-structures.md b/notes/analyses/20-data-structures.md new file mode 100644 index 0000000..3b0a730 --- /dev/null +++ b/notes/analyses/20-data-structures.md @@ -0,0 +1,86 @@ +# Cross-cutting: the data structures, lined up + +Every system in these notes ends up representing the same four things: a *unit of +source*, a *unit of execution*, a *unit of shared state*, and a *unit of result*. +The disagreements are entirely about which of those four are the same object. + +## The four units + +| System | source unit | execution unit | shared-state unit | result unit | +|---|---|---|---|---| +| CPython `doctest` | `Example` | `DocTest` | `DocTest.globs` | `TestResults` (2 ints + an attribute) | +| `_pytest.doctest` | `Example` | `DocTest` = one `Item` | `DocTest.globs`, wiped per item | `MultipleDoctestFailures` → `ReprFailDoctest` | +| `sphinx.ext.doctest` | `TestCode` | `TestGroup` | `ns` assigned post-construction | six ints + text to a file | +| Sybil | `Region` | `Example` = one `Item` | `Document.namespace` | truthy return or exception | +| xdoctest | `DoctestPart` | own `DocTest` | `global_namespace` | own report objects | +| pytest-examples | `CodeExample` | the block, exec'd once | explicit `module_globals=` | captured output, or a rewrite | +| ADR 0001 | `Block` | `DocTest` per block | one `globs` per **group**, on the `Item` | stdlib's, per block | + +Reading across the "shared-state unit" column against the "execution unit" column +is the whole design problem. Sphinx and Sybil both put the shared state at a +coarser granularity than the execution unit; only Sphinx also puts the *pytest +item* at that same coarser granularity, which is why Sphinx never hits Sybil's +`-k` failure. + +## Field-by-field: what a source unit carries + +| Field | `Example` | `TestCode` | `Region` | `CodeExample` | `Block` (ADR 0001) | +|---|---|---|---|---|---| +| source text | `source` | `code` | via `lexemes` | `source` | `source` | +| expected output | `want` | paired separately | — | written, not read | `want` | +| line | `lineno` (0-based, string-relative) | `lineno` | computed from span | `start_line` | `line` (nullable) | +| byte offsets | — | — | `start`, `end` | `start_index`, `end_index` | — (deferred) | +| dedent scalar | `indent` | — | `Lexeme.offset` | `indent` | — (deferred) | +| kind | — | `type` | inferred from evaluator | `prefix_tags()` | `kind` | +| group | — | via `TestGroup` | — | — | `groups` | +| options | `options` | `options` | — | — | `options` | +| gate | — | `skipif` on the node | — | — | `skipif` (unevaluated) | +| file | on the `DocTest` | `filename` | on the `Document` | `path` | `path` | +| compile mode | — | on the *builder*, mutable | — | always exec | on the `Example` subclass | + +Three observations. + +**Only pytest-examples carries byte offsets and an invertible dedent.** Those two +fields are the entire difference between a read-only tool and one that can rewrite +expected output later. ADR 0001 defers them, which is a decision to be revisited +rather than a decision made. + +**Compile mode is on the wrong object everywhere except ADR 0001.** Sphinx keeps +it as mutable builder state read through a process-global patch; stdlib hard-codes +it in the loop. Putting it on the example data is what lets it survive +`copy.copy`, merging and any reordering, without a flag some runner has to be +holding at the right moment. + +**Nobody but ADR 0001 makes the line nullable.** Every other system either always +has a line (because it computed the span itself) or fabricates one. With a real +doctree there are constructs that genuinely have no recoverable line, and pytest +has a branch for exactly that — `EXAMPLE LOCATION UNKNOWN` — which is unreachable +unless the model can express it. + +## What a result unit carries + +None of these systems produce a per-example result *value*: + +- stdlib returns `TestResults(failed, attempted)` with `skipped` bolted on as an + instance attribute, and pushes everything else through `out` as text. +- pytest works around that by repurposing `out` into a list so `report_*` can + append, then rebuilds a location by slicing `test.docstring`. +- Sphinx works around it by not producing machine-readable results at all. + +This absence is why a merged `DocTest` needs a synthetic page with blank-line +padding: the only channel for a location is `(test.lineno, test.docstring, +example.lineno)`, so a group holding many blocks must fabricate a docstring in +which those arithmetic relations still hold. + +Giving each block its own `DocTest` removes the need for the fabrication rather +than improving it — the three fields are then already true. + +## Anchors + +- [`Example` / `DocTest` / `TestResults`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L114) +- [`TestCode`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L235) · + [`TestGroup`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L200) +- [`MultipleDoctestFailures`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L172) · + [`repr_failure`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L317-L344) +- [Sybil `Region`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/region.py) · + [Sybil `Document`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/document.py) diff --git a/notes/analyses/21-data-flows.md b/notes/analyses/21-data-flows.md new file mode 100644 index 0000000..0264452 --- /dev/null +++ b/notes/analyses/21-data-flows.md @@ -0,0 +1,113 @@ +# Cross-cutting: the data flows, lined up + +Four pipelines that all end in the same `exec()`, drawn to the same scale so the +divergence points are visible. + +## The pipelines + +```text +CPython doctest + string ─► DocTestParser.parse ─► DocTestFinder.find ─► DocTestRunner.run + └─► __run ─► exec + +pytest --doctest-glob + path ─► pytest_collect_file ─► DoctestTextfile.collect ─► DoctestItem + (one DocTest for the whole file) ├─ setup(): globs.update(fixtures) + ├─ runtest(): runner.run(clear_globs=True) + └─ repr_failure(): per-failure location + +sphinx.ext.doctest + doctree ─► test_doc ─► condition filter ─► skipped? DROP + │ + ├─► TestCode ─► TestGroup{setup, tests, cleanup} + │ + └─► per group: ns={}; test.globs=ns (post-construction) + 3 runners (setup/test/cleanup), clear_globs=False + self.type flipped single/exec around each run + └─► doctest.compile (PROCESS-GLOBAL PATCH) ─► exec + +ADR 0001 + path ─► markup.parse_file ─► (Blocks, Diagnostics) + │ + ├─► project() ─► GroupTest{group, tests[], globs} + │ pure: no docutils, no pytest, no filesystem, + │ no user code — :skipif: passes through unevaluated + │ + └─► Document(pytest.Module) ─► DocutilsItem (one per GROUP) + ├─ setup(): globs cleared, then fixtures injected + ├─ runtest(): run_group() runs each block's DocTest + │ in phase order, clear_globs=False + │ :skipif: evaluated HERE + │ DocutilsRunner.__run ─► exec (mode from data) + └─ repr_failure(): inherited; reads each block's own DocTest +``` + +## Where they diverge + +**At the gate.** Sphinx evaluates `:skipif:` during collection and *drops* the +node. pytest cannot express that — an item must exist to have an outcome — so +`doctest_docutils` marks the block `SKIP` instead. ADR 0001 moves the evaluation +to `runtest()`, which additionally buys collection purity: with no user code +running at collection, worker collections cannot diverge and `--collect-only` +cannot have side effects. + +**At the globs assignment.** Every system that shares state has to assign +`test.globs` *after* `DocTest.__init__`, because the constructor +[copies](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L565). +Sphinx does this explicitly. pytest does not share at all, and its +[`runtest`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L295-L303) +runs with `clear_globs` defaulting to `True` — so any design that inherits +`runtest()` unchanged and expects sharing gets its mapping emptied after the first +block. That is a trap with no diagnostic; the symptom is a `NameError` in block +two. + +**At the compile call.** stdlib hard-codes `"single"`. Sphinx flips a mutable +builder attribute and reads it through a process-global rebinding of +`doctest.compile` that is never restored. `doctest_docutils` today clones the +mangled loop's code object to get a private version of that rebinding. ADR 0001 +carries the mode on the example data and reads it in a loop it owns — the only one +of the four that neither mutates process state nor copies a code object. + +**At the location.** stdlib computes `test.lineno + example.lineno + 1`. Sphinx +gives every block in a group the same `DocTest.name` and pays for it by overriding +a private method to swallow an `IndexError`. `doctest_docutils` today fabricates a +synthetic page so the arithmetic stays true across merged blocks. ADR 0001 gives +each block its own `DocTest`, so the arithmetic is true without fabrication. + +## The order-of-operations facts + +Three orderings are load-bearing and each has a failure mode with no error +message. + +**Fixtures are injected in `setup()`, into the mapping, in place.** +`DoctestItem.setup()` does `self.dtest.globs.update(globs)` +([`:288-293`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L288-L293)). +A design that clears and rebinds the mapping around that call either discards the +injected names — so `getfixture` raises `NameError` — or reuses the previous +attempt's mutations. Under `--reruns`, the second is a false green: an expectation +true only on attempt two reports as a pass. + +**Phase order and page order are different orders.** A group hands its blocks over +as setup, tests, cleanup; a reader meets them in whatever order the page writes +them. Any layout that anchors reported lines on the *run* order reports examples +against whichever block came first in that sequence — and can point past the end of +the file. Sphinx sidesteps this by not reporting useful lines at all. + +**Collection order must be numeric, never lexicographic.** `DocTest.__lt__` +compares names as text +([`:596`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L596)), +and names carry positions as text, so any accidental `sorted()` runs `page.md[10]` +before `page.md[1]`. Every test still passes, in the wrong sequence — and for a +group sharing state, the wrong sequence is the bug. + +## Anchors + +- [`DocTest.__init__` globs copy](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L565) · + [`__lt__`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L596) · + [`__run`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1344) · + [`compile(..., "single", ...)`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1400) +- [`DoctestItem.setup`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L288-L293) · + [`runtest`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L295-L303) +- [`test_doc`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L428) · + [gated-node drop](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L443-L444) · + [`doctest.compile` patch](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L310) diff --git a/notes/analyses/22-extension-seams.md b/notes/analyses/22-extension-seams.md new file mode 100644 index 0000000..eb39d65 --- /dev/null +++ b/notes/analyses/22-extension-seams.md @@ -0,0 +1,106 @@ +# Cross-cutting: extension seams + +Five mechanisms appear across these systems. They are not equally good, and the +ranking is not a matter of taste — each has an observed failure mode. + +## The five mechanisms + +| Mechanism | Where | Failure mode | +|---|---|---| +| **Nominal subclassing** — the API demands the class, not the shape | stdlib `doctest`, `sphinx.ext.doctest` directives | A structurally compatible object is rejected. `DocutilsDocTestFinder` exposes a compatible `find()` and still cannot be passed to `DocTestSuite(test_finder=...)` | +| **Callable aliases** — `Evaluator = Callable[[Example], str \| None]` | Sybil | Types nothing. You cannot express "this lexer emits `source` and `arguments`", so a mismatched pairing raises `KeyError` at run time. Sybil's own source comments say the payload "could likely be a `TypedDict`" | +| **Named registry** — a module dict plus a `register_*` function | `doctest.register_optionflag`, docutils directives, xdoctest's two facades | Process-global mutable state. Order-dependent, unscoped, and silently overwritable | +| **`Protocol`** — structural typing | xdoctest's `StdlibExampleLike` | None inherent; but a `Protocol` alone does not satisfy a nominal consumer | +| **Hookspec** — the host's own plugin protocol | pytest, pytest-asyncio's `PytestAsyncioSpecs` | Requires a host with a plugin system; not available to a library core | + +## Why `register_optionflag` works and the docutils registry does not + +Both are process-global mutable dicts. One is fine and one is a recurring bug +source, and the difference is instructive. + +[`register_optionflag`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L153) +is **append-only and idempotent**: registering a name that exists returns the +existing bit. It cannot be overwritten, so two libraries registering `NUMBER` agree +rather than fight. Its only sharp edge is ordering — ints are +`1 << len(OPTIONFLAGS_BY_NAME)` — which matters because typeshed hard-codes the +builtin values, so a flag registered *before* the builtins would change every +stdlib constant. Registering at import of the core rather than from a plugin hook +is what keeps that ordering stable, and it is also required because an +unregistered name makes a page fail to **parse**. + +The docutils directive table is **overwrite-by-default and rebindable**. Sphinx's +`docutils_namespace()` restores a snapshot by rebinding the module attribute, so +the dict's *identity* changes and a cached boolean guard is wrong. +`Sphinx.add_directive` overwrites unconditionally with only a warning. The result +is that a directive class you registered may not be the one that ran. + +**The rule extracted:** a global registry is acceptable when registration is +append-only and idempotent, and a liability when it is last-writer-wins. Where the +registry is someone else's and last-writer-wins, do not depend on having won — +depend on the *data* both writers produce. That is why ADR 0001 reads +`BlockAttributes` off the node rather than trusting its own directive classes to +have run: the attribute set is byte-compatible with what `sphinx.ext.doctest` +stamps, so either winner is fine. + +`asyncio` reached the same conclusion about global indirection from the other +direction and is retiring `set_event_loop_policy` in favour of an explicit +`loop_factory=` argument — see [`14-asyncio.md`](14-asyncio.md). + +## The nominal/structural trap, and the cheap way out + +Typeshed's signatures demand classes: `DocTestFinder.__init__(parser: DocTestParser += ...)`, `DocTestSuite(test_finder: DocTestFinder | None)`. A `Protocol` gives a +third party structural typing; a nominal subclass keeps you passable to the +stdlib's own builders. + +Doing both costs nothing: + +```python +class Frontend(t.Protocol): + suffixes: t.ClassVar[frozenset[str]] + + def parse( + self, text: str, path: pathlib.Path, *, settings: Settings + ) -> ParseResult: ... + + +class DocutilsDocTestParser( + doctest.DocTestParser +): # nominal, for DocFileSuite(parser=...) + ... +``` + +Sybil, xdoctest and stdlib each took one half. Taking both is the only reason this +is worth stating as a rule. + +## What deserves a seam, and what does not + +The project rule is that a new public API waits until a caller outside the module +needs it. Applied to the candidates that came up: + +| Candidate | Verdict | +|---|---| +| `BlockKind` registry | **Yes.** Turns "a new block kind" from an edit to a method branching on string literals into adding a tuple. Its `node_types` field is what keeps the `nodes.comment` requirement reviewable rather than buried in a `findall` call | +| Output checker injection | **Yes.** The highest-demand seam, and the one Sybil closed entirely by hard-coding `checker=OutputChecker()` — adding one there requires subclassing two classes | +| `Frontend` protocol | **Yes.** Four implementations ship on day one (rST, MyST, text, Python docstrings) | +| Compile-policy callable | **No.** One implementation, and a second would have to reproduce the `` filename shape or break `linecache` | +| A per-example observer protocol | **No.** pytest gets failures through `report_*` and `MultipleDoctestFailures`; the CLI uses `summarize()`. No third consumer | +| A registry *object* with builders, digests and manifests | **No.** Two module-level dicts and two `register_*` functions do the same work | +| Entry-point plugin discovery | **No.** Not until a caller outside the package exists. It also breaks xdist's collection-purity requirement, since discovery is an import side effect rather than a function of (files, argv, ini) | + +That last row is worth keeping in mind generally: **any registry populated by +import side effects is in tension with +[`12-pytest-xdist.md`](12-pytest-xdist.md)'s requirement** that every worker +collect identical ids in identical order. A registry whose contents depend on +which conftest happened to be imported is not a pure function of the inputs. + +## Anchors + +- [`register_optionflag`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L153) · + [`report_*` hooks](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1286-L1314) · + [`OutputChecker`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1690) +- [`_split_scope`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L284) · + [`pytest_xdist_setupnodes` consumers](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/workermanage.py#L26-L37) +- [`PytestAsyncioSpecs`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L90) +- [`AbstractEventLoop`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/events.py#L254) · + [`set_event_loop_policy`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/events.py#L817) diff --git a/notes/analyses/23-namespace-scope-and-test-identity.md b/notes/analyses/23-namespace-scope-and-test-identity.md new file mode 100644 index 0000000..524fbd3 --- /dev/null +++ b/notes/analyses/23-namespace-scope-and-test-identity.md @@ -0,0 +1,127 @@ +# Cross-cutting: namespace scope and test identity + +The one axis where a wrong answer is silent. Every other design choice in these +notes produces inconvenience — a renamed test, a conversion layer, an extra knob. +This one produces a `NameError` in a test the user believed they could select, or a +green run that should have been red. + +## The two questions, which are not the same question + +1. **Which blocks share a `globs` mapping?** (scope) +2. **Which blocks can be selected, reported and scheduled independently?** + (identity) + +Everything downstream — `-k`, `--lf`, `--deselect`, `-x`, `--reruns`, every +`--dist` mode, JUnit rows, flake history — depends on the second. Everything a +narrative page needs depends on the first. + +## The product space + +| | one node id | N node ids | +|---|---|---| +| **one `DocTest`** | `sphinx.ext.doctest`; `doctest_docutils` `merged` | incoherent | +| **N `DocTest`s** | *unoccupied until ADR 0001* | Sybil; `doctest_docutils` `per-block` | + +The bottom-right cell is where the silent failure lives, and two shipped projects +are in it. + +Sybil is there **unacknowledged**: one `Document.namespace` shared by reference, +one pytest item per region. `pytest -k` on an example whose predecessor bound a +name raises `NameError`, and nothing in its documentation says so. + +`doctest_docutils`'s `per-block` mode is there **acknowledged and guarded** — the +guards are the xdist scheduler substitution, the scheduler refusal, and the +run-twice refusal. Those guards are the reason `_worker_count`, `_shared_page`, +`_is_page` and `_splitting_scheduler` exist at all. + +The bottom-left cell gives per-block reporting *and* an unsplittable sharing unit, +and it needs no guards, because there is nothing to split. + +## Why the guards are expensive + +The constraints come from [`12-pytest-xdist.md`](12-pytest-xdist.md) and are all +structural, not incidental: + +- A live mapping is a Python object; only execnet-serializable builtins cross a + worker boundary. +- The controller never collects, so it cannot ask "which items share state" — it + sees node-id strings and nothing else. Any protection must be *inferred from + string shape* or *encoded into the id*. +- The only affinity primitive is + [`_split_scope(nodeid) -> str`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L284). + `load` and `worksteal` have no scope concept at any layer, so under a user-typed + `--dist load` the options are refuse, substitute, or do not need protecting. +- `xdist_group` is applied + [worker-side](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/remote.py#L245-L254) + and only when the worker's own `--dist` is literally `loadgroup`. A + controller-side substitution never reaches a worker. +- A worker crash re-runs only the *uncompleted* items of a work unit, on a fresh + process. Blocks 3..N then run against an empty mapping. Restarts are on by + default. +- A retry (`--reruns`, `--count`) re-runs a block against globals it already + mutated, so an expectation true only on attempt two reports **PASS**. + +Every one of these evaporates when the item is the sharing unit. + +## Test identity: never positional + +Two of the surveyed projects derive node ids from position — Sybil's +[`line:{line},column:{column}`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/sybil.py#L155-L157) +and pytest-examples' `path:start-end`. + +For a *documentation* test runner this is indefensible, because prose above +examples is the thing that changes most often. Adding a sentence renames every +downstream test, which breaks `--lf`, `--nf`, checked-in deselect files, xfail +lists and CI flake history. pytest-examples compounds it: the same string is the +dedupe key for its write-back, so an identity collision becomes file corruption. + +The rule: **author-declared name first, stable ordinal as fallback, and the +fallback shape invariant across configuration.** The test to apply is concrete — +adding a sentence to a page must rename zero tests. + +Two corollaries: + +- `DocTest.name` must be machine-independent. Embedding an absolute path makes a + checked-in `--deselect` resolve only on the machine that produced it, and puts a + home directory in JUnit XML. +- Column is not worth carrying. It adds churn and disambiguates nothing once names + exist. + +## What a node id does *not* promise + +Worth stating plainly, because it is the honest limit of the recommended design: +**no surveyed implementation makes a node id a promise of independent +runnability.** Selecting block two of a stateful page raises `NameError` under +Sybil, under `per-block`, and under any scheme that hands out per-block ids over +shared state. + +The choice is therefore not between "selectable blocks" and "unselectable blocks". +It is between an id that *claims* to be selectable and is not, and an id whose +granularity honestly matches what can be run alone. `merged` and ADR 0001 both +choose the latter; they differ only in whether the reporting granularity has to +match the selection granularity, and ADR 0001's answer is that it does not. + +## Fixture lifetime falls out of this + +A page collected as a `pytest.Module` **is** the module scope, so +`@pytest.fixture(scope="module")` already has page lifetime — no shim required. +Sybil reaches the same outcome through a `getparent` override that returns the +file collector when pytest asks for `Module`. + +The corollary is a trap for any design that shares state across items without +sharing the item: fixtures do not follow. A block that stashes a fixture-derived +object under a name keeps answering after that fixture has been finalized, because +the name outlives the object's lifetime. Making the item the sharing unit aligns +the two — the mapping and the fixtures have the same lifetime because they have the +same owner. + +## Anchors + +- [`DocTest.__init__` globs copy](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L565) +- [`runtest` with `clear_globs=True`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L295-L303) · + [`setup` globs update](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L288-L293) +- [`_split_scope`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L284) · + [`xdist_group` append](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/remote.py#L245-L254) · + [collection-mismatch abort](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/load.py#L259) +- [Sybil `identify`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/sybil.py#L155-L157) · + [Sybil pytest integration](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/integration/pytest.py) diff --git a/notes/analyses/90-bibliography.md b/notes/analyses/90-bibliography.md new file mode 100644 index 0000000..d627b6f --- /dev/null +++ b/notes/analyses/90-bibliography.md @@ -0,0 +1,162 @@ +# Bibliography + +Every external anchor cited by `docs/adrs/` and these notes, in one place. All +links are pinned to a tag, or — where a project publishes no tags — to a commit +reachable from trunk. Line anchors are only meaningful on a pinned ref and are not +used anywhere else. + +## CPython — `v3.14.2` + +[`python/cpython @ v3.14.2`](https://github.com/python/cpython/tree/v3.14.2) + +### `Lib/doctest.py` + +| Symbol | Anchor | Cited for | +|---|---|---| +| `TestResults` | [`:114`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L114) | 2-field namedtuple; `skipped` is an extra attribute | +| `register_optionflag` | [`:153`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L153) | the one append-only, idempotent cross-library registry | +| `_load_testfile` | [`:245`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L245) | private, reached by `doctest_docutils` today | +| `DocTest.__init__` | [`:565`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L565) | **copies** the globs mapping | +| `DocTest.__lt__` | [`:596`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L596) | compares names as text | +| `DocTestParser` | [`:609`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L609) | the injectable parser | +| `_EXAMPLE_RE` | [`:618`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L618) | private, used for prompt sniffing | +| `DocTestFinder` | [`:844`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L844) | nominal type demanded by `DocTestSuite` | +| `report_*` hooks | [`:1286-1314`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1286-L1314) | the four supported in-loop seams; no `report_skip` here | +| `__run` | [`:1344`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1344) | name-mangled loop; overridable by mechanism | +| `compile(..., "single", ...)` | [`:1400`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1400) | the hard-coded mode `{testcode}` cannot use | +| `__record_outcome` | [`:1485`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1485) | arity and accumulator differ across supported versions | +| `__patched_linecache_getlines` | [`:1501`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1501) | parses the `` filename shape back | +| `run()` save/restore | [`:1534-1573`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1534-L1573) | global interpreter state; not reentrant | +| `summarize` | [`:1590`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1590) | reads the accumulator the owned loop must write | +| `OutputChecker` | [`:1690`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1690) | the documented checker seam | +| `DebugRunner` | [`:1874`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1874) | `report_*` overriding as the sanctioned loop control | +| `testfile` | [`:2091`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L2091) | the API `testdocutils` mirrors | +| `DocTestSuite` | [`:2467`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L2467) | demands a nominal `DocTestFinder` | +| `DocFileSuite` | [`:2570`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L2570) | demands a nominal `DocTestParser` | + +Documentation: [`Doc/library/doctest.rst`](https://github.com/python/cpython/blob/v3.14.2/Doc/library/doctest.rst). + +### `Lib/asyncio/` + +| Symbol | Anchor | +|---|---| +| `AbstractEventLoop` | [`events.py:254`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/events.py#L254) | +| `get_event_loop_policy` / `set_event_loop_policy` | [`events.py:804`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/events.py#L804) · [`:817`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/events.py#L817) | +| `BaseEventLoop` | [`base_events.py:417`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/base_events.py#L417) | +| `BaseProtocol` / `Protocol` | [`protocols.py:9`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/protocols.py#L9) · [`:66`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/protocols.py#L66) | +| `BaseTransport` / `Transport` | [`transports.py:9`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/transports.py#L9) · [`:148`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/transports.py#L148) | +| `Runner` / `run` / `_cancel_all_tasks` | [`runners.py:21`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/runners.py#L21) · [`:169`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/runners.py#L169) · [`:207`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/runners.py#L207) | +| `Future` / `Task` | [`futures.py:31`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/futures.py#L31) · [`tasks.py:56`](https://github.com/python/cpython/blob/v3.14.2/Lib/asyncio/tasks.py#L56) | + +## pytest — `9.1.1` + +[`pytest-dev/pytest @ 9.1.1`](https://github.com/pytest-dev/pytest/tree/9.1.1) · +[`src/_pytest/doctest.py`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py) + +| Symbol | Anchor | Cited for | +|---|---|---| +| `pytest_collect_file` | [`:126`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L126) | not `firstresult` | +| `_is_setup_py` / `_is_main_py` | [`:141`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L141) · [`:155`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L155) | privates imported today | +| `_is_doctest` | [`:148-152`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L148-L152) | claims initpaths **before** `--doctest-glob` | +| `MultipleDoctestFailures` | [`:172`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L172) | the missing per-example result value, worked around | +| `_init_runner_class` / `PytestDoctestRunner` | [`:178`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L178) · [`:181`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L181) | **unreachable by name** | +| `DoctestItem` | [`:251`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L251) | the subclassed item | +| `setup` | [`:288-293`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L288-L293) | `globs.update(...)` in place | +| `runtest` | [`:295-303`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L295-L303) | `clear_globs` defaults to `True` | +| `repr_failure` | [`:317-344`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L317-L344) | reads each failure's own `test` | +| `_get_flag_lookup` | [`:385`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L385) | lazily registers `ALLOW_UNICODE`, `ALLOW_BYTES`, `NUMBER` | +| `get_optionflags` | [`:401`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L401) | read, not re-declared | +| `_get_continue_on_failure` | [`:410`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L410) | private helper imported today | +| `DoctestTextfile` | [`:420-421`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L420-L421) | `obj = None` as a class attribute | +| `_check_all_skipped` | [`:451`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L451) | fires only once the item is running | +| `DoctestModule` / `parsefactories` | [`:500`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L500) · [`:556`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L556) | autouse fixtures from conftest | +| `_get_checker` | [`:662`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L662) | the checker that would have to be reimplemented | +| `_get_report_choice` | [`:703`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L703) | private helper | +| `doctest_namespace` | [`:721`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L721) | the fixture that survives plugin blocking today | + +## pytest-xdist — `v3.8.0` + +[`pytest-dev/pytest-xdist @ v3.8.0`](https://github.com/pytest-dev/pytest-xdist/tree/v3.8.0) + +| Symbol | Anchor | Cited for | +|---|---|---| +| `parse_tx_spec_config` | [`workermanage.py:26-37`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/workermanage.py#L26-L37) | list `extend`, so a negative multiplier contributes zero | +| `LoadScopeScheduling._split_scope` | [`loadscope.py:284`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L284) | the only affinity primitive | +| `LoadFileScheduling._split_scope` | [`loadfile.py:35`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadfile.py#L35) | two-line override | +| `LoadGroupScheduling._split_scope` | [`loadgroup.py:24`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadgroup.py#L24) | two-line override | +| collection-mismatch abort | [`load.py:259`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/load.py#L259) · [`loadscope.py:359`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L359) | logs and runs zero tests | +| `xdist_group` node-id append | [`remote.py:245-254`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/remote.py#L245-L254) | worker-side, `loadgroup` only | + +## pytest-asyncio — `v1.4.0` + +[`pytest-dev/pytest-asyncio @ v1.4.0`](https://github.com/pytest-dev/pytest-asyncio/tree/v1.4.0) · +[`pytest_asyncio/plugin.py`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py) + +| Symbol | Anchor | Cited for | +|---|---|---| +| `Mode` | [`:82`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L82) | `str` enum so ini, CLI and internal value are one object | +| `PytestAsyncioSpecs` | [`:90`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L90) | its own hookspec namespace | +| `pytest_addoption` | [`:108`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L108) | every option `default=None` | +| `_make_asyncio_fixture_function` | [`:210`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L210) | stamping scope on the function | +| `_get_asyncio_mode` | [`:222`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L222) | resolve once, query once | +| `pytest_configure` | [`:295-301`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L295-L301) | detecting an unset default via the sentinel | + +## Sphinx — `v9.1.0` + +[`sphinx-doc/sphinx @ v9.1.0`](https://github.com/sphinx-doc/sphinx/tree/v9.1.0) · +[`sphinx/ext/doctest.py`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py) + +| Symbol | Anchor | Cited for | +|---|---|---| +| `is_allowed_version(spec, version)` | [`:45`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L45) | specifier first — the reverse of the local helper | +| `TestDirective` | [`:66`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L66) | the directive base and its option handling | +| comment nodetype rule | [`:92-93`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L92-L93) | `testsetup`/`testcleanup`/`:hide:` become `nodes.comment` | +| `:options:` gating | [`:111`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L111) | accepted only on `doctest` and `testoutput` | +| `TestGroup` / `add_code` | [`:200`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L200) · [`:207`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L207) | phase ordering; three silent-loss cases | +| `TestCode` | [`:235`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L235) | the parsed unit | +| `SphinxDocTestRunner` | [`:257`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L257) | overrides a private method to swallow an `IndexError` | +| `DocTestBuilder` | [`:292`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L292) | builder coupling | +| `doctest.compile` rebinding | [`:310`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L310) | process-global, never restored | +| `test_doc` | [`:428`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L428) | group resolution and `*` | +| gated-node drop | [`:443-444`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L443-L444) | no outcome, id or count | +| `type = "exec"` for testcode | [`:548`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L548) | the mode flip | + +Documentation: [`doc/usage/extensions/doctest.rst`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/doc/usage/extensions/doctest.rst). +Registry behaviour: [`sphinx/util/docutils.py`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/util/docutils.py), +[`sphinx/application.py`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/application.py). + +## MyST-Parser — `v5.1.0` + +[`executablebooks/MyST-Parser @ v5.1.0`](https://github.com/executablebooks/MyST-Parser/tree/v5.1.0) + +| Symbol | Anchor | +|---|---| +| `create_myst_settings_spec` | [`parsers/docutils_.py:208`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/parsers/docutils_.py#L208) | +| `Parser(RstParser)` | [`parsers/docutils_.py:235`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/parsers/docutils_.py#L235) | +| `settings_spec` | [`parsers/docutils_.py:241-245`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/parsers/docutils_.py#L241-L245) | +| `MdParserConfig` (`myst_enable_extensions`, `myst_fence_as_directive`) | [`config/main.py`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/config/main.py) | + +## docutils — `docutils-0.22.2` + +The canonical repository is on +[SourceForge](https://sourceforge.net/p/docutils/code/); the GitHub copies are +third-party mirrors and are not linked here. Anchors name file and symbol at the +tagged release: + +| File | Symbol | Cited for | +|---|---|---| +| `docutils/parsers/rst/directives/__init__.py` | `_directives` | the process-global, rebindable registry | +| `docutils/parsers/rst/states.py` | `state_classes`, `doctest_block` line assignment | per-instance substitutability; last-line convention | +| `docutils/utils/__init__.py` | `Reporter.attach_observer`, `system_message` | observation separable from display | +| `docutils/nodes.py` | `Element.attributes`, `literal_block`, `comment`, `doctest_block` | the untyped attribute channel | + +Typed surface: [`typeshed stubs/docutils`](https://github.com/python/typeshed/tree/8c7256c/stubs/docutils). + +## Prior art + +| Project | Ref | Key anchors | +|---|---|---| +| Sybil | [`10.0.1`](https://github.com/simplistix/sybil/tree/10.0.1) | [`sybil.py:155-157`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/sybil.py#L155-L157) (positional ids) · [`document.py`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/document.py) (one namespace, non-overlap invariant) · [`integration/pytest.py`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/integration/pytest.py) (one item per region) · [`region.py`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/region.py) · [`testing.py`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/testing.py) (public extension-test helpers) | +| xdoctest | [`v1.3.2`](https://github.com/Erotemic/xdoctest/tree/v1.3.2) | [`directive.py:58`](https://github.com/Erotemic/xdoctest/blob/v1.3.2/src/xdoctest/directive.py#L58) (`REQUIRES` carries its reason) · [`plugin.py`](https://github.com/Erotemic/xdoctest/blob/v1.3.2/src/xdoctest/plugin.py) (unregisters pytest's doctest plugin) | +| pytest-examples | [`v0.0.18`](https://github.com/pydantic/pytest-examples/tree/v0.0.18) | [`find_examples.py`](https://github.com/pydantic/pytest-examples/blob/v0.0.18/pytest_examples/find_examples.py) · [`run_code.py`](https://github.com/pydantic/pytest-examples/blob/v0.0.18/pytest_examples/run_code.py) · [`modify_files.py`](https://github.com/pydantic/pytest-examples/blob/v0.0.18/pytest_examples/modify_files.py) (byte offsets, invertible dedent, unguarded splice) | +| typeshed | [`8c7256c`](https://github.com/python/typeshed/tree/8c7256c) | [`stdlib/doctest.pyi`](https://github.com/python/typeshed/blob/8c7256c/stdlib/doctest.pyi) | diff --git a/notes/analyses/README.md b/notes/analyses/README.md new file mode 100644 index 0000000..4114e78 --- /dev/null +++ b/notes/analyses/README.md @@ -0,0 +1,69 @@ +# Doctest ecosystem structural analyses + +Structural analysis of how the systems `doctest_docutils` sits between are built — +their core data structures, data flows, extension seams, and configuration models. +These are research notes. They inform the ADRs in `docs/adrs/` and decide nothing +themselves. + +The question they exist to answer: a doctest engine that must be vanilla-compatible +at the core, pluggable, usable as a pytest plugin, usable with docutils and +myst-parser, *and* speak all three communities' idioms is standing on three +upstreams with three separate extension models and three overlapping vocabularies. +What exactly does each of them require, and where do they contradict each other? + +## Method + +1. **Portable citations first.** Every external reference is a deep link to a + specific file **pinned at a git tag** — never `main`, `master`, `HEAD` or a bare + SHA. Line anchors are only used on a pinned ref, because they are meaningless + without one. These links are the reproducible source surface for the analysis. +2. **Source review second.** Confirm and deepen against checked-out source with + `rg`/`fd`. Local notes may inform drafting; tracked notes must be readable and + verifiable without a workstation path. +3. **Execute the load-bearing claims.** Anything an ADR rests on is run, not read. + Where a note says "verified", a snippet was executed and its output recorded. + +## Pinned versions + +| Project | Repo | Ref | +|---|---|---| +| CPython (`doctest`, `asyncio`) | `python/cpython` | `v3.14.2` | +| pytest | `pytest-dev/pytest` | `9.1.1` | +| pytest-xdist | `pytest-dev/pytest-xdist` | `v3.8.0` | +| pytest-asyncio | `pytest-dev/pytest-asyncio` | `v1.4.0` | +| Sphinx | `sphinx-doc/sphinx` | `v9.1.0` | +| MyST-Parser | `executablebooks/MyST-Parser` | `v5.1.0` | +| Sybil | `simplistix/sybil` | `10.0.1` | +| xdoctest | `Erotemic/xdoctest` | `v1.3.2` | +| typeshed | `python/typeshed` | `8c7256c` (no tags; commit reachable from trunk) | +| docutils | SourceForge (the GitHub clones are third-party mirrors) | `docutils-0.22.2` | + +## Files + +- [`00-taxonomy.md`](00-taxonomy.md) — the design axes, as a classification matrix. +- Per-system structural docs: [`10-cpython-doctest.md`](10-cpython-doctest.md), + [`11-pytest-doctest.md`](11-pytest-doctest.md), + [`12-pytest-xdist.md`](12-pytest-xdist.md), + [`13-pytest-asyncio.md`](13-pytest-asyncio.md), + [`14-asyncio.md`](14-asyncio.md), + [`15-sphinx-ext-doctest.md`](15-sphinx-ext-doctest.md), + [`16-docutils-myst.md`](16-docutils-myst.md), + [`17-prior-art.md`](17-prior-art.md). +- Cross-cutting: [`20-data-structures.md`](20-data-structures.md), + [`21-data-flows.md`](21-data-flows.md), + [`22-extension-seams.md`](22-extension-seams.md), + [`23-namespace-scope-and-test-identity.md`](23-namespace-scope-and-test-identity.md). +- [`90-bibliography.md`](90-bibliography.md) — every pinned anchor cited by the + ADRs, in one place. + +Each per-system doc follows the same section order — classification · core data +structures · data flow · extension seams · configuration · what it cannot do · +anchors — so the systems are directly comparable, and the cross-cutting docs can +line them up column by column. + +`14-asyncio.md` is included even though `asyncio` has nothing to do with doctests. +It is the stdlib's own worked example of a pluggable architecture built out of +protocols, an abstract base, a policy indirection and a runner, by roughly the same +people and in roughly the same era as `doctest`'s extension model. Reading the two +side by side is the cheapest available answer to "what does the standard library +consider a good seam, and why does `doctest` have so few of them?" From 5a79db1043acf1a9ea56d1d7f0abd8586934a255 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 13:21:45 -0500 Subject: [PATCH 04/31] docs(adrs[0001]): Correct the record against verified evidence why: Review found the record described PR #87's unmerged design as the status quo, and several claims did not survive checking against source. Every corrected claim below was re-verified by reading a pinned tag or by executing it. what: - Rewrite Context: trunk collects one DocTest per page; the groups, merge, skip lifting, exec runner and xdist scheduler are PR #87's proposal, named as such - Drop "per-block SKIPPED" from what the shape buys free, and add an outcome contract: TestReport.outcome is one scalar per item, so a mixed group either erases the skip or over-reports the whole group. Record subtests as the only sanctioned alternative and why it is not adopted - Credit sphinx.ext.doctest with the execution shape; the pytest identity is what is novel, not N DocTests per namespace - Replace the bare DocTest tuple with PlannedBlock and GroupPlan, so run_group can order phases, evaluate the gate and guarantee cleanup - Replace the compile-mode literal with a private ExecutionProfile, since PR #59's top-level await is a second policy a mode string cannot express - Move docutils node vocabulary out of the stdlib-only leaf, and move settings below the layers that read it - Add the item lifecycle contract, since half-reusing DoctestItem reintroduces the clear_globs wipe - Correct Sphinx: :options: on testcode is an unknown-option error, not a silent discard; :pyversion: is the silent one; cleanup does not run after setup failure, so always-cleanup is a divergence - Correct the parsefactories claim: conftest autouse fixtures arrive via FixtureManager.pytest_plugin_registered - Correct the nominal-subclassing claim: stdlib accepts a duck-typed parser or finder; typeshed is what demands the class - Narrow the xdist sentence: identical collection still binds - Repin Sphinx anchors to v8.2.3, the version this project resolves --- docs/adrs/0001-typed-vanilla-doctest-core.md | 370 +++++++++++++------ 1 file changed, 262 insertions(+), 108 deletions(-) diff --git a/docs/adrs/0001-typed-vanilla-doctest-core.md b/docs/adrs/0001-typed-vanilla-doctest-core.md index 51a4c9f..23f86f5 100644 --- a/docs/adrs/0001-typed-vanilla-doctest-core.md +++ b/docs/adrs/0001-typed-vanilla-doctest-core.md @@ -7,66 +7,77 @@ Date: 2026-08-02 ## Context +### What ships today + `doctest_docutils` re-implements the *finding* half of {func}`doctest.testfile` -over a docutils or MyST doctree, keeps CPython's *running* half, and adds a -grouping layer — namespace, phase, merge, lift — that exists in neither parent. +over a docutils or MyST doctree and keeps CPython's *running* half. `pytest_doctest_docutils` wraps that in a pytest plugin. -The two modules work. They are also the join point of three separate -vocabularies and three separate extension models, and the seams between them -were never named. Four structural costs follow. +Two facts about the shipped code set the problem. + +**A page is one test.** `DocutilsDocTestFinder` produces a single +{class}`doctest.DocTest` per document. A page cannot build state across the prose +that explains it, because there is no unit smaller than the page and no unit that +groups blocks within one. + +**The plugin blocks the plugin whose internals it imports.** +`pytest_configure` calls `config.pluginmanager.set_blocked("doctest")`, and the +same module then imports that plugin's private helpers. This survives only +because `_pytest/fixtures.py` has no `pytest_plugin_unregistered` handler, so the +already-parsed `doctest_namespace` fixture outlives unregistration. + +### What a shared namespace costs + +[PR #87](https://github.com/git-pull/gp-libs/pull/87) is an open, unmerged +attempt at the first problem: it adds Sphinx-style groups, a merge step, phase +ordering, skip lifting, an exec-mode runner and an xdist scheduler. **None of it +has shipped in any release, and none of it is on trunk.** It is described here as +a design under review, not as the status quo, because what it had to build to +work is the evidence this record turns on. + +Three costs are worth naming, because a clean-room design must either pay them +again or explain why it does not. -**The finder is one method.** `DocutilsDocTestFinder._find` performs format -dispatch, node filtering, group resolution, wildcard expansion, name-collision -detection, `:skipif:` evaluation, `testoutput` pairing, test construction, -merging, splitting and naming, in one body with two nested closures, with -configuration threaded through as positional parameters. There is no boundary -between *document → blocks*, *blocks → tests*, and *tests → items*, so a change -to any one of them is a change to all three. +**Merging blocks fights line-number fidelity.** A merged group is one `DocTest` +with one `docstring` and one `lineno`, and both doctest's `%03d` gutter and +pytest's `repr_failure` reconstruct locations by slicing that single string. So +the blocks must be laid out on a synthetic page with blank-line padding and a +clamp, and a wholly-skipped block must be lifted back out to report at all. -**It reaches through name-mangled CPython privates.** Sphinx's prompt-free -`{testcode}` form needs `compile()` in `"exec"` mode, but the per-example loop +**Prompt-free `{testcode}` needs a CPython private.** The per-example loop [`DocTestRunner.__run`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1344) hard-codes `"single"` -([`Lib/doctest.py:1400`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1400)). -`sphinx.ext.doctest` solves this by rebinding `doctest.compile` process-wide and -never restoring it -([`sphinx/ext/doctest.py:310`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L310)) — +([`Lib/doctest.py:1400`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1400)), +and `sphinx.ext.doctest` gets around that by rebinding `doctest.compile` +process-wide and never restoring it +([`sphinx/ext/doctest.py:310`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L310)) — unavailable to a library that loads into every pytest session that installed it. -`_exec_mode_run` therefore clones the mangled method's code object into a fresh -{class}`types.FunctionType` whose globals map `compile` to a local helper, guarded -by probing `co_freevars` and `co_names` and degrading with a logged error. That -mechanism also carries a latent defect: its globals are a snapshot of -`vars(doctest)` taken at import, so a later rebind of a module-level name in -`doctest` is invisible to the clone while remaining visible to the stock runner, -and two runners in one process disagree. - -**It blocks the plugin whose internals it imports.** `pytest_configure` calls -`config.pluginmanager.set_blocked("doctest")`, and the same module then imports -that plugin's private helpers — `_is_setup_py`, `_is_main_py`, `_get_checker`, -`_get_continue_on_failure` — and reads four ini and CLI options the blocked -plugin declared. This survives only because `_pytest/fixtures.py` has no -`pytest_plugin_unregistered` handler, so the already-parsed `doctest_namespace` -fixture outlives unregistration. - -**It forks pytest-xdist to protect state xdist cannot see.** Because a live -`globs` mapping is a Python object and only execnet-serializable builtins cross a -worker boundary, a shared namespace must either be merged into one -{class}`doctest.DocTest` or kept on one worker. Keeping it there means a -scheduler, and the only affinity primitive in all of xdist is +PR #87's answer clones the mangled method's code object into a fresh +{class}`types.FunctionType` whose globals map `compile` to a local helper. That +carries a latent defect: those globals are a snapshot of `vars(doctest)` taken at +import, so a later rebind of a module-level name in `doctest` is invisible to the +clone while remaining visible to the stock runner, and two runners in one process +disagree. + +**A live shared mapping forces an xdist fork.** Only execnet-serializable +builtins cross a worker boundary, so a shared namespace must either be merged +into one `DocTest` or kept on one worker. Keeping it there means a scheduler, and +the only affinity primitive in all of xdist is [`LoadScopeScheduling._split_scope`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L284), a pure function on node-id strings. The controller never collects; it learns the -suite only as node ids arriving from workers. So `_shared_page` and `_is_page` -re-derive "these ids share state" from strings, and `_worker_count` re-implements +suite only as node ids arriving from workers. So PR #87 re-derives "these ids +share state" from strings, and re-implements [`parse_tx_spec_config`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/workermanage.py#L26) including its quirks. -Underneath all four is one conflation. **The granularity of test identity and the -granularity of shared state are different axes, and the current design couples -them.** `namespace_items = merged` gives one `DocTest` and one node id per group; -`per-block` gives N `DocTest`s, N node ids, and one live mapping — a node id that -raises `NameError` when selected alone. Every surveyed project makes the same -mistake or a worse one (see [](#prior-art)). +### The conflation underneath + +All three costs follow from one thing. **The granularity of test identity and the +granularity of shared state are different axes, and every surveyed design couples +them.** PR #87's two settings are the clearest illustration: `merged` gives one +`DocTest` and one node id per group; `per-block` gives N `DocTest`s, N node ids +and one live mapping — a node id that raises `NameError` when selected alone. +Sybil ships the second shape without acknowledging it (see [](#prior-art)). ## Decision @@ -76,11 +87,27 @@ One {class}`pytest.Item` per (document, group). Inside it, a tuple of per-block `DocTest`s run in phase order against one live `globs` mapping that never leaves the item. -This occupies the cell no surveyed project occupies: N `DocTest`s, one node id. -Per-block failure locations, per-block gutters, per-block `SKIPPED`, per-block -"location unknown" — while `-k`, `--lf`, `-x`, `--reruns` and every `--dist` mode -are structurally incapable of splitting the shared state, because there is only -one item to schedule. +The execution shape is not new: `sphinx.ext.doctest` already runs several +`DocTest`s against one shared group namespace. What is new is giving that shape a +**pytest identity**. Sphinx produces no selectable, reportable unit for a group — +every block in it shares one `DocTest.name`, which is why `SphinxDocTestRunner` +overrides a private stdlib method to swallow the resulting `IndexError`. Mapping +the group onto one {class}`pytest.Item` is the contribution. + +What that buys, for free and with no override of `repr_failure` or `reportinfo`: +per-block failure locations, per-block gutters, and per-block "location unknown". +Meanwhile `-k`, `--lf`, `-x`, `--reruns` and every `--dist` mode are structurally +incapable of splitting the shared state, because there is only one item to +schedule. + +**A per-block `SKIPPED` outcome is not among them.** +{class}`pytest.TestReport`'s `outcome` is one scalar per item, so a group holding +one all-`SKIP` block and one passing block reports `PASSED` with the skip erased. +Signalling the skip instead flips the *whole* group to `SKIPPED`. Surfacing it +per block requires pytest's builtin-but-experimental `subtests` plugin, appears +in the terminal gutter and `-rs` only at `verbosity_subtests >= 1`, and never +becomes a separate ``, `--lf` entry or rerunnable unit. See +[](#the-outcome-contract). ### The three facts this rests on @@ -109,9 +136,9 @@ and `lineno` for free. A block reached through `.. include::` reports the takes pytest's honest `EXAMPLE LOCATION UNKNOWN` branch **without poisoning its siblings**. No override of `repr_failure` or `reportinfo` is required. -This is what retires `_merge_blocks`: the synthetic merged page, its blank-line -padding and its `max(..., len(lines))` clamp exist only to reconstruct locations -from a single spliced docstring. +This is what makes merging unnecessary: the synthetic page, its blank-line +padding and its clamp exist only to reconstruct locations from a single spliced +docstring, and there is no spliced docstring here. **2. `_DocTestRunner__run` is an ordinary attribute override.** Name mangling rewrites the *call site* at compile time, so the `self.__run(...)` lookup inside @@ -148,43 +175,109 @@ variables, all in its own `finally` ([`Lib/doctest.py:1534-1573`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1534-L1573)). That contract is not reproduced; it is inherited. -**3. Making the item the sharing unit dissolves distribution.** A live mapping -never crosses a process boundary, so there is nothing for xdist to split, under -any `--dist` mode. No affinity primitive, no scheduler substitution, no +**3. Making the item the sharing unit dissolves the *affinity* problem.** A live +mapping never crosses a process boundary, so there is nothing for xdist to split, +under any `--dist` mode. No affinity primitive, no scheduler substitution, no `parse_tx_spec_config` fork, no node-id string sniffing. +The identical-collection requirement is untouched by this and still binds at any +granularity. It is satisfied separately, by collection being a pure function of +(bytes, argv, ini) — which is why `:skipif:` is carried through collection +unevaluated. + +(the-outcome-contract)= + +### The outcome contract + +One item means one item outcome. That is a real cost of this design and it is +stated here rather than discovered later. + +| Signal | Granularity | Notes | +|---|---|---| +| Failure location, `want`/`got`, gutter | **per block** | `repr_failure` iterates failures and reads each one's own `DocTest` | +| `EXAMPLE LOCATION UNKNOWN` | **per block** | a block with `lineno=None` does not affect its siblings | +| `passed` / `failed` / `skipped` | **per item** | `TestReport.outcome` is one scalar | +| JUnit `` | **per item** | node reporters are keyed by node id | +| `--lf`, `-k`, `--deselect`, rerun unit | **per item** | | + +The skip case is the sharp edge, and it cuts both ways. If the item swallows a +block's skip, a group with one all-`SKIP` block and one passing block reports +`PASSED` and the skip leaves no record — no count, no `-rs` line, no JUnit +``. If the item raises instead, the whole group reports `SKIPPED` even +though a sibling passed. pytest's own doctest plugin takes the second horn only +when *every* example is skipped, via `_check_all_skipped`. + +This design takes the same position: **skip the item when every runnable block is +skipped; otherwise report partial skips as typed block detail, not as a pytest +outcome.** No extra reports are synthesized. + +`subtests` — a builtin pytest plugin since 9.0, exporting `pytest.Subtests` and +`pytest.SubtestReport` — can emit per-block outcomes, and is the only sanctioned +mechanism that can. It is not adopted here: pytest documents it as experimental, +its output is invisible at default verbosity, and it produces no separate JUnit +entry, so it would buy terminal detail at the cost of depending on an unstable +surface. Revisit if it stabilizes. + ### Layers Dependencies flow one way, from the leaf toward the hosts. No layer may import a layer above it. ```text +settings one frozen Settings, resolved once <- leaf; everything reads it + | blocks inert data: Block, BlockKind, Phase, Diagnostic, Example | (stdlib imports only) markup/ _rst, _myst, _text, _python -> (blocks, diagnostics) | -project grouping, pairing, phase order, naming -> GroupTest +project grouping, pairing, phase order, naming -> GroupPlan | runner owns the per-example loop; phase sequencing | -settings one frozen Settings, resolved once - | pytest_doctest_docutils collection, items, globs lifetime, reporting ``` +`settings` sits at the bottom, not beside the host, because `Frontend.parse` and +`project()` both take a `Settings`. A layer every other layer reads is a leaf. + | Layer | Owns | Must not know | |---|---|---| -| `blocks` | `Block`, `BlockKind` registry, `Phase`, `Diagnostic`, `BlockAttributes`, `Example(doctest.Example)` carrying compile mode | docutils, MyST, Sphinx, pytest, xdist, the filesystem. Stdlib imports only, enforced by an `import-linter` contract | -| `markup/` | Text → `(blocks, diagnostics)`. Line-number recovery and its per-front-end meaning, `.. include::` attribution, `nodes.comment` traversal, reporter capture, idempotent directive registration | Groups as a runtime concept, `DocTest`, pytest, pairing | +| `settings` | One frozen `Settings` resolved once, with `None` sentinels at the resolve boundary so a future default change is announceable | pytest's `Config`, argparse, ini format, Sphinx's `app`. The host extracts; this resolves | +| `blocks` | `Block`, `BlockKind` registry, `Phase`, `Diagnostic`, `PlannedBlock`, `GroupPlan`, `ExecutionProfile`, `Example(doctest.Example)` | docutils, MyST, Sphinx, pytest, xdist, the filesystem. Stdlib imports only, enforced by an `import-linter` contract | +| `markup/` | Text → `(blocks, diagnostics)`. **The whole docutils vocabulary**: which node classes each kind may arrive as, the `BlockAttributes` stamp, line-number recovery and its per-front-end meaning, `.. include::` attribution, `nodes.comment` traversal, reporter capture, idempotent directive registration, per-kind `option_spec` | Groups as a runtime concept, `DocTest`, pytest, pairing | | `project` | The **only** place grouping exists: `*` expansion, anonymous naming, phase order, `testcode`/`testoutput` pairing, option defaults, name minting. A pure function | docutils, pytest, the filesystem, whether anything will run. Evaluates no user code | -| `runner` | `_DocTestRunner__run`; option merge, `SKIP`-after-merge, `FAIL_FAST`, `report_*` dispatch, version shims. `run_group()` owns phase sequencing and the `try`/`finally` guaranteeing `testcleanup` | docutils, markup, pytest. Never overrides `run()` | -| `settings` | One frozen `Settings` resolved once, with `None` sentinels at the resolve boundary so a future default change is announceable | pytest's `Config`, argparse, ini format, Sphinx's `app` | -| `pytest_doctest_docutils` | Options, `Document(pytest.Module)`, `DocutilsItem`, group `globs` lifetime, built-in-plugin dedup, surfacing diagnostics | docutils node classes, MyST configuration, grouping rules | +| `runner` | `_DocTestRunner__run`; option merge, `SKIP`-after-merge, `FAIL_FAST`, `report_*` dispatch, version shims. `run_group()` owns phase sequencing, run-time `:skipif:`, the profile's context manager, and the `try`/`finally` guaranteeing `testcleanup` | docutils, markup, pytest. Never overrides `run()` | +| `pytest_doctest_docutils` | Options, `Document(pytest.Module)`, `DocutilsItem`, group `globs` lifetime, the outcome contract, built-in-plugin composition, surfacing diagnostics | docutils node classes, MyST configuration, grouping rules | `pytest_doctest_docutils._compat` is the only module that imports `_pytest.doctest`, behind a pinned support matrix. See {doc}`0006-pytest-private-api-compatibility`. +### Item lifecycle + +The custom item is load-bearing, and half-reusing {class}`pytest.DoctestItem` +reintroduces the exact bug this design exists to avoid. The contract, stated so +an implementer cannot get it wrong by omission: + +1. **Collection** builds one `GroupPlan` per (document, group) and one item per + plan. An empty plan yields no item. +2. **`setup()`** clears the group mapping in place, then calls `super().setup()` + so fixtures inject into that same object. Clearing in place rather than + rebinding is what keeps `item.globs is plan.globs` true for every block, and + what stops attempt two of a `--reruns` run from reading attempt one's + mutations. +3. **`runtest()`** is overridden. It must not delegate to + `DoctestItem.runtest`, which runs a single `dtest` with `clear_globs` + defaulting to `True` — that would empty the shared mapping after the first + block. It calls `run_group()`, which runs each `PlannedBlock` in phase order + with `clear_globs=False`, evaluates `:skipif:` against the group mapping, and + wraps the body in a `try`/`finally` so cleanup runs whether or not the body + raised. +4. **Outcome** follows [](#the-outcome-contract): skip the item only when every + runnable block is skipped. +5. **`repr_failure`** is inherited unchanged. It already reads each failure's own + `DocTest`. + ### Vocabulary Goal (e) — speaking doctest's, pytest's *and* Sphinx's idioms — is mostly a @@ -197,8 +290,8 @@ referents. Each term below is decided once and used only that way. | namespace | — | [`doctest_namespace`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L721) means *injected names* | — | **Not** used for the sharing unit; pytest owns the word | | group | — | `xdist_group` is a *scheduling* affinity marker ([`remote.py:245-254`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/remote.py#L245-L254)) | the author-facing bucket: `.. doctest:: intro`, `default`, `*` | Adopt `group` for the sharing unit. The xdist affinity key is *derived*, never the group name | | scope | — | the fixture-lifetime ladder | — | Reserved for pytest. The block-vs-document axis becomes `share` | -| test / item / block | `DocTest`, `Example` | `Item` | [`TestCode`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L235) is the parsed unit | Three nouns: `Example` (stdlib), `Block` (parsed), `DocTest` (runnable) | -| skip | the `SKIP` flag, short-circuiting before `report_start` | a reported outcome with a reason | [drops the node entirely](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L443-L444) | pytest's meaning, doctest's mechanism. Sphinx's drop is deliberately rejected | +| test / item / block | `DocTest`, `Example` | `Item` | [`TestCode`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L235) is the parsed unit | Three nouns: `Example` (stdlib), `Block` (parsed), `DocTest` (runnable) | +| skip | the `SKIP` flag, short-circuiting before `report_start` | a reported outcome with a reason, **at item granularity** | [drops the node entirely](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L443-L444) | doctest's mechanism (set `SKIP`, never drop the node); pytest's outcome where the granularity allows it — see [](#the-outcome-contract). Sphinx's drop is deliberately rejected | | directive | inline `# doctest: +FLAG` | — | a docutils directive with an `option_spec` | Reserved for the docutils meaning. doctest's form is "inline flags" | | optionflags | an int bitmask; `register_optionflag` | `doctest_optionflags` ini | `:options:` plus `doctest_default_flags` | Keep verbatim. `register_optionflag` is the one genuinely cross-library extension point | | setup / cleanup | `setUp`/`tearDown` on the suite builders | fixtures | `testsetup`/`testcleanup` directives | Author-facing names stay Sphinx's; `phase` is the internal ordering axis; a fixture is never "setup" | @@ -229,17 +322,21 @@ class Block(t.NamedTuple): class BlockKind(t.NamedTuple): name: str phase: Phase - mode: t.Literal["single", "exec"] + profile: ExecutionProfile # how a body of this kind is compiled and run pairs_with: str | None grouped: bool - node_types: tuple[str, ...] # docutils tagnames; "comment" for testsetup/:hide: - option_spec: t.Mapping[str, t.Callable[[str], object]] -class GroupTest(t.NamedTuple): +class PlannedBlock(t.NamedTuple): + phase: Phase + skipif: str | None # UNEVALUATED; gated in run_group(), not at collection + test: doctest.DocTest # one block, its own filename/lineno/docstring + + +class GroupPlan(t.NamedTuple): group: str - tests: tuple[doctest.DocTest, ...] # one per block, in phase order - globs: dict[str, t.Any] # the live mapping, shared by every test above + blocks: tuple[PlannedBlock, ...] # in phase order + globs: dict[str, t.Any] # the live mapping, shared by every block above ``` `Block.line` being nullable is load-bearing, not defensive. A bare `>>>` block @@ -248,26 +345,51 @@ source=None` from docutils, and an `.. include::`-ed block numbers against the *included* file. Both propagate to `DocTest.lineno=None` and pytest's honest "location unknown", rather than to a fabricated number. -`BlockKind.node_types` is likewise load-bearing: `testsetup`, `testcleanup` and -any `:hide:` block are emitted as {class}`docutils.nodes.comment`, not -`literal_block` -([`sphinx/ext/doctest.py:92-93`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L92-L93)). -A walker restricted to `literal_block` silently loses every one of them while the -page still renders. +`PlannedBlock` exists because `run_group()` owns phase sequencing, run-time +`:skipif:` evaluation and a cleanup `finally` — and cannot do any of the three +from a bare tuple of `DocTest`s. A `DocTest` carries no phase and no gate, so the +plan has to. Tagging each entry also makes the ordering self-describing rather +than a convention a comment asserts. + +`ExecutionProfile` is a **private** protocol carrying compile mode, extra compile +flags, and an optional per-group context manager. It is not a `Literal["single", +"exec"]`, because a second execution policy already exists in this repository: +[PR #59](https://github.com/git-pull/gp-libs/pull/59) adds top-level `await`, +which needs `ast.PyCF_ALLOW_TOP_LEVEL_AWAIT` and an event-loop lifetime that a +mode string cannot express. The profile's context manager is what `run_group()` +enters, so that lifetime is served without overriding `run()` and stdlib's +save-and-restore `finally` stays inherited. It stays private until a caller +outside the package needs it; PR #59 is the in-repo caller that justifies it +existing at all. + +**Which docutils node classes a kind may arrive as is a front-end concern, not a +`BlockKind` field.** `testsetup`, `testcleanup` and any `:hide:` block are +emitted as {class}`docutils.nodes.comment`, not `literal_block` +([`sphinx/ext/doctest.py:92-93`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L92-L93)), +and a walker restricted to `literal_block` silently loses every one of them while +the page still renders. That requirement is real, but it belongs to `markup/` +alongside the rest of the node vocabulary — putting it on `BlockKind` would drag +docutils into a layer declared stdlib-only. ### Typing The runtime objects are stdlib's, unconditionally. Precision lives in a parallel layer that never changes what is constructed. -- **`Protocol` for the seams, nominal subclassing for the concrete classes.** - `Frontend` is a `Protocol` so a third party can supply one structurally; the - shipped parser also subclasses {class}`doctest.DocTestParser` so it stays - passable to `doctest.testfile(parser=...)` and - {func}`doctest.DocFileSuite`. Typeshed's signatures demand the class, not the - shape — `DocTestFinder.__init__(parser: DocTestParser = ...)` — so a compatible - `find()` alone is not enough, which is why `DocutilsDocTestFinder` cannot be - passed to `DocTestSuite(test_finder=...)` today. +- **`Protocol` for the seams, nominal subclassing for type-checkability.** + `Frontend` is a `Protocol` so a third party can supply one structurally. The + shipped parser *also* subclasses {class}`doctest.DocTestParser` — not because + the interpreter requires it (stdlib `doctest` performs no `isinstance` check on + `parser` or `test_finder`; a duck-typed object works at runtime) but because + typeshed's signatures name the class, so a checker rejects what the interpreter + accepts. The subclass buys type-checkability, not passability. Passability is a + matter of matching the call signature: a finder whose `find()` takes a string + first cannot be handed to `DocTestSuite`, which passes a module — and + subclassing does not fix that. + + One genuine nominal edge does exist: `DocTestSuite` sorts its results, and + `DocTest.__lt__` returns `NotImplemented` for a non-`DocTest`, so a custom + finder must return real `DocTest` objects. - **`TypedDict` at the docutils boundary.** `BlockAttributes` types what a directive stamps on a node, with one narrowing accessor that validates once. This is where `Any` currently enters: `str(node.get("test") or ...)` and @@ -312,7 +434,8 @@ that owns the loop must probe for it rather than assume it. | `DoctestItem.setup()` does `self.dtest.globs.update(globs)`, so the mapping must be mutable and survive collection → setup → run | [`doctest.py:288-293`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L288-L293) | | `runtest()` calls `run(self.dtest, out=failures)` with `clear_globs` defaulting to `True` — which would empty a shared mapping after the first block | [`doctest.py:295-303`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L295-L303) | | `PytestDoctestRunner` is defined *inside* `_init_runner_class()` and is not importable, so the `OutcomeException` re-raise, `BdbQuit` → `outcomes.exit` and `continue_on_failure` handling do **not** come for free | [`doctest.py:178-181`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L178-L181) | -| A page must be a `pytest.Module` with `obj = None`, and the collector must call `parsefactories` | [`doctest.py:421`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L421), [`:556`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L556) | +| A page must be a `pytest.Module` with `obj = None` as a **class** attribute, or the `Module` machinery tries to import the `.rst`/`.md` file | [`doctest.py:420-421`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L420-L421) | +| Conftest autouse fixtures reach page items through `FixtureManager.pytest_plugin_registered`, **not** through a collector calling `parsefactories` — that call is `DoctestModule`-only, for fixtures defined in the collected `.py` itself | [`fixtures.py`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/fixtures.py), [`doctest.py:556`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L556) | | `_is_doctest` claims any `.txt`/`.rst` **initial path before consulting `--doctest-glob`**, so `pytest docs/page.rst` is claimed by the built-in plugin regardless of glob | [`doctest.py:148-152`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L148-L152) | | An empty `DocTest` must not be yielded as an item | [`doctest.py:451`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L451) | @@ -325,18 +448,33 @@ that owns the loop must probe for it rather than assume it. | `xdist_group` is honoured only when the *worker's own* `--dist` is `loadgroup`, and works by appending `@name` to `item._nodeid` | [`remote.py:245-254`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/remote.py#L245-L254) | | `parse_tx_spec_config` builds a list, so a negative multiplier contributes zero specs — `xspeclist.extend([spec] * num)`, not a sum | [`workermanage.py:26-37`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/workermanage.py#L26-L37) | -Making the item the sharing unit removes the need to satisfy the first three at -all. The fourth is why `_worker_count` is deleted rather than fixed. +Making the item the sharing unit removes the need to satisfy the second and third +at all. **The first still binds at any granularity** — identical collection is +required whether a page yields one item or fifty — and is satisfied instead by +collection being a pure function of (bytes, argv, ini). The fourth is why a +worker-count fork is not worth carrying: `pytest_xdist_setupnodes(config, specs)` +hands over the already-expanded spec list and never raises. -### Sphinx (v9.1.0) +There is a fifth hazard the single-item shape also removes, worth naming because +it has no guard otherwise: a worker crash re-runs only the *uncompleted* items of +a work unit on a fresh process, so blocks 3..N of a shared group would run +against an empty mapping. Worker restarts are on by default. + +### Sphinx (v8.2.3, the version this project resolves) | Constraint | Anchor | |---|---| -| `testsetup`, `testcleanup` and `:hide:` blocks are emitted as `nodes.comment` | [`doctest.py:92-93`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L92-L93) | -| A `:skipif:`-gated node is dropped during collection, with no outcome, id or count | [`doctest.py:443-444`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L443-L444) | -| `:options:` is accepted only on `doctest` and `testoutput`; a `testcode`'s own options are discarded in favour of its output block's | [`doctest.py:111`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L111), [`:207`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L207) | -| `is_allowed_version(spec, version)` takes the specifier **first** — the reverse of this project's local helper | [`doctest.py:45`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L45) | -| `DocTestBuilder` flips a mutable `self.type` between `"single"` and `"exec"` and reads it through a process-global `doctest.compile` patch | [`doctest.py:310`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L310), [`:548`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L548) | +| `testsetup`, `testcleanup` and `:hide:` blocks are emitted as `nodes.comment` | [`doctest.py:92-93`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L92-L93) | +| A `:skipif:`-gated node is dropped during collection, with no outcome, id or count | [`doctest.py:449-450`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L449-L450) | +| `:options:` is **not in `TestcodeDirective.option_spec`**, so writing it on a `testcode` is an unknown-option error that drops the block — a loud rejection, not a silent discard | [`doctest.py:174-180`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L174-L180), [`:111`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L111) | +| `:pyversion:` **is** in `TestcodeDirective.option_spec` and is silently ignored there — the real silent loss on a testcode | [`doctest.py:177`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L177) | +| Cleanup does **not** run when setup fails: the group returns early | [`doctest.py:554-556`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L554-L556) | +| `is_allowed_version(spec, version)` takes the specifier **first** | [`doctest.py:45`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L45) | +| `DocTestBuilder` flips a mutable `self.type` between `"single"` and `"exec"` and reads it through a process-global `doctest.compile` patch | [`doctest.py:310`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L310), [`:549`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L549) | + +Sphinx v9.1.0 changes the default group an unargumented block joins. Any +group-naming behaviour matched against "Sphinx" has to say which Sphinx, and +{doc}`0005-line-recovery-for-nested-blocks` proposes moving this floor. ## Tensions @@ -371,25 +509,41 @@ evaluates no user Python, so it is a pure function of (bytes, argv, ini). *Price `--collect-only` no longer shows which blocks will skip. **Sphinx compatibility versus silent-loss behaviours.** Sphinx silently discards -an orphan `testoutput`, silently overwrites a duplicate one, and silently drops a -`testcode`'s own `:options:`. *Position:* keep the behaviour, add a diagnostic -with a stable code. *Price:* a page warns under pytest and is silent under -`sphinx-build`; the results still match. +an orphan `testoutput`, silently discards a `testoutput` following a `doctest` +block, silently overwrites a duplicate `testoutput`, and silently ignores +`:pyversion:` on a `testcode`. *Position:* keep the behaviour, add a diagnostic +with a stable code for each of the four. *Price:* a page warns under pytest and +is silent under `sphinx-build`; the results still match. + +**Guaranteed cleanup versus Sphinx's setup-failure short-circuit.** When setup +fails, Sphinx returns before the cleanup phase, skipping page `testcleanup` +blocks and `doctest_global_cleanup` alike. *Position:* run cleanup +unconditionally in a `try`/`finally`, because a page that spawns a server in +setup and fails mid-way should not leak it. *Price:* a page whose setup fails +leaves different residue under pytest than under `sphinx-build`, and that is a +deliberate divergence rather than an oversight. **Owning the loop versus tracking CPython.** *Position:* own it, because constraint 4 makes it the only way to control compile mode without a process-global patch or a code-object clone. *Price:* a version shim and a conformance harness. See {doc}`0002-runner-conformance-across-cpython`. -## What this deletes +## What this avoids + +Only one item here exists on trunk today: `set_blocked("doctest")` and its +unblock path, which {doc}`0006-pytest-private-api-compatibility` replaces. The +rest are machinery [PR #87](https://github.com/git-pull/gp-libs/pull/87) has to +build to make a shared namespace work, and which this shape never needs: -`_find`'s monolith · `_merge_blocks` and its padding and clamp · -`_split_skipped_blocks` and `_lifted_name` · `_exec_mode_run`, `_compile_source` -and `_ExecSource` · `_worker_count` · `_shared_page` and `_is_page` · -`_splitting_scheduler` and both xdist hooks · the `_init_runner_class` fork · -`set_blocked("doctest")` and its unblock path. +- the merge step, its blank-line padding and its `max()` clamp +- skip lifting, which pulls a wholly-skipped block back out of a running group +- the code-object clone with its stale `vars(doctest)` snapshot +- the worker-count fork of `parse_tx_spec_config` +- the node-id string sniffing that infers "these ids share state" +- the scheduler substitution and both xdist hooks -**The result is not smaller.** It lands roughly flat against the current source. +**The result is not smaller.** It lands roughly flat against a finder that does +the same job. Under this project's rule that every function carries a NumPy docstring with a working doctest, splitting one long method into five functions costs prose it did not previously pay. The value is fewer hazards, not fewer lines: a code-object From 7394c730ce9b338ed167d3a7417a52e445fd70d6 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 13:22:47 -0500 Subject: [PATCH 05/31] docs(adrs[0003]): Recast per-block items as a rejected design MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit why: The record proposed deprecating doctest_docutils_namespace_items, but neither that setting nor its scope twin has shipped — both live on PR #87, in no release and on no tag. Deprecating an unshipped setting fails the Published-Release Test, and there is no downstream to warn. what: - Retitle and reslug: the question is whether the shape should ship, not how to retire it - State plainly that nothing shipped, so there is no migration path, no warning and no downstream grep - Give the shape its due first: merging costs node ids, fixture lifetime and gutter locality, which is what per-block answers - Then give the four reasons against, each with its guard: an id that NameErrors when selected, a mapping that cannot cross a worker, a mapping that cannot survive a rerun, and a crash tail that has no guard at all - Note that the guards are the cost of the shape, not incidental - Record the honest limit of the alternative: no per-block outcome and no per-block node id --- docs/adrs/0001-typed-vanilla-doctest-core.md | 2 +- docs/adrs/0003-rejecting-per-block-items.md | 81 +++++++++++++++++++ ...0003-retiring-per-block-namespace-items.md | 54 ------------- docs/adrs/index.md | 2 +- 4 files changed, 83 insertions(+), 56 deletions(-) create mode 100644 docs/adrs/0003-rejecting-per-block-items.md delete mode 100644 docs/adrs/0003-retiring-per-block-namespace-items.md diff --git a/docs/adrs/0001-typed-vanilla-doctest-core.md b/docs/adrs/0001-typed-vanilla-doctest-core.md index 23f86f5..6feb84f 100644 --- a/docs/adrs/0001-typed-vanilla-doctest-core.md +++ b/docs/adrs/0001-typed-vanilla-doctest-core.md @@ -688,7 +688,7 @@ the narrow default set in {doc}`0004-diagnostics-as-data`. This ADR fixes the architecture. Five decisions it defers get their own records: {doc}`0002-runner-conformance-across-cpython` (how the owned loop is proven -equivalent), {doc}`0003-retiring-per-block-namespace-items` (the deprecation +equivalent), {doc}`0003-rejecting-per-block-items` (the deprecation path), {doc}`0004-diagnostics-as-data` (what is reported and what is suppressed), {doc}`0005-line-recovery-for-nested-blocks` (the optional last step), and {doc}`0006-pytest-private-api-compatibility` (the quarantine and its matrix). diff --git a/docs/adrs/0003-rejecting-per-block-items.md b/docs/adrs/0003-rejecting-per-block-items.md new file mode 100644 index 0000000..f3c5ebc --- /dev/null +++ b/docs/adrs/0003-rejecting-per-block-items.md @@ -0,0 +1,81 @@ +(adr-0003-rejecting-per-block-items)= + +# ADR 0003: Rejecting per-block items over a shared mapping + +Status: Draft +Date: 2026-08-02 + +## Context + +[PR #87](https://github.com/git-pull/gp-libs/pull/87) proposes two settings that +together choose how a page's blocks are collected. One of them, +`doctest_docutils_namespace_items = per-block`, keeps a node id for every block +of a shared page and hands those blocks one live `globs` mapping rather than +merging them into a single test. + +**Neither setting has shipped.** Both live on an open branch, in no release and +on no tag. There is nothing to deprecate, and this record does not propose a +deprecation — it records why the shape should not ship. + +## The shape, and why it is attractive + +`per-block` answers a real complaint about merging. Merging a group into one +`DocTest` collapses N node ids into one, merges fixture lifetime across the whole +group, and makes the failure gutter span the page. Keeping one id per block fixes +all three, and on a large documentation tree the difference is the bulk of the +suite's visible granularity. + +## Why it should not ship + +**A node id that cannot be selected is not a node id.** Selecting block three of +a stateful page raises `NameError`, because the blocks that bound the names it +reads did not run. The id promises an addressable unit and does not deliver one. + +**A live mapping cannot cross a process.** Only execnet-serializable builtins +reach an xdist worker, so the shape needs a scheduler that keeps a page whole — +and the only affinity primitive in xdist is +[`_split_scope(nodeid) -> str`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L284). +Under a user-typed `--dist load` there is no scheduler to influence, so the +options collapse to refusing the run. + +**A live mapping cannot survive an item running twice.** A retry re-runs a block +against globals it already mutated, so an expectation true only on the second +attempt reports as a pass. Guarding that means refusing reruns. + +**A worker crash re-runs only the uncompleted tail** of a work unit, on a fresh +process — so blocks 3..N of a shared group run against an empty mapping, and +worker restarts are on by default. This one has no guard at all. + +Those four are why the branch also carries a worker-count fork, a page-inference +heuristic over node-id strings, a scheduler substitution, a scheduler refusal and +a rerun refusal. The guards are the cost of the shape, not incidental. + +## Decision + +Do not ship per-block items over a shared mapping, under this or any spelling. + +{doc}`0001-typed-vanilla-doctest-core` reaches the same granularity goal from the +other side: one item per group, holding one `DocTest` per block. That gives +per-block failure locations, gutters and "location unknown" without a shared +mapping ever becoming schedulable, so none of the four guards is needed. + +What it does not give is a per-block *outcome* or a per-block *node id*. That +limit is honest and is recorded in {doc}`0001-typed-vanilla-doctest-core`'s +outcome contract, rather than papered over with an id that raises when used. + +## Consequences + +Because nothing shipped, there is no migration path to write, no deprecation +warning to add and no downstream grep to run. If PR #87 lands before this +architecture does, this record converts into exactly those things — and that is +the argument for settling the two in order rather than in parallel. + +## Open + +- Whether a human-facing block *label* — in failure text and the report header, + never as a node id — is worth adding later, so a reader can find the failing + block without the design promising `-k` isolation. Not in a first version. +- Whether `--doctest-docutils-namespace-scope` should be renamed to + `--doctest-docutils-share` before or after this architecture lands. + {doc}`0001-typed-vanilla-doctest-core` settles the vocabulary; the rename is + independently schedulable and, since neither spelling has shipped, cheap. diff --git a/docs/adrs/0003-retiring-per-block-namespace-items.md b/docs/adrs/0003-retiring-per-block-namespace-items.md deleted file mode 100644 index e9c7787..0000000 --- a/docs/adrs/0003-retiring-per-block-namespace-items.md +++ /dev/null @@ -1,54 +0,0 @@ -(adr-0003-retiring-per-block-namespace-items)= - -# ADR 0003: Retiring `namespace_items = per-block` - -Status: Draft -Date: 2026-08-02 - -## Context - -`doctest_docutils_namespace_items` selects between `merged` (the default: a group -collects as one test) and `per-block` (each block keeps its own node id while the -blocks share one live `globs` mapping). - -{doc}`0001-typed-vanilla-doctest-core` makes the setting unnecessary by -decoupling the two axes it was invented to trade between: one pytest item per -group, with one {class}`doctest.DocTest` per block inside it. That gives -`per-block`'s per-block failure locations, gutters and `SKIPPED` reporting -without `per-block`'s shared live mapping. - -It also removes `per-block`'s cost. A live mapping is a Python object, so it -neither crosses a worker process nor survives an item running twice. Guarding it -is the sole reason `_worker_count`, `_shared_page`, `_is_page`, -`_splitting_scheduler` and both xdist hooks exist. - -The setting's remaining distinguishing feature is a node id per block — and those -ids raise `NameError` when selected alone, because selecting one block does not -run the blocks that bound the names it reads. A node id that cannot be selected -is not a node id. - -## Question - -`per-block` is a shipped CLI option with an ini twin and its own how-to sections. -How does it get retired without breaking a user who set it? - -## Direction - -Announce it as a breaking change with the migration path stated plainly, and keep -the option accepting `per-block` for one minor release, mapped to `merged` with a -{class}`pytest.PytestDeprecationWarning`. - -Before landing, confirm no downstream consumer sets it. -`--doctest-docutils-modules` and `--no-doctest-docutils-modules` keep their exact -current spelling and `dest`, because downstream projects carry them in `addopts`. - -## Open - -- Whether the deprecation shim warns once per session or once per page. -- Whether `--doctest-docutils-namespace-scope` is renamed to `--doctest-docutils-share` - in the same release or a later one. {doc}`0001-typed-vanilla-doctest-core` - decides the vocabulary — `scope` is reserved for pytest's fixture-lifetime - ladder — but the rename is independently schedulable. -- Whether the `pytest11` entry point rename (from `sphinx`, which is what - `-p no:sphinx` disables today) ships in the same release as this removal or its - own. diff --git a/docs/adrs/index.md b/docs/adrs/index.md index ed763f1..02af2f3 100644 --- a/docs/adrs/index.md +++ b/docs/adrs/index.md @@ -31,7 +31,7 @@ the anchor lands on unrelated code while still resolving. 0001-typed-vanilla-doctest-core 0002-runner-conformance-across-cpython -0003-retiring-per-block-namespace-items +0003-rejecting-per-block-items 0004-diagnostics-as-data 0005-line-recovery-for-nested-blocks 0006-pytest-private-api-compatibility From d86ecf61c6599749943abfdabcc7cb12ab24e940 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 13:23:19 -0500 Subject: [PATCH 06/31] docs(adrs[0005]): Raise the docutils floor instead why: The proposed mechanism does not work and was not needed. Nested state machines build from nested_sm_kwargs, so substituting a parser instance's state_classes never reaches the constructs with missing lines; and RSTState.nested_sm_cache is a shared class attribute, so the substitution is not scoped to one parse either. Meanwhile docutils 0.22 fixed the defect upstream. what: - Replace the substitution mechanism, the feature probe and the fallback design with a floor bump to docutils >=0.22 - Record why the mechanism failed, so it is not proposed again - Name the real cost: docutils >=0.22 requires Sphinx >=9.1 - Version-qualify the line conventions, and state that an unqualified claim about docutils line numbers is a bug in the claim - Keep the nullable line and the normalization layer, which serve .. include:: attribution regardless of version --- .../0005-line-recovery-for-nested-blocks.md | 89 ++++++++++++------- 1 file changed, 57 insertions(+), 32 deletions(-) diff --git a/docs/adrs/0005-line-recovery-for-nested-blocks.md b/docs/adrs/0005-line-recovery-for-nested-blocks.md index f24616b..79325b1 100644 --- a/docs/adrs/0005-line-recovery-for-nested-blocks.md +++ b/docs/adrs/0005-line-recovery-for-nested-blocks.md @@ -1,59 +1,84 @@ (adr-0005-line-recovery-for-nested-blocks)= -# ADR 0005: Exact line recovery for nested blocks +# ADR 0005: Line recovery for nested blocks Status: Draft Date: 2026-08-02 ## Context -docutils does not report a usable line for every node, and what it does report -differs by front-end. +docutils does not report a usable line for every node, and what it reports +differs by front-end and by version. -A bare `>>>` block nested in a `.. note::`, a list item, a block quote or a -`{tab}` directive reports `line=None, source=None`. A top-level reStructuredText -`doctest_block` reports its **last** line; a MyST fence reports its **first**. An +At **docutils 0.21.2**, which this project currently pins, a bare `>>>` block +nested in a `.. note::`, a list item, a block quote or a `{tab}` directive +reports `line=None, source=None`. A top-level reStructuredText `doctest_block` +reports its **last** line. A MyST fence reports its **first** line. An `.. include::`-ed block numbers against the *included* file. -{doc}`0001-typed-vanilla-doctest-core` handles all three honestly rather than -approximately. `Block.line` is nullable, the per-front-end meaning is normalized +{doc}`0001-typed-vanilla-doctest-core` handles all of this honestly rather than +approximately: `Block.line` is nullable, the per-front-end meaning is normalized inside the front-end that knows it, `Block.path` carries the file the text actually lives in, and a block with no recoverable line propagates `DocTest.lineno=None` into pytest's `EXAMPLE LOCATION UNKNOWN` branch. That is correct but not maximal. A nested block's failure says the location is -unknown when the information exists in the parser and was discarded. +unknown when the parser knew it and threw it away. -## Question +## The mechanism this record originally proposed does not work -Can the exact line be recovered for a nested block without a fragile dependency? +The idea was to substitute `docutils.parsers.rst.Parser.state_classes` per +parser instance, on the reasoning that `state_classes` is an instance attribute +and therefore scoped to one parse. -The mechanism identified is per-instance substitution of -`docutils.parsers.rst.Parser.state_classes`. `state_classes` is an instance -attribute, so substitution is fully scoped to one parse with no process-global -mutation — unlike the directive registry, which has no such scoping. +It fails on two counts, both checked: -It also depends on undocumented docutils structure, and a refactor upstream would -break it as `None` spans rather than as an exception. +**It does not reach nested blocks.** A nested parse builds its state machine from +`nested_sm_kwargs`, so a substitution applied only to the top-level +`state_classes` never reaches the constructs that need it — which are exactly the +constructs with the missing lines. -## Direction +**It is not scoped.** `RSTState.nested_sm_cache` is a shared *class* attribute, +so substituted classes leak into subsequent parses that did not ask for them. +The claim that substitution is "fully scoped to one parse with no process-global +mutation" is wrong. -Land it last, as a strictly optional improvement behind a feature probe. +## Decision -When the expected structure is absent, fall back to `node.line` normalization and -emit `line=None`. Degrading to an honest disclaimer is acceptable; degrading to a -fabricated number is not, and that is the whole reason this is separable from -{doc}`0001-typed-vanilla-doctest-core` rather than part of it. +**Raise the docutils floor instead.** -Pin with tests asserting exact `(path, line)` for a bare block nested in each of -the four constructs, plus a test that forces the probe to fail and asserts the -fallback yields `lineno=None`. +docutils 0.22 fixed the underlying defect upstream: a nested block reports a real +line, and top-level and nested blocks agree on reporting the **first** line +rather than the last. Every case this record was invented to work around is +resolved by a floor bump, with no probe, no substitution and no fallback path. + +The cost is that docutils ≥ 0.22 requires Sphinx ≥ 9.1, so this is a coordinated +dependency move rather than a one-line pin change. + +## Consequences + +The nullable `Block.line` stays. It is not a workaround for this defect; it is +the honest representation of a front-end that may legitimately not know, and +`.. include::` attribution still needs `Block.path` regardless of version. + +The line-convention normalization in `markup/` gets *simpler* at the new floor — +both reStructuredText and MyST report the first line — but the normalization +layer stays, because the conventions still differ below the floor and a front-end +is the right place to know which it is dealing with. + +Every line-convention claim elsewhere in these records is version-qualified. +A statement about "docutils" that does not name a version is a bug in the +statement. ## Open -- Whether the same technique gives MyST nested blocks anything, or whether - markdown-it's token stream already carries enough. -- Whether `.. include::` line attribution needs the same treatment or is already - correct once `Block.path` is respected. -- Whether the probe result is reported anywhere, so a user can tell which mode - produced a given report. +- Whether to raise the floor now or support both, since 0.21.2 is what the + project pins today. Supporting both means keeping the normalization branch and + documenting two behaviours for the same page. +- Whether the Sphinx ≥ 9.1 move belongs in this record or its own. It changes the + default group an unargumented block joins, which is a semantics change beyond + line numbers. +- Whether tests should pin exact `(path, line)` for a bare block nested in a + `.. note::`, a list item, a block quote and a `{tab}` directive. They should — + they are the regression net for the floor, and this repository already has + `{tab}` coverage from the GH-48 regression. From 6fb5c6bc949c937ee6307db1a8ca71953b1c861b Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 13:24:23 -0500 Subject: [PATCH 07/31] docs(adrs[0006]): Filter the collector instead of deselecting items why: Deselecting in pytest_collection_modifyitems runs too late. DoctestTextfile.collect() reads and parses the page inside collect(), so by then the built-in has already produced an item, or already reported a collection error that deselection cannot retract. what: - Filter the built-in's collector out of the pytest_collect_file result in a hookimpl wrapper, before it parses anything - Record that narrowing --doctest-glob cannot help either, because _is_doctest claims .rst initial paths before consulting the glob - Move the compatibility failure from plugin registration to the collection of an affected document, matching 0001 and 0002 on not taking down a session whose tests never touch a doctest - Replace the settled open question with the one the filter raises: scoping it to paths this plugin actually claims --- .../0006-pytest-private-api-compatibility.md | 40 ++++++++++++++----- 1 file changed, 29 insertions(+), 11 deletions(-) diff --git a/docs/adrs/0006-pytest-private-api-compatibility.md b/docs/adrs/0006-pytest-private-api-compatibility.md index 943e9d9..35c6503 100644 --- a/docs/adrs/0006-pytest-private-api-compatibility.md +++ b/docs/adrs/0006-pytest-private-api-compatibility.md @@ -39,6 +39,14 @@ configuration, and `pytest_collect_file` is not `firstresult` — the directory collector yields the results of *every* implementation for a path. Declining the path is therefore not sufficient; the duplicate has to be removed. +**Narrowing `--doctest-glob` cannot help**, because `_is_doctest` returns `True` +for an `.rst` initial path *before* it consults the glob at all. + +**And removing the duplicate late is too late.** `DoctestTextfile.collect()` +reads and parses the page inside `collect()`, so by the time +`pytest_collection_modifyitems` runs, the built-in has already produced an item — +or already reported a collection error, which deselection cannot retract. + ## Question What private surface is depended on, and how does a pytest release that changes @@ -56,13 +64,22 @@ reimplemented rather than inherited. ## Direction Quarantine every private import in one module, `pytest_doctest_docutils._compat`, -with a pinned support matrix and an import-time probe that raises a named error at -plugin registration — naming the pytest version and the missing symbol — rather -than failing somewhere in the middle of collection. - -Claim `.rst` and `.md` paths and deselect the built-in plugin's duplicate items in -a `pytest_collection_modifyitems` hookwrapper, firing `pytest_deselected` so the -reported counts stay consistent. +with a pinned support matrix. + +**Filter the built-in's collector out of the `pytest_collect_file` result, in a +`@pytest.hookimpl(wrapper=True)`.** The directory collector consumes the +multicall result directly, and returning a modified result from a hook wrapper is +documented and supported. Filtering there removes the duplicate *before* the +built-in collector parses anything, so neither the duplicate item nor its +collection error is ever produced — which late deselection cannot achieve. + +**Fail on an unsupported pytest only when an affected document is collected**, +not at plugin registration. A `pytest11` plugin that raises at import takes down +sessions whose majority of tests never touch a doctest, and +{doc}`0001-typed-vanilla-doctest-core` and +{doc}`0002-runner-conformance-across-cpython` both reject session-wide startup +failures for the same reason. The error names the pytest version and the missing +symbol, and it names the document that triggered it. CI carries a job pinned to the minimum supported pytest and one tracking its prerelease. @@ -72,10 +89,11 @@ prerelease. - Whether requiring the built-in plugin should be stated as a hard dependency. `-p no:doctest` already fails today with a raw `ValueError` about an unknown option, so this is not a regression — but the message should become actionable. -- Whether deselecting is preferable to narrowing the built-in's globs. Deselecting - is explicit and reportable; narrowing is quieter but mutates another plugin's - configuration. - Whether the probe should accept a *newer* pytest it has not been tested against, - or refuse it. Refusing is safer and more annoying. + or refuse it. Refusing is safer and more annoying; for a private-API quarantine + with a small matrix, safer probably wins. +- What the filtering wrapper should do when the built-in's collector is the *only* + one for a path — that is the ordinary `--doctest-glob` case this plugin has no + business touching, so the filter must be scoped to paths it actually claims. - Whether any of these helpers can be promoted upstream, which would delete the quarantine entirely. From 370deaeee9316da6009529e0a5a441108c01b45a Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 13:24:23 -0500 Subject: [PATCH 08/31] docs(adrs[0002]): Assert the result triple, not TestResults equality why: TestResults is a two-field namedtuple carrying skipped off-tuple, so == compares two of three values and a skip-count regression passes the harness silently. attempted is also incremented before the SKIP check, so a skip that wrongly executes moves neither counter. what: - Assert (failed, attempted, skipped) explicitly, and say why the tuple comparison is insufficient - Add an exec-mode case to the matrix, and note it has no stock counterpart since a stock runner rejects a multi-statement body --- .../0002-runner-conformance-across-cpython.md | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/docs/adrs/0002-runner-conformance-across-cpython.md b/docs/adrs/0002-runner-conformance-across-cpython.md index bd7882e..f7183d9 100644 --- a/docs/adrs/0002-runner-conformance-across-cpython.md +++ b/docs/adrs/0002-runner-conformance-across-cpython.md @@ -50,9 +50,22 @@ gating the build step that lands the runner. A fixed case matrix — pass, fail, unexpected exception, `SyntaxError`, all examples skipped, partially skipped, `FAIL_FAST`, `REPORT_ONLY_FIRST_FAILURE`, -`IGNORE_EXCEPTION_DETAIL` — is run through both this runner and a stock -{class}`doctest.DocTestRunner`, asserting equality of `TestResults`, the captured -`report_*` text, `summarize()` output, and the accumulator contents. +`IGNORE_EXCEPTION_DETAIL`, and an exec-mode body — is run through both this +runner and a stock {class}`doctest.DocTestRunner`, asserting the captured +`report_*` text, `summarize()` output, the accumulator contents, and the result +as `(failed, attempted, skipped)`. + +**Assert the triple, not `TestResults` equality.** `TestResults` is a two-field +namedtuple carrying `skipped` off-tuple, so `==` compares only two of the three +values and a skip-count regression passes silently. `attempted` is also +incremented *before* the `SKIP` check, so a skip that wrongly executes moves +neither counter — it is invisible to both the tuple and to `summarize()` at zero +failures, and only the `report_*` text distinguishes it. + +The exec-mode case has no stock counterpart to compare against: a stock runner +raises `SyntaxError: multiple statements found` on a multi-statement body. It is +asserted against a recorded expectation instead, and that asymmetry is the point +— it is the one behaviour this runner exists to add. Version handling is by capability probe, never by version comparison, so a backport, a vendored interpreter or a fork behaves correctly rather than by From 0dc57a32eb23c8aee5d04c65c2d0cbe6fbb5cfcd Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 13:28:29 -0500 Subject: [PATCH 09/31] notes(analyses): Correct the claims review falsified why: Review checked these notes against source and found several assertions wrong. Each correction below was re-verified by reading a pinned tag or by executing it. what: - Repin to what the project resolves: docutils 0.21.2, Sphinx 8.2.3, and myst-parser split across two Python floors. Repin every Sphinx anchor, including the gated-drop line, which moved between tags - Sphinx runs one DocTest per BLOCK against one shared group namespace. Move it into the N-DocTests row of the product space; its "one node id" is really no id, since every block shares DocTest.name - Sphinx rejects :options: on a testcode with an unknown-option error rather than discarding it. The silent loss there is :pyversion:. Add the doctest-then-testoutput case, and the setup-failure short-circuit that skips cleanup - parsefactories collects fixtures from the .py being collected; conftest autouse fixtures arrive via pytest_plugin_registered - stdlib doctest accepts a duck-typed parser or finder. Typeshed is what names the class, so a nominal subclass buys type-checkability, not passability - A regex can parse directive options; Sybil does. Justify the docutils dependency on host fidelity instead - Version-qualify the line conventions, since docutils 0.22 changed them - Correct the pytest-asyncio note: it does own an item class, it does convert str to enum, and the None sentinel is used on three of six options rather than all - Record subtests as pytest's only sanctioned sub-item mechanism --- notes/analyses/00-taxonomy.md | 10 ++- notes/analyses/11-pytest-doctest.md | 13 ++- notes/analyses/13-pytest-asyncio.md | 58 ++++++++---- notes/analyses/15-sphinx-ext-doctest.md | 90 ++++++++++++------- notes/analyses/16-docutils-myst.md | 15 ++-- notes/analyses/17-prior-art.md | 15 ++-- notes/analyses/20-data-structures.md | 4 +- notes/analyses/21-data-flows.md | 6 +- notes/analyses/22-extension-seams.md | 13 ++- .../23-namespace-scope-and-test-identity.md | 10 ++- notes/analyses/90-bibliography.md | 47 +++++----- notes/analyses/README.md | 6 +- 12 files changed, 180 insertions(+), 107 deletions(-) diff --git a/notes/analyses/00-taxonomy.md b/notes/analyses/00-taxonomy.md index 0f6bf22..dc53189 100644 --- a/notes/analyses/00-taxonomy.md +++ b/notes/analyses/00-taxonomy.md @@ -67,10 +67,12 @@ abandoning the stdlib object model, and it is now building the bridge back. The cost of divergence is paid years later, in knobs that exist only to restore the default that was abandoned. -**Axis 4 is decided by axis 6.** A regex cannot see `:skipif:` or a group name, -which is why Sybil has no group concept at all and tells users to clear the -namespace instead. If directive options are part of the product, the document -model must be a parse tree. +**Axis 4 is decided by host fidelity, not by lexing power.** A regex *can* parse +directive arguments and options — Sybil's directive lexers do it. What a regex +cannot give you is the same tree Sphinx renders from, and that is what makes a +page behave identically under `sphinx-build` and under pytest. Sybil having no +group concept, and telling users to clear the namespace instead, is a design +choice rather than a limit of its lexer. **Axis 7 correlates with hostility.** Two of the surveyed projects disable pytest's doctest plugin — one in `pytest_configure`, one by telling users to pass diff --git a/notes/analyses/11-pytest-doctest.md b/notes/analyses/11-pytest-doctest.md index 5f616fa..b365dac 100644 --- a/notes/analyses/11-pytest-doctest.md +++ b/notes/analyses/11-pytest-doctest.md @@ -16,7 +16,8 @@ replace it, and the model this project's pytest layer should resemble. DoctestItem(Item) dtest: DocTest, runner: DocTestRunner, fixture_request | obj = None (class attribute) DoctestTextfile(Module) obj = None; one DocTest for the whole file -DoctestModule(Module) one DocTest per docstring; calls parsefactories +DoctestModule(Module) one DocTest per docstring; parsefactories for + fixtures defined in the collected .py itself MultipleDoctestFailures carries a list; the workaround for stdlib having no per-example result value ReprFailDoctest (ReprFileLocation, lines) pairs @@ -26,9 +27,15 @@ ReprFailDoctest (ReprFileLocation, lines) pairs ([`:421`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L421)) is what keeps `Module` from trying to import a `.txt`/`.rst` file. Subclassing `Module` rather than `File` is what makes `scope="module"` fixtures resolve -against the page — a page collector *is* the module scope. `parsefactories` +against the page — a page collector *is* the module scope. + +`parsefactories` ([`:556`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L556)) -is what makes autouse fixtures from a visible `conftest.py` apply at all. +is `DoctestModule`-only, and it collects fixtures defined *in the `.py` being +collected*. It is **not** what makes conftest autouse fixtures apply — those are +registered through `FixtureManager.pytest_plugin_registered` when the conftest is +loaded, independently of any collector. A page collector needs no `parsefactories` +call to see them. ## Data flow diff --git a/notes/analyses/13-pytest-asyncio.md b/notes/analyses/13-pytest-asyncio.md index ecdd322..3ba4400 100644 --- a/notes/analyses/13-pytest-asyncio.md +++ b/notes/analyses/13-pytest-asyncio.md @@ -9,10 +9,18 @@ a default it needed to change without breaking anyone. ## Classification -A hook-driven behaviour plugin. It contributes no collector of its own for the -common case; it normalizes configuration into one question, answers it once, and -then injects behaviour through pytest's existing machinery rather than owning the -item class. +A hook-driven behaviour plugin that **does** own its item class. It defines +`PytestAsyncioFunction(Function)` +([`:506`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L506)) +with four concrete subclasses — `Coroutine`, `AsyncGenerator`, +`AsyncStaticMethod`, `AsyncHypothesisTest` — and a `pytest_pycollect_makeitem` +hookwrapper +([`:689-723`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L689-L723)) +that substitutes them for every collected async `Function`. + +That substitution-by-hookwrapper is itself the pattern worth noting: it swaps the +item class without owning collection, so pytest still decides *what* is a test +and the plugin only decides *how* it runs. ## Core data structures @@ -23,9 +31,13 @@ PytestAsyncioSpecs its own hookspec namespace [:90] _ScopeName reuses pytest's scope vocabulary verbatim ``` -`Mode` inheriting `str` is a small but deliberate choice: the ini value, the CLI -value and the internal enum are the same object, so no conversion layer exists to -drift. +`Mode` inherits `str`, but a conversion layer still exists and is exactly where +drift would occur: `_get_asyncio_mode` +([`:222-232`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L222-L232)) +reads the CLI value, falls back to the ini value, and calls `Mode(val)` inside a +`try`, translating a `ValueError` into a {exc}`pytest.UsageError` that lists the +valid modes. The lesson is not "no conversion" — it is that the conversion +happens **once**, in one named function, with a good error. Declaring its own `HookspecMarker("pytest")` namespace is the interesting one. A third party extends pytest-asyncio by implementing a hook, not by subclassing @@ -47,7 +59,7 @@ _get_asyncio_mode(config) -> Mode, resolved once [:222] | v in AUTO mode: item.add_marker("asyncio") - | => the rest of the plugin has exactly ONE query path + | => marker presence becomes the single question downstream asks v fixture/loop resolution by scope, then pyfunc call wrapping ``` @@ -64,14 +76,16 @@ fixture/loop resolution by scope, then pyfunc call wrapping ## What is worth stealing -**The `default=None` sentinel.** Every option is declared with `default=None` -([`:114`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L114), -[`:122`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L122), -[`:140`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L140)) -rather than with its effective default. The plugin can therefore distinguish "the -user chose the current default" from "the user has not chosen", which is what -makes a future default change *announceable* — it warns only the second group. -`pytest_configure` uses exactly this to warn about an unset +**The `default=None` sentinel, used selectively.** Of the six options +`pytest_addoption` declares +([`:108-147`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L108-L147)), +three carry `None` and three carry their effective default — so this is a +technique applied where it earns its keep, not a blanket rule. + +It earns its keep on the options whose default the project intends to move. A +`None` lets the plugin distinguish "the user chose the current default" from "the +user has not chosen", which is what makes a future change *announceable* — only +the second group is warned. `pytest_configure` does exactly that for an unset `asyncio_default_fixture_loop_scope` ([`:296-301`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L296-L301)). @@ -91,9 +105,15 @@ scope names verbatim. It does not invent a third word for lifetime. Compare the current `namespace_scope`/`namespace_items` naming, which collides with two pytest concepts at once. -**Session-wide errors raise `pytest.UsageError`; per-item errors raise -`ValueError`.** The severity matches the blast radius. A misspelled session -setting should stop the session; a bad per-item value should fail that item. +**Session-wide errors raise {exc}`pytest.UsageError`.** An invalid `asyncio_mode` +and an invalid loop-scope ini value both raise it, and both are reached from +session-level config in `pytest_configure`. A misspelled session setting stops the +session, which is the right blast radius for a value that would otherwise +mis-apply to every item. + +The mirror rule — per-item errors raising something narrower — is *not* something +this plugin demonstrates cleanly, so do not cite it as precedent. Marker parsing +is one function with one blast radius. The session half is the transferable part. ## What it cannot tell us diff --git a/notes/analyses/15-sphinx-ext-doctest.md b/notes/analyses/15-sphinx-ext-doctest.md index 7be7bbc..3590dd7 100644 --- a/notes/analyses/15-sphinx-ext-doctest.md +++ b/notes/analyses/15-sphinx-ext-doctest.md @@ -1,6 +1,8 @@ # `sphinx.ext.doctest` -Pinned at [`v9.1.0`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py). +Pinned at [`v8.2.3`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py), +the version this project resolves. v9.1.0 changes the default group an +unargumented block joins; nothing else below differs between the two. ## Classification @@ -26,10 +28,20 @@ SphinxDocTestRunner(doctest.DocTestRunner) [:257] ``` `TestGroup.tests` holds heterogeneous entries — `[code]` for a bare block, -`[code, output]` for a paired testcode/testoutput. `add_code` is where three -silent losses live: an orphan `testoutput` is discarded, a second `testoutput` -*replaces* the first, and a `testcode`'s own options are dropped in favour of its -output block's. +`[code, output]` for a paired testcode/testoutput. `add_code` is where the silent +losses live: an orphan `testoutput` is discarded; a `testoutput` following a +`doctest` block is discarded, because a doctest entry has length 1 and fails the +`len(latest_test) == 2` guard; and a second `testoutput` *replaces* the first. + +A fourth silent loss is in the directive rather than `add_code`: `:pyversion:` is +in `TestcodeDirective.option_spec` +([`:174-180`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L174-L180)) +and is then ignored, because the version gate only runs for `doctest` and +`testoutput`. + +`:options:` on a `testcode` is **not** a silent loss. It is absent from +`TestcodeDirective.option_spec` entirely, so writing it is an unknown-option +error that drops the block — loud, and visible in the build output. ## Data flow @@ -46,17 +58,20 @@ doctree | DocTestBuilder.test_doc(docname, doctree) [:428] | for node in doctree.findall(condition): - | if self.skipped(node): continue <- GATED BLOCK IS DROPPED [:443-444] + | if self.skipped(node): continue <- GATED BLOCK IS DROPPED [:449-450] | code = TestCode(...) | "*" in groups -> add to every group | else groups[name].add_code(code) v per group: ns = {} | three runners: setup / test / cleanup, sharing one _fakeout - | test.globs = ns (assigned AFTER construction) + | ONE doctest.DocTest PER BLOCK, each with test.globs = ns + | (assigned AFTER construction, since __init__ copies) | runner.run(test, out=..., clear_globs=False) - | self.type flipped to "exec" for setup, cleanup, testcode [:548, :608] + | self.type flipped to "exec" for setup, cleanup, testcode [:549, :608] | to "single" for ordinary doctests [:580] + | + | if setup fails -> RETURN. Cleanup does not run. [:554-556] v six integer counters + text streamed to outdir/output.txt ``` @@ -65,10 +80,10 @@ six integer counters + text streamed to outdir/output.txt | Seam | Kind | |---|---| -| `TestDirective` subclassing, with `option_spec` ([`:66`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L66)) | subclass | +| `TestDirective` subclassing, with `option_spec` ([`:66`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L66)) | subclass | | The node attribute stamp — `testnodetype`, `groups`, `options`, `skipif`, `test` | implicit protocol | | `doctest_global_setup`, `doctest_global_cleanup`, `doctest_test_doctest_blocks`, `doctest_default_flags` | Sphinx confvals | -| `is_allowed_version(spec, version)` ([`:45`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L45)) | function | +| `is_allowed_version(spec, version)` ([`:45`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L45)) | function | The node attribute stamp is the important one, and it is undocumented. It is the only decoupled interface in the module: any directive that emits a @@ -82,22 +97,29 @@ rather than trusting one's own directive classes is the only defence against | Rule | Anchor | |---|---| -| `testsetup`, `testcleanup` and `:hide:` render as `nodes.comment` | [`:92-93`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L92-L93) | -| `:options:` is accepted only on `doctest` and `testoutput` | [`:111`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L111) | -| A gated block is dropped during collection — no outcome, id or count | [`:443-444`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L443-L444) | -| `*` means every group the document declares; `default` means no argument was given | [`:428`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L428) onward | -| Setup runs before tests, cleanup after, whatever order the page writes them | `TestGroup` [`:200-226`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L200-L226) | -| `is_allowed_version` takes the **specifier first** | [`:45`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L45) | -| `doctest.compile` is rebound process-wide and never restored | [`:310`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L310) | +| `testsetup`, `testcleanup` and `:hide:` render as `nodes.comment` | [`:92-93`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L92-L93) | +| `:options:` is accepted only on `doctest` and `testoutput`; on a `testcode` it is an unknown-option error, not a discard | [`:111`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L111), [`:174-180`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L174-L180) | +| Cleanup does **not** run when setup fails — the group returns early | [`:554-556`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L554-L556) | +| A gated block is dropped during collection — no outcome, id or count | [`:449-450`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L449-L450) | +| `*` means every group the document declares; `default` means no argument was given | [`:428`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L428) onward | +| Setup runs before tests, cleanup after, whatever order the page writes them | `TestGroup` [`:200-226`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L200-L226) | +| `is_allowed_version` takes the **specifier first** | [`:45`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L45) | +| `doctest.compile` is rebound process-wide and never restored | [`:310`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L310) | The last two are where this project deliberately diverges. The argument order was a real defect in the local helper. The `compile` rebinding is unavailable to a library that loads into every pytest session, which is the entire origin of ADR 0001's decision to own the per-example loop instead. -The gated-block rule is the one semantic this project rejects on purpose: Sphinx's -drop destroys the node id, the count and the `-rs` line, and pytest users -reasonably expect a `SKIPPED` outcome with a reason. +Two more are rejected on purpose. Sphinx's gated-block drop destroys the node id, +the count and the `-rs` line, where pytest users reasonably expect a `SKIPPED` +outcome with a reason. And the setup-failure short-circuit leaves a page's +`testcleanup` unrun, which for a page that spawns a server in setup means a leak. + +**One thing Sphinx already does that is worth stating plainly:** it runs one +`DocTest` *per block* against one shared group namespace. That execution shape is +not novel to ADR 0001. What Sphinx lacks is any selectable, reportable identity +for those blocks — they all share one `DocTest.name`, which is the defect below. ## What it cannot do @@ -108,20 +130,20 @@ reasonably expect a `SKIPPED` outcome with a reason. - **Distinguish two blocks in one group.** Every block in a group shares `DocTest.name`, which is why `SphinxDocTestRunner` overrides a private stdlib method to swallow the resulting `IndexError` - ([`:257`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L257)). + ([`:257`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L257)). ## Anchors -- [`is_allowed_version`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L45) · - [`TestDirective`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L66) · - [`comment nodetype rule`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L92-L93) -- [`TestGroup`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L200) · - [`add_code`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L207) · - [`TestCode`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L235) -- [`SphinxDocTestRunner`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L257) · - [`DocTestBuilder`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L292) · - [`doctest.compile` patch](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L310) -- [`test_doc`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L428) · - [`skipped-node drop`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L443-L444) · - [`type = "exec"` for testcode](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L548) -- [User-facing contract](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/doc/usage/extensions/doctest.rst) +- [`is_allowed_version`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L45) · + [`TestDirective`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L66) · + [`comment nodetype rule`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L92-L93) +- [`TestGroup`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L200) · + [`add_code`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L207) · + [`TestCode`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L235) +- [`SphinxDocTestRunner`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L257) · + [`DocTestBuilder`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L292) · + [`doctest.compile` patch](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L310) +- [`test_doc`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L428) · + [`skipped-node drop`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L449-L450) · + [`type = "exec"` for testcode](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L548) +- [User-facing contract](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/doc/usage/extensions/doctest.rst) diff --git a/notes/analyses/16-docutils-myst.md b/notes/analyses/16-docutils-myst.md index e1fb1d3..9b524c4 100644 --- a/notes/analyses/16-docutils-myst.md +++ b/notes/analyses/16-docutils-myst.md @@ -1,6 +1,6 @@ # docutils and MyST-Parser -docutils pinned at `docutils-0.22.2` (canonical repository is on SourceForge; the +docutils pinned at `docutils-0.21.2` (canonical repository is on SourceForge; the GitHub copies are third-party mirrors, so anchors here name file and symbol rather than a permalink). MyST-Parser pinned at [`v5.1.0`](https://github.com/executablebooks/MyST-Parser/tree/v5.1.0). @@ -33,7 +33,12 @@ myst_parser.parsers.docutils_.Parser(RstParser) [v5.1.0:235] ## The two line conventions -| Front-end | Construct | `.line` reports | +**These are docutils 0.21.2 behaviours.** docutils 0.22 fixed the nested case +upstream and made top-level and nested blocks agree on the **first** line, so any +claim here that does not name a version is a bug in the claim. ADR 0005 +(`docs/adrs/0005-line-recovery-for-nested-blocks.md`) covers the floor question. + +| Front-end | Construct | `.line` reports (0.21.2) | |---|---|---| | reStructuredText | top-level `doctest_block` | its **last** line | | reStructuredText | any block nested in a directive, list item or block quote | `None`, with `.source` also `None` | @@ -122,9 +127,9 @@ losing the underlying record. [`create_myst_settings_spec`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/parsers/docutils_.py#L208) · [`settings_spec`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/parsers/docutils_.py#L241-L245) · [`MdParserConfig`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/config/main.py) -- Sphinx's registry snapshot/rebind: [`sphinx/util/docutils.py`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/util/docutils.py) · - unconditional override: [`sphinx/application.py`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/application.py) +- Sphinx's registry snapshot/rebind: [`sphinx/util/docutils.py`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/util/docutils.py) · + unconditional override: [`sphinx/application.py`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/application.py) - docutils: `docutils/parsers/rst/directives/__init__.py` (`_directives`), `docutils/parsers/rst/states.py` (`state_classes`, `doctest_block` line assignment), `docutils/utils/__init__.py` (`Reporter.attach_observer`, - `system_message`), at `docutils-0.22.2`. + `system_message`), at `docutils-0.21.2`. diff --git a/notes/analyses/17-prior-art.md b/notes/analyses/17-prior-art.md index c9da655..4933274 100644 --- a/notes/analyses/17-prior-art.md +++ b/notes/analyses/17-prior-art.md @@ -46,11 +46,16 @@ downstream test, breaking `--lf`, `--nf`, deselect files, xfail lists and CI fla history. For a *documentation* test runner, prose above examples is the thing that changes most often. -**What it cannot do.** Groups. A regex cannot see a directive's options, so Sybil -has no group concept at all and directs users to clear the namespace instead. This -is the clearest available argument for paying the docutils dependency: if -`:skipif:`, `:options:` and group names are part of the product, the document -model has to be a parse tree. +**What it does not do.** Groups. Sybil has no group concept and directs users to +clear the namespace instead — but that is a *design choice*, not a lexing limit: +its directive lexers do parse directive arguments and options. + +So the argument for paying the docutils dependency is not "a regex cannot see +`:skipif:`". It is **host fidelity**: collecting from the same doctree Sphinx +renders means a page behaves the same under `sphinx-build` and under pytest, and +this project already depends on docutils regardless. Nesting and source +attribution are not the argument either — Sybil locates a nested block exactly, +where docutils 0.21.2 reports `None`. **Verdict on the invariant.** The non-overlap check does not survive contact with docutils: two `.. include::` directives naming the same file legitimately produce diff --git a/notes/analyses/20-data-structures.md b/notes/analyses/20-data-structures.md index 3b0a730..e38c83b 100644 --- a/notes/analyses/20-data-structures.md +++ b/notes/analyses/20-data-structures.md @@ -78,8 +78,8 @@ than improving it — the three fields are then already true. ## Anchors - [`Example` / `DocTest` / `TestResults`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L114) -- [`TestCode`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L235) · - [`TestGroup`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L200) +- [`TestCode`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L235) · + [`TestGroup`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L200) - [`MultipleDoctestFailures`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L172) · [`repr_failure`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L317-L344) - [Sybil `Region`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/region.py) · diff --git a/notes/analyses/21-data-flows.md b/notes/analyses/21-data-flows.md index 0264452..ad371e8 100644 --- a/notes/analyses/21-data-flows.md +++ b/notes/analyses/21-data-flows.md @@ -108,6 +108,6 @@ group sharing state, the wrong sequence is the bug. [`compile(..., "single", ...)`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1400) - [`DoctestItem.setup`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L288-L293) · [`runtest`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L295-L303) -- [`test_doc`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L428) · - [gated-node drop](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L443-L444) · - [`doctest.compile` patch](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L310) +- [`test_doc`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L428) · + [gated-node drop](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L449-L450) · + [`doctest.compile` patch](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L310) diff --git a/notes/analyses/22-extension-seams.md b/notes/analyses/22-extension-seams.md index eb39d65..013c20c 100644 --- a/notes/analyses/22-extension-seams.md +++ b/notes/analyses/22-extension-seams.md @@ -7,7 +7,7 @@ ranking is not a matter of taste — each has an observed failure mode. | Mechanism | Where | Failure mode | |---|---|---| -| **Nominal subclassing** — the API demands the class, not the shape | stdlib `doctest`, `sphinx.ext.doctest` directives | A structurally compatible object is rejected. `DocutilsDocTestFinder` exposes a compatible `find()` and still cannot be passed to `DocTestSuite(test_finder=...)` | +| **Nominal subclassing** — the *type checker* demands the class, the interpreter does not | stdlib `doctest` (via typeshed), `sphinx.ext.doctest` directives | Typed callers are rejected for objects that work fine at runtime. stdlib performs no `isinstance` check on `parser` or `test_finder`; the pressure comes entirely from `doctest.pyi`. One genuine runtime edge: `DocTestSuite` sorts, and `DocTest.__lt__` returns `NotImplemented` for a non-`DocTest`, so a custom finder must return real `DocTest`s | | **Callable aliases** — `Evaluator = Callable[[Example], str \| None]` | Sybil | Types nothing. You cannot express "this lexer emits `source` and `arguments`", so a mismatched pairing raises `KeyError` at run time. Sybil's own source comments say the payload "could likely be a `TypedDict`" | | **Named registry** — a module dict plus a `register_*` function | `doctest.register_optionflag`, docutils directives, xdoctest's two facades | Process-global mutable state. Order-dependent, unscoped, and silently overwritable | | **`Protocol`** — structural typing | xdoctest's `StdlibExampleLike` | None inherent; but a `Protocol` alone does not satisfy a nominal consumer | @@ -49,9 +49,14 @@ direction and is retiring `set_event_loop_policy` in favour of an explicit ## The nominal/structural trap, and the cheap way out Typeshed's signatures demand classes: `DocTestFinder.__init__(parser: DocTestParser -= ...)`, `DocTestSuite(test_finder: DocTestFinder | None)`. A `Protocol` gives a -third party structural typing; a nominal subclass keeps you passable to the -stdlib's own builders. += ...)`, `DocTestSuite(test_finder: DocTestFinder | None)`. The interpreter does +not — a duck-typed parser or finder runs fine. So a `Protocol` gives a third party +structural typing, and a nominal subclass keeps you *type-checkable* against the +stub. + +The subclass does not buy passability. That is a matter of matching the call +signature: a finder whose `find()` takes a string first cannot be handed to +`DocTestSuite`, which passes a module — and subclassing does not change that. Doing both costs nothing: diff --git a/notes/analyses/23-namespace-scope-and-test-identity.md b/notes/analyses/23-namespace-scope-and-test-identity.md index 524fbd3..70bcb75 100644 --- a/notes/analyses/23-namespace-scope-and-test-identity.md +++ b/notes/analyses/23-namespace-scope-and-test-identity.md @@ -19,8 +19,14 @@ narrative page needs depends on the first. | | one node id | N node ids | |---|---|---| -| **one `DocTest`** | `sphinx.ext.doctest`; `doctest_docutils` `merged` | incoherent | -| **N `DocTest`s** | *unoccupied until ADR 0001* | Sybil; `doctest_docutils` `per-block` | +| **one `DocTest`** | PR #87's `merged` | incoherent | +| **N `DocTest`s** | `sphinx.ext.doctest`, but with *no* ids at all; ADR 0001 gives the shape a pytest identity | Sybil; PR #87's `per-block` | + +Sphinx belongs in the bottom row: it already builds one `DocTest` per block +against one shared group namespace. Its "one node id" is really *no* id — every +block in a group shares one `DocTest.name`, which is why `SphinxDocTestRunner` +overrides a private stdlib method to swallow the resulting `IndexError`. So the +execution shape is well-trodden; the contribution is making it addressable. The bottom-right cell is where the silent failure lives, and two shipped projects are in it. diff --git a/notes/analyses/90-bibliography.md b/notes/analyses/90-bibliography.md index d627b6f..5f6daed 100644 --- a/notes/analyses/90-bibliography.md +++ b/notes/analyses/90-bibliography.md @@ -20,7 +20,7 @@ used anywhere else. | `DocTest.__lt__` | [`:596`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L596) | compares names as text | | `DocTestParser` | [`:609`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L609) | the injectable parser | | `_EXAMPLE_RE` | [`:618`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L618) | private, used for prompt sniffing | -| `DocTestFinder` | [`:844`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L844) | nominal type demanded by `DocTestSuite` | +| `DocTestFinder` | [`:844`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L844) | the type typeshed names; accepted structurally at runtime | | `report_*` hooks | [`:1286-1314`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1286-L1314) | the four supported in-loop seams; no `report_skip` here | | `__run` | [`:1344`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1344) | name-mangled loop; overridable by mechanism | | `compile(..., "single", ...)` | [`:1400`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1400) | the hard-coded mode `{testcode}` cannot use | @@ -31,8 +31,8 @@ used anywhere else. | `OutputChecker` | [`:1690`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1690) | the documented checker seam | | `DebugRunner` | [`:1874`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1874) | `report_*` overriding as the sanctioned loop control | | `testfile` | [`:2091`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L2091) | the API `testdocutils` mirrors | -| `DocTestSuite` | [`:2467`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L2467) | demands a nominal `DocTestFinder` | -| `DocFileSuite` | [`:2570`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L2570) | demands a nominal `DocTestParser` | +| `DocTestSuite` | [`:2467`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L2467) | no `isinstance` on `test_finder`; sorts, so results must be real `DocTest`s | +| `DocFileSuite` | [`:2570`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L2570) | no `isinstance` on `parser` | Documentation: [`Doc/library/doctest.rst`](https://github.com/python/cpython/blob/v3.14.2/Doc/library/doctest.rst). @@ -69,7 +69,8 @@ Documentation: [`Doc/library/doctest.rst`](https://github.com/python/cpython/blo | `_get_continue_on_failure` | [`:410`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L410) | private helper imported today | | `DoctestTextfile` | [`:420-421`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L420-L421) | `obj = None` as a class attribute | | `_check_all_skipped` | [`:451`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L451) | fires only once the item is running | -| `DoctestModule` / `parsefactories` | [`:500`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L500) · [`:556`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L556) | autouse fixtures from conftest | +| `DoctestModule` / `parsefactories` | [`:500`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L500) · [`:556`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L556) | fixtures defined in the collected `.py`; **not** conftest autouse, which arrives via `FixtureManager.pytest_plugin_registered` | +| `subtests` | [`src/_pytest/subtests.py`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/subtests.py) | builtin since 9.0; the only sanctioned sub-item outcome mechanism, and experimental | | `_get_checker` | [`:662`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L662) | the checker that would have to be reimplemented | | `_get_report_choice` | [`:703`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L703) | private helper | | `doctest_namespace` | [`:721`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L721) | the fixture that survives plugin blocking today | @@ -103,27 +104,27 @@ Documentation: [`Doc/library/doctest.rst`](https://github.com/python/cpython/blo ## Sphinx — `v9.1.0` -[`sphinx-doc/sphinx @ v9.1.0`](https://github.com/sphinx-doc/sphinx/tree/v9.1.0) · -[`sphinx/ext/doctest.py`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py) +[`sphinx-doc/sphinx @ v9.1.0`](https://github.com/sphinx-doc/sphinx/tree/v8.2.3) · +[`sphinx/ext/doctest.py`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py) | Symbol | Anchor | Cited for | |---|---|---| -| `is_allowed_version(spec, version)` | [`:45`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L45) | specifier first — the reverse of the local helper | -| `TestDirective` | [`:66`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L66) | the directive base and its option handling | -| comment nodetype rule | [`:92-93`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L92-L93) | `testsetup`/`testcleanup`/`:hide:` become `nodes.comment` | -| `:options:` gating | [`:111`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L111) | accepted only on `doctest` and `testoutput` | -| `TestGroup` / `add_code` | [`:200`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L200) · [`:207`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L207) | phase ordering; three silent-loss cases | -| `TestCode` | [`:235`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L235) | the parsed unit | -| `SphinxDocTestRunner` | [`:257`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L257) | overrides a private method to swallow an `IndexError` | -| `DocTestBuilder` | [`:292`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L292) | builder coupling | -| `doctest.compile` rebinding | [`:310`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L310) | process-global, never restored | -| `test_doc` | [`:428`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L428) | group resolution and `*` | -| gated-node drop | [`:443-444`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L443-L444) | no outcome, id or count | -| `type = "exec"` for testcode | [`:548`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/ext/doctest.py#L548) | the mode flip | - -Documentation: [`doc/usage/extensions/doctest.rst`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/doc/usage/extensions/doctest.rst). -Registry behaviour: [`sphinx/util/docutils.py`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/util/docutils.py), -[`sphinx/application.py`](https://github.com/sphinx-doc/sphinx/blob/v9.1.0/sphinx/application.py). +| `is_allowed_version(spec, version)` | [`:45`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L45) | specifier first — the reverse of the local helper | +| `TestDirective` | [`:66`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L66) | the directive base and its option handling | +| comment nodetype rule | [`:92-93`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L92-L93) | `testsetup`/`testcleanup`/`:hide:` become `nodes.comment` | +| `:options:` gating | [`:111`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L111) | accepted only on `doctest` and `testoutput` | +| `TestGroup` / `add_code` | [`:200`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L200) · [`:207`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L207) | phase ordering; three silent-loss cases | +| `TestCode` | [`:235`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L235) | the parsed unit | +| `SphinxDocTestRunner` | [`:257`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L257) | overrides a private method to swallow an `IndexError` | +| `DocTestBuilder` | [`:292`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L292) | builder coupling | +| `doctest.compile` rebinding | [`:310`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L310) | process-global, never restored | +| `test_doc` | [`:428`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L428) | group resolution and `*` | +| gated-node drop | [`:443-444`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L449-L450) | no outcome, id or count | +| `type = "exec"` for testcode | [`:548`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L548) | the mode flip | + +Documentation: [`doc/usage/extensions/doctest.rst`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/doc/usage/extensions/doctest.rst). +Registry behaviour: [`sphinx/util/docutils.py`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/util/docutils.py), +[`sphinx/application.py`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/application.py). ## MyST-Parser — `v5.1.0` @@ -136,7 +137,7 @@ Registry behaviour: [`sphinx/util/docutils.py`](https://github.com/sphinx-doc/sp | `settings_spec` | [`parsers/docutils_.py:241-245`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/parsers/docutils_.py#L241-L245) | | `MdParserConfig` (`myst_enable_extensions`, `myst_fence_as_directive`) | [`config/main.py`](https://github.com/executablebooks/MyST-Parser/blob/v5.1.0/myst_parser/config/main.py) | -## docutils — `docutils-0.22.2` +## docutils — `docutils-0.21.2` (the version this project pins) The canonical repository is on [SourceForge](https://sourceforge.net/p/docutils/code/); the GitHub copies are diff --git a/notes/analyses/README.md b/notes/analyses/README.md index 4114e78..74df866 100644 --- a/notes/analyses/README.md +++ b/notes/analyses/README.md @@ -31,12 +31,12 @@ What exactly does each of them require, and where do they contradict each other? | pytest | `pytest-dev/pytest` | `9.1.1` | | pytest-xdist | `pytest-dev/pytest-xdist` | `v3.8.0` | | pytest-asyncio | `pytest-dev/pytest-asyncio` | `v1.4.0` | -| Sphinx | `sphinx-doc/sphinx` | `v9.1.0` | -| MyST-Parser | `executablebooks/MyST-Parser` | `v5.1.0` | +| Sphinx | `sphinx-doc/sphinx` | `v8.2.3` — what this project resolves. `v9.1.0` is cited only where a difference is called out | +| MyST-Parser | `executablebooks/MyST-Parser` | `v5.1.0` on Python ≥ 3.11; `v4.0.1` below | | Sybil | `simplistix/sybil` | `10.0.1` | | xdoctest | `Erotemic/xdoctest` | `v1.3.2` | | typeshed | `python/typeshed` | `8c7256c` (no tags; commit reachable from trunk) | -| docutils | SourceForge (the GitHub clones are third-party mirrors) | `docutils-0.22.2` | +| docutils | SourceForge (the GitHub clones are third-party mirrors) | `docutils-0.21.2` | ## Files From 1f45fdb55e66ace90231fb0cb15d6892c56fcd26 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 13:28:51 -0500 Subject: [PATCH 10/31] docs(adrs[0001]): Repin the gated-drop anchor to v8.2.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit why: The skipif-drop line moved between Sphinx tags, and the vocabulary table still pointed at the v9.1.0 offset under a v8.2.3 path — an anchor that resolves onto the wrong lines. --- docs/adrs/0001-typed-vanilla-doctest-core.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/adrs/0001-typed-vanilla-doctest-core.md b/docs/adrs/0001-typed-vanilla-doctest-core.md index 6feb84f..a28a10b 100644 --- a/docs/adrs/0001-typed-vanilla-doctest-core.md +++ b/docs/adrs/0001-typed-vanilla-doctest-core.md @@ -291,7 +291,7 @@ referents. Each term below is decided once and used only that way. | group | — | `xdist_group` is a *scheduling* affinity marker ([`remote.py:245-254`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/remote.py#L245-L254)) | the author-facing bucket: `.. doctest:: intro`, `default`, `*` | Adopt `group` for the sharing unit. The xdist affinity key is *derived*, never the group name | | scope | — | the fixture-lifetime ladder | — | Reserved for pytest. The block-vs-document axis becomes `share` | | test / item / block | `DocTest`, `Example` | `Item` | [`TestCode`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L235) is the parsed unit | Three nouns: `Example` (stdlib), `Block` (parsed), `DocTest` (runnable) | -| skip | the `SKIP` flag, short-circuiting before `report_start` | a reported outcome with a reason, **at item granularity** | [drops the node entirely](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L443-L444) | doctest's mechanism (set `SKIP`, never drop the node); pytest's outcome where the granularity allows it — see [](#the-outcome-contract). Sphinx's drop is deliberately rejected | +| skip | the `SKIP` flag, short-circuiting before `report_start` | a reported outcome with a reason, **at item granularity** | [drops the node entirely](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L449-L450) | doctest's mechanism (set `SKIP`, never drop the node); pytest's outcome where the granularity allows it — see [](#the-outcome-contract). Sphinx's drop is deliberately rejected | | directive | inline `# doctest: +FLAG` | — | a docutils directive with an `option_spec` | Reserved for the docutils meaning. doctest's form is "inline flags" | | optionflags | an int bitmask; `register_optionflag` | `doctest_optionflags` ini | `:options:` plus `doctest_default_flags` | Keep verbatim. `register_optionflag` is the one genuinely cross-library extension point | | setup / cleanup | `setUp`/`tearDown` on the suite builders | fixtures | `testsetup`/`testcleanup` directives | Author-facing names stay Sphinx's; `phase` is the internal ordering axis; a fixture is never "setup" | From 32ae5497338725596773e0c7835d806cbebf5b13 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 14:56:15 -0500 Subject: [PATCH 11/31] docs(adrs[0001]): Fix the baseline, the Example model and purity MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit why: A second review round found the corrected baseline was still wrong, and three model claims did not survive execution. Trunk's _find appends one DocTest per matched node named page.md[k] and the collector yields one item per test, so released gp-libs already has per-block identity with isolated copied globals — it lacks only a sharing unit. Example.__eq__ gates on exact type identity, so a bare subclass compares unequal to a stock Example in both directions. what: - Restate the baseline from trunk's source: per-block identity ships today, and it is what this design preserves rather than invents - Restate the invariant as scheduling identity versus diagnostic identity, which is what the decoupling actually is - Keep the Example subclass but override __eq__ with an isinstance check and rebind __hash__, pinned by a doctest asserting symmetry both ways; record the stock-Example fallback - Correct Example.source: stdlib-normalized executable body, not the author's verbatim text, which is what Block.source holds - Drop the purity claim. Includes read transitive files, directives execute during parsing, the registry is process-global and MyST plugins alter the tree; the contract is determinism over source closure, normalized settings and a frozen registry - Split GroupPlan from GroupRun so planning stays immutable and the live mapping belongs to an attempt - Add BlockResult and GroupResult as the channel for partial-skip detail, with its visibility and its accepted loss stated - Record that a paired want depends on a run-time gate, that wildcard membership needs a distinct DocTest per group, and that the gate's namespace diverges from Sphinx's fresh context - Complete the item lifecycle: seed restore, cleanup failure precedence, and the runner behaviours that must be reimplemented --- docs/adrs/0001-typed-vanilla-doctest-core.md | 184 +++++++++++++++---- 1 file changed, 151 insertions(+), 33 deletions(-) diff --git a/docs/adrs/0001-typed-vanilla-doctest-core.md b/docs/adrs/0001-typed-vanilla-doctest-core.md index a28a10b..c306597 100644 --- a/docs/adrs/0001-typed-vanilla-doctest-core.md +++ b/docs/adrs/0001-typed-vanilla-doctest-core.md @@ -13,12 +13,18 @@ Date: 2026-08-02 over a docutils or MyST doctree and keeps CPython's *running* half. `pytest_doctest_docutils` wraps that in a pytest plugin. -Two facts about the shipped code set the problem. - -**A page is one test.** `DocutilsDocTestFinder` produces a single -{class}`doctest.DocTest` per document. A page cannot build state across the prose -that explains it, because there is no unit smaller than the page and no unit that -groups blocks within one. +**Released gp-libs already has per-block identity, and no sharing unit at all.** +`DocutilsDocTestFinder._find` walks the doctree and appends one +{class}`doctest.DocTest` per matched node, named `page.md[k]` where `k` is the +document-order index. The collector yields one {class}`pytest.DoctestItem` per +test. Each test is built by handing `globs` to +{meth}`doctest.DocTestParser.get_doctest`, and `DocTest.__init__` **copies** the +mapping — so every block runs against its own isolated namespace. + +That is the real starting point, and it frames the problem precisely: the +granularity this design wants to *preserve* is already shipped. What is missing +is any unit coarser than a block — no groups, no phases, no way for a narrative +page to build state across the prose that explains it. **The plugin blocks the plugin whose internals it imports.** `pytest_configure` calls `config.pluginmanager.set_blocked("doctest")`, and the @@ -81,7 +87,12 @@ Sybil ships the second shape without acknowledging it (see [](#prior-art)). ## Decision -**The pytest item is the group. The {class}`doctest.DocTest` is the block.** +**One pytest item owns one shared-state group; inside it, each source block +remains a real, independent {class}`doctest.DocTest`.** + +The decoupling is not "group versus block". It is **scheduling identity versus +diagnostic identity**: pytest schedules the group, while each `DocTest` keeps its +own source location, examples and failure gutter. One {class}`pytest.Item` per (document, group). Inside it, a tuple of per-block `DocTest`s run in phase order against one live `globs` mapping that never leaves @@ -211,6 +222,17 @@ This design takes the same position: **skip the item when every runnable block i skipped; otherwise report partial skips as typed block detail, not as a pytest outcome.** No extra reports are synthesized. +"Typed block detail" needs a channel, or implementers will re-invent skip lifting +or write to stderr. The channel is a `GroupResult` — one `BlockResult` per block, +each carrying phase, outcome, gate reason and location — attached to the item and +rendered in two places: the failure longrepr when the group fails, and a +terminal-summary line at `-rs`. It is **not** visible at default verbosity, and +it never becomes a JUnit `` entry. + +That is a real product loss relative to lifting a gated block into its own item, +and it is accepted deliberately: **a gated block inside a mixed group gives up its +selectable skip row.** The information survives; the addressable unit does not. + `subtests` — a builtin pytest plugin since 9.0, exporting `pytest.Subtests` and `pytest.SubtestReport` — can emit per-block outcomes, and is the only sanctioned mechanism that can. It is not adopted here: pytest documents it as experimental, @@ -261,18 +283,26 @@ an implementer cannot get it wrong by omission: 1. **Collection** builds one `GroupPlan` per (document, group) and one item per plan. An empty plan yields no item. -2. **`setup()`** clears the group mapping in place, then calls `super().setup()` - so fixtures inject into that same object. Clearing in place rather than - rebinding is what keeps `item.globs is plan.globs` true for every block, and - what stops attempt two of a `--reruns` run from reading attempt one's - mutations. +2. **`setup()`** starts an attempt. In order: clear the live mapping **in place**; + restore the plan's `seed`, `extraglobs` and `__name__`; then call + `super().setup()` so fixtures inject into that same object. Clearing in place + rather than rebinding is what keeps `item.globs is run.globs` true for every + block, and what stops attempt two of a `--reruns` run from reading attempt + one's mutations. Every block's `DocTest.globs` is assigned this object *after* + construction, because `DocTest.__init__` copies. 3. **`runtest()`** is overridden. It must not delegate to `DoctestItem.runtest`, which runs a single `dtest` with `clear_globs` defaulting to `True` — that would empty the shared mapping after the first block. It calls `run_group()`, which runs each `PlannedBlock` in phase order - with `clear_globs=False`, evaluates `:skipif:` against the group mapping, and - wraps the body in a `try`/`finally` so cleanup runs whether or not the body - raised. + with `clear_globs=False`, evaluates `:skipif:` against the live mapping, + finalizes each paired `want` from its gated `ExpectedOutput`, and wraps the + body in a `try`/`finally` so cleanup runs whether or not the body raised. When + cleanup *also* fails, the body's failure is the one raised; cleanup's is + recorded in the `GroupResult`. + + The `OutcomeException` re-raise, the `bdb.BdbQuit` → `outcomes.exit` + conversion and `continue_on_failure` are reimplemented here, because + `PytestDoctestRunner` is nested inside a factory and cannot be imported. 4. **Outcome** follows [](#the-outcome-contract): skip the item only when every runnable block is skipped. 5. **`repr_failure`** is inherited unchanged. It already reads each failure's own @@ -335,8 +365,27 @@ class PlannedBlock(t.NamedTuple): class GroupPlan(t.NamedTuple): group: str - blocks: tuple[PlannedBlock, ...] # in phase order - globs: dict[str, t.Any] # the live mapping, shared by every block above + blocks: tuple[PlannedBlock, ...] # in phase order; IMMUTABLE + seed: t.Mapping[str, t.Any] # initial namespace; copied per attempt + + +class GroupRun: + """One execution attempt. Owns the live mapping; a plan never does.""" + + plan: GroupPlan + globs: dict[str, t.Any] # cleared and reseeded per attempt + + +class BlockResult(t.NamedTuple): + block: PlannedBlock + outcome: t.Literal["passed", "failed", "skipped"] + reason: str | None # the gate expression, when skipped + + +class GroupResult(t.NamedTuple): + group: str + blocks: tuple[BlockResult, ...] + failures: tuple[doctest.DocTestFailure | doctest.UnexpectedException, ...] ``` `Block.line` being nullable is load-bearing, not defensive. A bare `>>>` block @@ -351,6 +400,25 @@ from a bare tuple of `DocTest`s. A `DocTest` carries no phase and no gate, so th plan has to. Tagging each entry also makes the ordering self-describing rather than a convention a comment asserts. +**A `PlannedBlock.test` cannot always be fully preconstructed.** Sphinx accepts +`:skipif:` on a `testoutput`, and when that output is gated away its `testcode` +still runs — expecting *empty* output. So the `want` of a paired block is a +function of a gate evaluated at run time, and the plan must carry the paired +output as data (`ExpectedOutput`, itself gated) with the `DocTest` finalized in +`run_group()`. Freezing `want` at projection time silently runs the wrong +assertion. + +**A wildcard block needs a distinct `DocTest` per group it joins.** `DocTest.globs` +is a plain mutable attribute, so one object shared across two `GroupPlan`s would +have the second group's assignment win and both groups would execute against one +mapping. `*` membership therefore materializes a separate `DocTest` per group. + +**The gate's evaluation namespace is a deliberate divergence.** Sphinx evaluates +each `:skipif:` in a fresh context seeded with `doctest_global_setup`; this design +evaluates it against the live group mapping, after fixture injection and after +earlier blocks have run. That is more useful — a gate can consult a fixture — and +it is not what `sphinx-build` does. Recorded rather than hidden. + `ExecutionProfile` is a **private** protocol carrying compile mode, extra compile flags, and an optional per-group context manager. It is not a `Literal["single", "exec"]`, because a second execution policy already exists in this repository: @@ -450,8 +518,9 @@ that owns the loop must probe for it rather than assume it. Making the item the sharing unit removes the need to satisfy the second and third at all. **The first still binds at any granularity** — identical collection is -required whether a page yields one item or fifty — and is satisfied instead by -collection being a pure function of (bytes, argv, ini). The fourth is why a +required whether a page yields one item or fifty — and is approached instead +through determinism over source closure, normalized settings and a frozen +registry, not through a purity claim collection cannot make. The fourth is why a worker-count fork is not worth carrying: `pytest_xdist_setupnodes(config, specs)` hands over the already-expanded spec list and never raises. @@ -482,15 +551,50 @@ Each is a genuine conflict where satisfying one goal costs another. "Both" is no an answer; the position taken and its price are recorded. **A vanilla `DocTest` cannot carry the front-end's metadata.** It has exactly -`(examples, globs, name, filename, lineno, docstring)`. *Position:* metadata that -the runner needs rides on a nominal {class}`doctest.Example` subclass — -`doctest.Example` has no `__slots__`, so attributes survive -{func}`copy.copy`, {mod}`pickle`, and a third party's naive +`(examples, globs, name, filename, lineno, docstring)`. *Position:* metadata the +runner needs at run time rides on a {class}`doctest.Example` subclass — +`doctest.Example` has no `__slots__`, so attributes survive {func}`copy.copy`, +{mod}`pickle`, and a third party's naive `DocTest(examples, globs, name, filename, lineno, docstring)` rebuild, because that rebuild reuses the same `Example` objects. Metadata the runner does *not* need — groups, wildcards, pairing — never touches a stdlib object and dies in the -projection layer. *Price:* one subclass to explain, and a rule that nothing may -smuggle through `DocTest.name`. +projection layer. + +*Price:* the subclass must restore equality, because +{meth}`doctest.Example.__eq__` gates on **exact type identity** +([`Lib/doctest.py:518`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L518)) — +a bare subclass compares unequal to a stock `Example` with identical fields, in +both directions, while hashing the same. So the subclass overrides `__eq__` with +an {func}`isinstance` check and re-binds `__hash__` explicitly; Python's +reflected-operand rule then makes equality symmetric again: + +```{doctest} +>>> import doctest +>>> class Tagged(doctest.Example): +... def __eq__(self, other): +... if not isinstance(other, doctest.Example): +... return NotImplemented +... return (self.source, self.want, self.lineno, self.indent, +... self.options, self.exc_msg) == ( +... other.source, other.want, other.lineno, other.indent, +... other.options, other.exc_msg) +... __hash__ = doctest.Example.__hash__ +>>> plain, tagged = doctest.Example("1\n", "1\n"), Tagged("1\n", "1\n") +>>> tagged == plain, plain == tagged, tagged in [plain] +(True, True, True) + +Without the override, a bare subclass is unequal both ways: + +>>> class Bare(doctest.Example): pass +>>> bare = Bare("1\n", "1\n") +>>> bare == plain, plain == bare +(False, False) +``` + +The alternative — setting the attribute on a *stock* `Example` with no subclass +at all — is equality-safe by construction and equally durable. It is rejected +only because it forfeits the typed surface; if the subclass ever proves +troublesome, that is the fallback. **Node-id granularity versus shared state.** *Position:* decouple them — N `DocTest`s under one node id. *Price:* selecting a group runs all its blocks; @@ -504,9 +608,20 @@ block gains an item relative to `sphinx-build -b doctest`, and `--collect-only` must not evaluate the gate, which is why `:skipif:` is carried through collection unevaluated and run in `runtest()`. -**Collection purity versus `--collect-only` fidelity.** *Position:* collection -evaluates no user Python, so it is a pure function of (bytes, argv, ini). *Price:* -`--collect-only` no longer shows which blocks will skip. +**Collection determinism versus `--collect-only` fidelity.** *Position:* +collection evaluates no *author-supplied* Python — `:skipif:` is carried through +unevaluated and run in `runtest()`. *Price:* `--collect-only` no longer shows +which blocks will skip. + +Collection is **not** a pure function of (bytes, argv, ini), and claiming so +would be wrong on four counts: `.. include::` reads transitive files, docutils +directive implementations execute during parsing, the directive registry is +process-global, and MyST plugins change the tree. The defensible contract is +determinism over **complete source closure + normalized settings + frozen +registry**. Deferring the author's gate removes the largest divergence risk; it +does not make xdist divergence structurally impossible, and the +{doc}`registry freeze <0006-pytest-private-api-compatibility>` is what closes the +rest. **Sphinx compatibility versus silent-loss behaviours.** Sphinx silently discards an orphan `testoutput`, silently discards a `testoutput` following a `doctest` @@ -640,8 +755,8 @@ conformance test in CI. fabricated line, and does not affect its siblings. - Every `--dist` mode works, because there is no shared state to split. - No CPython code-object clone, and no process-global rebinding of anything. -- Collection is a pure function of (bytes, argv, ini), so worker divergence is - structurally impossible and `--collect-only` runs no user code. +- Collection runs no author-supplied Python, so `--collect-only` has no side + effects and the largest source of worker divergence is removed. - Grouping is one pure function with no docutils, pytest or filesystem dependency, and is testable without any of them. - A new block kind is a registration, not an edit to a method branching on string @@ -696,9 +811,12 @@ path), {doc}`0004-diagnostics-as-data` (what is reported and what is suppressed) ## Final position The core produces real {class}`doctest.DocTest` objects holding real -{class}`doctest.Example` objects, and `Example.source` is the author's verbatim -text. Everything else — groups, phases, pairing, diagnostics, distribution — is a -layer above that fact, and no layer reaches around another. +{class}`doctest.Example` objects. `Example.source` is the stdlib-normalized +executable body — prompts and indentation stripped, trailing newline added, the +stripped column recorded in `Example.indent` — not a synthesized wrapper and not +the author's verbatim text, which is what `Block.source` holds. Everything else — +groups, phases, pairing, diagnostics, distribution — is a layer above that fact, +and no layer reaches around another. The unit that shares a `globs` mapping is the unit pytest schedules. That is the one invariant every other property in this document follows from, and it is not From 02efb1f4d91ecc2868ba1de524b66059ebe96062 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 14:58:59 -0500 Subject: [PATCH 12/31] docs(adrs[0002]): Correct the SyntaxError claim and scope the matrix MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit why: A stock DocTestRunner does not raise on a multi-statement body — the compile call sits inside the loop's own try, so it records an unexpected exception and returns failed=1. Only DebugRunner propagates. The harness also needs report_* events, since a skip that wrongly executes moves neither counter. what: - State that stock records rather than raises, and assert the exec-mode case as a pair: stock fails, this runner passes - Add report_* hook events and repeated runs to the matrix - Exclude cross-block FAIL_FAST and cleanup aggregation, which belong to run_group's lifecycle tests rather than the per-example loop --- .../0002-runner-conformance-across-cpython.md | 29 ++++++++++++++++--- 1 file changed, 25 insertions(+), 4 deletions(-) diff --git a/docs/adrs/0002-runner-conformance-across-cpython.md b/docs/adrs/0002-runner-conformance-across-cpython.md index f7183d9..90bc45c 100644 --- a/docs/adrs/0002-runner-conformance-across-cpython.md +++ b/docs/adrs/0002-runner-conformance-across-cpython.md @@ -62,10 +62,31 @@ incremented *before* the `SKIP` check, so a skip that wrongly executes moves neither counter — it is invisible to both the tuple and to `summarize()` at zero failures, and only the `report_*` text distinguishes it. -The exec-mode case has no stock counterpart to compare against: a stock runner -raises `SyntaxError: multiple statements found` on a multi-statement body. It is -asserted against a recorded expectation instead, and that asymmetry is the point -— it is the one behaviour this runner exists to add. +The exec-mode case is the one the two runners are *meant* to disagree on, and it +still compares against stock. `compile()` raises on a multi-statement body, but +that call sits inside the loop's own `try` +([`Lib/doctest.py:1398-1408`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1398-L1408)), +so a stock {class}`doctest.DocTestRunner` catches the `SyntaxError` and records +it as an unexpected exception rather than propagating it: one +`report_unexpected_exception` call, `TestResults(failed=1, attempted=1)`, and +`_stats` at `(1, 1, 0)`. Only {class}`doctest.DebugRunner` — and pytest's runner +beneath it — converts that into a raise, as +{exc}`doctest.UnexpectedException`. So the case is asserted as a pair: stock +records the failure, this runner records a pass. A regression that silently +reverts to `"single"` mode shows up as the two converging. + +**What else belongs in the matrix, and what does not.** Add `report_*` hook +events — the only channel that distinguishes a skip which wrongly *executed*, +since `attempted` increments before the `SKIP` check and neither counter moves — +and repeated runs of one test, which exercise accumulator arithmetic across +calls. + +Cross-block `FAIL_FAST` and cleanup aggregation stay out. Both are properties of +`run_group()` rather than of the per-example loop, so a stock runner offers +nothing to compare them against; they belong to +{doc}`0001-typed-vanilla-doctest-core`'s item-lifecycle tests. A +{exc}`pytest.skip` raised inside an example and a debugger exit are likewise +pytest-layer concerns, testable only through a pytest session. Version handling is by capability probe, never by version comparison, so a backport, a vendored interpreter or a fork behaves correctly rather than by From 9eb69cb7e7d4123e928fdf83c1f90d29eeda02ea Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 14:58:59 -0500 Subject: [PATCH 13/31] docs(adrs[0004]): Replace the mechanism that does not exist MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit why: Both mechanism assumptions were wrong. A docutils system_message carries a level and text and nothing semantically stable, so keying suppression on a stable code is impossible for exactly the messages this record wants to suppress. And attaching an observer is additive — the message still reaches the warning stream. what: - Say codes exist only for diagnostics this project emits, and that docutils-originated messages must be classified instead - Give the full three-part recipe: halt_level above 4, report_level 5 or warning_stream disabled, then the observer - Open the classifier question, with the two candidate answers and the two docutils dialects a text table would have to handle - Replace the promotion rule: registered-name is inverted for a typo and never fires for a swallowed foreign container. Promote on body content matching the example regex, with near-miss as an additive rule, and record that this is reST-only - Mark the direction not yet implementable, rather than implying it is --- docs/adrs/0004-diagnostics-as-data.md | 47 ++++++++++++++++++++++----- 1 file changed, 39 insertions(+), 8 deletions(-) diff --git a/docs/adrs/0004-diagnostics-as-data.md b/docs/adrs/0004-diagnostics-as-data.md index 0eb82a1..aceb8cb 100644 --- a/docs/adrs/0004-diagnostics-as-data.md +++ b/docs/adrs/0004-diagnostics-as-data.md @@ -19,9 +19,24 @@ tests and the session exits green. A page that checks nothing reports the same way as a page that passes. {doc}`0001-typed-vanilla-doctest-core` gives the front-end layer a second return -value for this: `Diagnostic(level, code, message, path, line)`, produced by -setting `halt_level` past the abort threshold and attaching a reporter observer. -Suppression and promotion key on the stable `code`, never on message text. +value for this: `Diagnostic(level, code, message, path, line)`. + +**Two mechanism assumptions in the first draft were wrong, and the fix is not +cosmetic.** + +*There is no stable code to key on.* A docutils `system_message` carries a level +and text, and nothing semantically stable. So codes exist only for diagnostics +**this project emits**; docutils-originated messages arrive code-less and have to +be *classified* before they can be suppressed or promoted. The classifier is the +open question below, and it cannot be "key on the code", because for these +messages there is none. + +*An observer does not silence the stream.* Attaching one is additive: the message +still reaches the warning stream. Turning reporter output into values needs three +settings together — `halt_level` above 4 (both to avoid the mid-parse abort and +because a halting message bypasses observer notification entirely), +`report_level` at 5 or `warning_stream` disabled to stop the write, and then the +observer. ## Question @@ -40,8 +55,10 @@ surface. ## Direction -Suppress by code, narrowly, and only for the two classes a bare-docutils parse -cannot judge: unknown roles and unknown directives. +Suppress narrowly, and only for the two classes a bare-docutils parse cannot +judge: unknown roles and unknown directives. "By code" is the intent; the +classifier that assigns a code to a docutils message is unsettled, so this +direction is not yet implementable. Every diagnostic raised by this project's own layers defaults to visible, and `level="error"` from those layers fails collection with the file and line named. @@ -56,8 +73,22 @@ global on/off switch. - Whether diagnostics surface as {class}`pytest.PytestWarning` subclasses, giving `-W error::` control for free, or as a dedicated report section. -- Whether an unknown-directive diagnostic should be promoted when the directive - name is one this project registers — that case is not Sphinx supplying it, it - is a registration that did not happen, which is the GH-48 failure mode. +- **What classifies a code-less docutils message.** The options are an owned, + version-pinned message-text table with a test that fails on upstream rewording + (and which must handle two dialects — reST's `Unknown directive type "x".` at + ERROR/3 versus MyST's `Unknown directive type: 'x'` at WARNING/2), or + pre-empting at the source by overriding the directive-dispatch path so the + unknown case never becomes a reporter message at all. This is the decision + ADR 0004 cannot ship without. +- **What promotes an unknown-directive message back to visible.** The first + draft proposed "when the name is one this project registers", but that is + inverted for the typo case — a misspelled `.. doctset::` is precisely *not* a + registered name — and never fires for a foreign container whose body was + swallowed unparsed. The rule that covers both is **body content**: promote when + the swallowed body matches `DocTestParser._EXAMPLE_RE`. Near-miss-to-a-registered + -name is a useful additional rule for the typo case, where the body check does + not help. +- The body-content rule is reST-only as stated, because myst-parser discards the + fence body. The Markdown equivalent is unsolved. - Whether the CLI (`python -m doctest_docutils`) and the pytest plugin share one formatter or two. From cba2ec0d81f2c55145ee6c8b88d3a0c51ad2c5f9 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 14:58:59 -0500 Subject: [PATCH 14/31] docs(adrs[0005,0006]): Settle the matrices these records depend on MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit why: 0005 said docutils >=0.22 requires Sphinx >=9.1. Sphinx 9.0 already permits it — but requires Python 3.11, and gp-sphinx's sphinx<9 cap makes the requirement unsatisfiable regardless. 0006 proposed wrapper=True, which is gated on pluggy >=1.2 rather than pytest 8, and pytest 7 permits a pluggy that raises at plugin import. what: - 0005: state the move as three parts, with the gp-sphinx cap named as the binding constraint and upstream of this repo; add the resolved matrix per interpreter; target >=0.22,<0.23 rather than an open-ended floor; correct "pins" to "resolves"; make dropping Python 3.10 the blocking open question - 0006: use old-style hookwrapper with force_result, which needs no floor and was verified on pytest 7 and 9; name the minimum supported pytest; record that DoctestItem is public and the filtered collector is not --- .../0005-line-recovery-for-nested-blocks.md | 51 +++++++++++++++---- .../0006-pytest-private-api-compatibility.md | 29 ++++++++--- 2 files changed, 62 insertions(+), 18 deletions(-) diff --git a/docs/adrs/0005-line-recovery-for-nested-blocks.md b/docs/adrs/0005-line-recovery-for-nested-blocks.md index 79325b1..99cd1f8 100644 --- a/docs/adrs/0005-line-recovery-for-nested-blocks.md +++ b/docs/adrs/0005-line-recovery-for-nested-blocks.md @@ -10,7 +10,8 @@ Date: 2026-08-02 docutils does not report a usable line for every node, and what it reports differs by front-end and by version. -At **docutils 0.21.2**, which this project currently pins, a bare `>>>` block +At **docutils 0.21.2** — which this project does not pin but does resolve, via +its Sphinx and myst-parser constraints — a bare `>>>` block nested in a `.. note::`, a list item, a block quote or a `{tab}` directive reports `line=None, source=None`. A top-level reStructuredText `doctest_block` reports its **last** line. A MyST fence reports its **first** line. An @@ -43,17 +44,39 @@ so substituted classes leak into subsequent parses that did not ask for them. The claim that substitution is "fully scoped to one parse with no process-global mutation" is wrong. -## Decision +## Direction -**Raise the docutils floor instead.** +**Raise the docutils floor instead** — but that is a support-matrix decision, not +a pin change, and it is not this record's to make alone. docutils 0.22 fixed the underlying defect upstream: a nested block reports a real line, and top-level and nested blocks agree on reporting the **first** line rather than the last. Every case this record was invented to work around is -resolved by a floor bump, with no probe, no substitution and no fallback path. +resolved by the floor, with no probe, no substitution and no fallback path. -The cost is that docutils ≥ 0.22 requires Sphinx ≥ 9.1, so this is a coordinated -dependency move rather than a one-line pin change. +Getting there is three moves, and the second is upstream of this repository: + +1. **Raise `requires-python` to `>= 3.11`.** Sphinx 9.0 is the first release that + permits docutils 0.22, and it declares `requires-python >= 3.11`. Dropping 3.10 + also touches the classifiers, the mypy and ruff target versions, and the CI + matrix. +2. **Ship a `gp-sphinx` release that widens its `sphinx < 9` cap.** This is the + binding constraint today, and it is not in this repository. With the cap in + place, a resolver asked for `docutils >= 0.22` reports the requirements + unsatisfiable. +3. **Then** declare `docutils >= 0.22, < 0.23`. An open-ended floor breaks the + moment a resolver reaches 0.23. + +Resolved versions per interpreter, with `docutils >= 0.22` requested: + +| Python | docutils | myst-parser | Sphinx | +|---|---|---|---| +| 3.10 | 0.23 | 0.13.6 | 3.5.3 — a degenerate backtrack, not viable | +| 3.11 | 0.22.4 | 5.1.0 | 9.0.4 | +| 3.12–3.14 | 0.22.4 | 5.1.0 | 9.1.0 | + +Today's lock resolves Sphinx 8.1.3 on Python 3.10 and 8.2.3 elsewhere, and +neither permits docutils 0.22. ## Consequences @@ -72,12 +95,18 @@ statement. ## Open -- Whether to raise the floor now or support both, since 0.21.2 is what the - project pins today. Supporting both means keeping the normalization branch and +- **The blocking question: does this project drop Python 3.10?** Everything else + here is downstream of that, and it is a support-matrix decision that outlives + this record. Until it is settled, "raise the floor" is a direction, not a + decision — which is why this record's status stays `Draft`. +- Whether to raise the floor at all or support both, since docutils 0.21.2 is + what resolves today. Supporting both means keeping the normalization branch and documenting two behaviours for the same page. -- Whether the Sphinx ≥ 9.1 move belongs in this record or its own. It changes the - default group an unargumented block joins, which is a semantics change beyond - line numbers. +- Sequencing with the `gp-sphinx` cap. That release has to land first, and this + repository does not control it. +- Whether the Sphinx move belongs in this record or its own. Sphinx 9.1 changes + the default group an unargumented block joins, which is a semantics change + beyond line numbers. - Whether tests should pin exact `(path, line)` for a bare block nested in a `.. note::`, a list item, a block quote and a `{tab}` directive. They should — they are the regression net for the floor, and this repository already has diff --git a/docs/adrs/0006-pytest-private-api-compatibility.md b/docs/adrs/0006-pytest-private-api-compatibility.md index 35c6503..8cae260 100644 --- a/docs/adrs/0006-pytest-private-api-compatibility.md +++ b/docs/adrs/0006-pytest-private-api-compatibility.md @@ -53,8 +53,11 @@ What private surface is depended on, and how does a pytest release that changes it fail? The current surface is `_get_checker`, `get_optionflags`, -`_get_continue_on_failure`, `_get_report_choice`, `MultipleDoctestFailures`, and -a `pytest.DoctestItem` subclass. `_init_runner_class` is explicitly *not* usable: +`_get_continue_on_failure`, `_get_report_choice` and `MultipleDoctestFailures`. +Not everything in the quarantine is equally risky: {class}`pytest.DoctestItem` is +**public** — exported from `pytest` — so subclassing it is ordinary API use. The +collector class filtered out of the multicall result is private, and that filter +is the part that needs a version matrix. `_init_runner_class` is explicitly *not* usable: `PytestDoctestRunner` is defined inside it ([`_pytest/doctest.py:178-181`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L178-L181)) and is unreachable by name, which is why the `OutcomeException` re-raise, @@ -67,11 +70,23 @@ Quarantine every private import in one module, `pytest_doctest_docutils._compat` with a pinned support matrix. **Filter the built-in's collector out of the `pytest_collect_file` result, in a -`@pytest.hookimpl(wrapper=True)`.** The directory collector consumes the -multicall result directly, and returning a modified result from a hook wrapper is -documented and supported. Filtering there removes the duplicate *before* the -built-in collector parses anything, so neither the duplicate item nor its -collection error is ever produced — which late deselection cannot achieve. +hook wrapper.** The directory collector consumes the multicall result directly, +and returning a modified result from a wrapper is documented and supported. +Filtering there removes the duplicate *before* the built-in collector parses +anything, so neither the duplicate item nor its collection error is ever produced +— which late deselection cannot achieve. + +**Use the old-style `hookwrapper=True` with `outcome.force_result()`.** New-style +`wrapper=True` is gated on **pluggy ≥ 1.2**, not on pytest 8 — it works fine +under pytest 7 with a new enough pluggy. But pytest 7 declares only +`pluggy>=0.12,<2.0`, so a resolver may legally install pluggy 1.0 or 1.1, where +`wrapper=True` raises `TypeError` *while importing the plugin* — a session-wide +abort, which this record and {doc}`0001-typed-vanilla-doctest-core` both forbid. +Old style needs no floor at all and was verified working on both pytest 7 and 9. + +Whichever spelling is used, **name the minimum supported pytest**. The CI matrix +floor is 7 and the package declares no pytest dependency, so today the support +statement exists only in the workflow file. **Fail on an unsupported pytest only when an affected document is collected**, not at plugin registration. A `pytest11` plugin that raises at import takes down From 647217f94fbf628e916d4f832447cfaefe5519a8 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 15:01:29 -0500 Subject: [PATCH 15/31] notes(analyses): Correct the drift a second review found MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit why: Several notes still described PR #87 as shipped, carried the superseded GroupTest model, or repeated claims later verification falsified. The true baseline is now known from trunk's source: released gp-libs appends one DocTest per matched node named page.md[k] and gives each its own copied globs. what: - Split the taxonomy rows: released gp-libs is one block, one item, isolated globals; PR #87 is a separate proposed row - Move Sphinx into the N-DocTests row and drop "unoccupied", since Sphinx already executes that shape without addressable ids - Rename the identity rule to never-source-coordinate-derived, and say an ordinal among extracted blocks satisfies it — which is what the released finder already uses - Replace GroupTest with GroupPlan in the data-flow diagram, and attribute the clone and the synthetic page to PR #87 - Separate Sphinx's three units in 20: runner call per block, shared state per group, result as process-wide counters - Drop BlockKind.node_types from the seam list and replace the rejected compile-policy callable with the private ExecutionProfile, matching ADR 0001 - Record that :pyversion: is declared on both testcode and testoutput and honoured on neither - Correct the reporter section: an observer is additive and does not silence the stream, and a system_message carries no stable code - Record why the state_classes substitution is not parse-scoped - Fix the Sphinx heading and link that disagreed on version --- notes/analyses/00-taxonomy.md | 22 ++++++++++------ notes/analyses/15-sphinx-ext-doctest.md | 10 ++++--- notes/analyses/16-docutils-myst.md | 26 ++++++++++++++----- notes/analyses/20-data-structures.md | 13 +++++++--- notes/analyses/21-data-flows.md | 6 ++--- notes/analyses/22-extension-seams.md | 4 +-- .../23-namespace-scope-and-test-identity.md | 10 +++++-- notes/analyses/90-bibliography.md | 4 +-- 8 files changed, 63 insertions(+), 32 deletions(-) diff --git a/notes/analyses/00-taxonomy.md b/notes/analyses/00-taxonomy.md index dc53189..dbc6d12 100644 --- a/notes/analyses/00-taxonomy.md +++ b/notes/analyses/00-taxonomy.md @@ -9,7 +9,7 @@ a different philosophy. | # | Axis | Positions | |---|---|---| | 1 | **Sharing unit vs. selection unit** | same object · different objects, acknowledged · different objects, unacknowledged | -| 2 | **Test identity** | author-declared name · symbol-derived · positional (line/column or byte range) | +| 2 | **Test identity** | author-declared name · symbol-derived · ordinal among extracted blocks · source-coordinate-derived (line/column or byte range) | | 3 | **Runtime object model** | stdlib `DocTest`/`Example` · own model with a bridge · own model, no bridge | | 4 | **Document model** | real parse tree · flat character spans · regex over text · none | | 5 | **Option representation** | `int` bitmask · structured state · enum | @@ -28,7 +28,8 @@ a different philosophy. | Sybil | **different, unacknowledged** | positional `line:N,column:N` | stdlib `Example`, one-line `DocTest` fork | flat character spans | | xdoctest | same (one `DocTest` per docstring) | `Callname:N` | own, with a late bridge back | none for `.rst`/`.txt` | | pytest-examples | none — no implicit sharing | positional `path:start-end` | none | regex over fences | -| `doctest_docutils` today | configurable; `per-block` is different-and-guarded | group name, or `page.md[k]` | stdlib | real doctree | +| `doctest_docutils` released | same object (one block, one item, isolated copied globals) | `page.md[k]` ordinal | stdlib | real doctree | +| PR #87 (proposed) | configurable; `per-block` is different-and-guarded | group name, or `page.md[k]` | stdlib | real doctree | | ADR 0001 | **different, decoupled by construction** | group name, or `page.md[k]` | stdlib | real doctree | | System | 5 options | 6 extension | 7 vs. pytest doctest | 8 strictness | 9 direction | @@ -39,7 +40,7 @@ a different philosophy. | Sybil | `int` | callable aliases | replace by instruction (`-p no:doctest`) | strict | read-only | | xdoctest | structured `TypedDict` + bridge | two registries, else fork | unregisters it | **permissive** | read-only | | pytest-examples | n/a | none | no collector — composes trivially | no want at all | **write-back** | -| `doctest_docutils` today | `int` + a forked lookup | directive subclassing | blocks it, imports its privates | strict | read-only | +| `doctest_docutils` released | `int` | directive subclassing | blocks it, imports its privates | strict | read-only | | ADR 0001 | `int` + registry | `Protocol` + nominal + `BlockKind` registry | **compose; require it** | strict | read-only | ## What the matrix shows @@ -55,12 +56,17 @@ product space is four cells: | | one node id | N node ids | |---|---|---| -| **one `DocTest`** | Sphinx, `doctest_docutils` `merged` | — (incoherent) | -| **N `DocTest`s** | *unoccupied until ADR 0001* | Sybil, `doctest_docutils` `per-block` | +| **one `DocTest`** | PR #87's `merged` | — (incoherent) | +| **N `DocTest`s** | `sphinx.ext.doctest`, but with *no* ids; ADR 0001 adds the pytest identity | Sybil, PR #87's `per-block`, released `doctest_docutils` (no sharing) | -The bottom-right cell is where the silent failure lives. The bottom-left cell — -per-block `DocTest`s under one item — gives per-block reporting *and* an -unsplittable sharing unit, and no surveyed project occupies it. +The bottom-right cell is where the silent failure lives — but only when the +blocks share state. Released `doctest_docutils` sits there safely because its +blocks share nothing: each gets its own copied `globs`. + +The bottom-left cell is where the design goes. Sphinx already *executes* that +shape — N `DocTest`s over one group namespace — but produces no addressable unit +for any of them, since every block shares one `DocTest.name`. Giving that shape a +pytest identity is the contribution; the execution shape is not new. **Axis 3 has an empirical answer.** xdoctest is the controlled experiment for abandoning the stdlib object model, and it is now building the bridge back. The diff --git a/notes/analyses/15-sphinx-ext-doctest.md b/notes/analyses/15-sphinx-ext-doctest.md index 3590dd7..538366b 100644 --- a/notes/analyses/15-sphinx-ext-doctest.md +++ b/notes/analyses/15-sphinx-ext-doctest.md @@ -33,11 +33,13 @@ losses live: an orphan `testoutput` is discarded; a `testoutput` following a `doctest` block is discarded, because a doctest entry has length 1 and fails the `len(latest_test) == 2` guard; and a second `testoutput` *replaces* the first. -A fourth silent loss is in the directive rather than `add_code`: `:pyversion:` is -in `TestcodeDirective.option_spec` +A fourth silent loss is in the directives rather than `add_code`: `:pyversion:` is +declared on **both** `testcode` ([`:174-180`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L174-L180)) -and is then ignored, because the version gate only runs for `doctest` and -`testoutput`. +and `testoutput` +([`:184-190`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L184-L190)), +and honoured on neither — the version gate runs only for `doctest`. An author who +writes it on either gets no error and no gate. `:options:` on a `testcode` is **not** a silent loss. It is absent from `TestcodeDirective.option_spec` entirely, so writing it is an unknown-option diff --git a/notes/analyses/16-docutils-myst.md b/notes/analyses/16-docutils-myst.md index 9b524c4..f1b002c 100644 --- a/notes/analyses/16-docutils-myst.md +++ b/notes/analyses/16-docutils-myst.md @@ -24,7 +24,7 @@ docutils.nodes.Element attributes: dict[str, Any] docutils.parsers.rst.Parser .state_classes an INSTANCE attribute, therefore substitutable - per parse with no process-global mutation + per parse — but NOT scoped in practice: see below myst_parser.parsers.docutils_.Parser(RstParser) [v5.1.0:235] .settings_spec = (..., create_myst_settings_spec(), *RstParser.settings_spec) @@ -105,16 +105,28 @@ bare language fence onto a directive name, so a project that prefers not to writ ## Reporter behaviour Default settings send reporter output to stderr and raise `SystemMessage` at -`halt_level`, aborting mid-parse. Both are configurable: raising `halt_level` past -the abort threshold and attaching an observer with `Reporter.attach_observer` -turns messages into values. `Reporter.system_message` notifies observers for any -level above `DEBUG` independently of `report_level`, so observation and display are -separable — which is what makes ADR 0004's code-keyed suppression possible without -losing the underlying record. +`halt_level`, aborting mid-parse. + +Turning messages into values takes **three** settings, not one. `attach_observer` +is *additive*: the observer receives the message and the warning stream still gets +written. So all of `halt_level` above 4 (both to avoid the abort and because a +halting message bypasses observer notification entirely), `report_level` at 5 or +`warning_stream` disabled to stop the write, and the observer itself. + +A `system_message` carries a level and text and **nothing semantically stable** — +no code. So a downstream that wants to suppress or promote by category has to +*classify* the message, and cannot key on an attribute docutils does not provide. +That is the open problem in ADR 0004. ## What it cannot do - **Scope a directive registration** to one parse, one document or one thread. +- **Scope a `state_classes` substitution either.** `state_classes` is an instance + attribute, which makes substitution *look* parse-local — but a nested parse + builds its machine from `nested_sm_kwargs`, so a top-level substitution never + reaches a nested block, and `RSTState.nested_sm_cache` is a shared **class** + attribute that leaks substituted classes into later parses. This is why ADR 0005 + abandoned the mechanism. - **Report a line for every node.** See the table above. - **Type its own attribute channel.** `Element.attributes` is `dict[str, Any]`, and typeshed's stub for `get(key, failobj: _T) -> _T` is actively wrong — it claims diff --git a/notes/analyses/20-data-structures.md b/notes/analyses/20-data-structures.md index e38c83b..dea809d 100644 --- a/notes/analyses/20-data-structures.md +++ b/notes/analyses/20-data-structures.md @@ -10,7 +10,7 @@ The disagreements are entirely about which of those four are the same object. |---|---|---|---|---| | CPython `doctest` | `Example` | `DocTest` | `DocTest.globs` | `TestResults` (2 ints + an attribute) | | `_pytest.doctest` | `Example` | `DocTest` = one `Item` | `DocTest.globs`, wiped per item | `MultipleDoctestFailures` → `ReprFailDoctest` | -| `sphinx.ext.doctest` | `TestCode` | `TestGroup` | `ns` assigned post-construction | six ints + text to a file | +| `sphinx.ext.doctest` | `TestCode` | one `DocTest` **per block** (`TestGroup` is the batching unit, not the execution unit) | `ns`, assigned post-construction to every block's `DocTest` | six ints + text to a file | | Sybil | `Region` | `Example` = one `Item` | `Document.namespace` | truthy return or exception | | xdoctest | `DoctestPart` | own `DocTest` | `global_namespace` | own report objects | | pytest-examples | `CodeExample` | the block, exec'd once | explicit `module_globals=` | captured output, or a rewrite | @@ -18,9 +18,14 @@ The disagreements are entirely about which of those four are the same object. Reading across the "shared-state unit" column against the "execution unit" column is the whole design problem. Sphinx and Sybil both put the shared state at a -coarser granularity than the execution unit; only Sphinx also puts the *pytest -item* at that same coarser granularity, which is why Sphinx never hits Sybil's -`-k` failure. +coarser granularity than the execution unit. Sphinx gets away with it by having +no selectable unit at all — it is a builder, not a test runner, so there is +nothing for a `-k` to split. Sybil does not: it hands out one pytest item per +span over one shared mapping, which is the failure. + +Three units are easy to conflate for Sphinx specifically, so keep them apart: the +**runner call** is per block, the **shared state** is per group, and the **result** +is six process-wide integers. ## Field-by-field: what a source unit carries diff --git a/notes/analyses/21-data-flows.md b/notes/analyses/21-data-flows.md index ad371e8..f6b1f40 100644 --- a/notes/analyses/21-data-flows.md +++ b/notes/analyses/21-data-flows.md @@ -29,7 +29,7 @@ sphinx.ext.doctest ADR 0001 path ─► markup.parse_file ─► (Blocks, Diagnostics) │ - ├─► project() ─► GroupTest{group, tests[], globs} + ├─► project() ─► GroupPlan{group, blocks[], seed} │ pure: no docutils, no pytest, no filesystem, │ no user code — :skipif: passes through unevaluated │ @@ -63,14 +63,14 @@ two. **At the compile call.** stdlib hard-codes `"single"`. Sphinx flips a mutable builder attribute and reads it through a process-global rebinding of -`doctest.compile` that is never restored. `doctest_docutils` today clones the +`doctest.compile` that is never restored. PR #87 proposes cloning the mangled loop's code object to get a private version of that rebinding. ADR 0001 carries the mode on the example data and reads it in a loop it owns — the only one of the four that neither mutates process state nor copies a code object. **At the location.** stdlib computes `test.lineno + example.lineno + 1`. Sphinx gives every block in a group the same `DocTest.name` and pays for it by overriding -a private method to swallow an `IndexError`. `doctest_docutils` today fabricates a +a private method to swallow an `IndexError`. PR #87 fabricates a synthetic page so the arithmetic stays true across merged blocks. ADR 0001 gives each block its own `DocTest`, so the arithmetic is true without fabrication. diff --git a/notes/analyses/22-extension-seams.md b/notes/analyses/22-extension-seams.md index 013c20c..05de168 100644 --- a/notes/analyses/22-extension-seams.md +++ b/notes/analyses/22-extension-seams.md @@ -85,10 +85,10 @@ needs it. Applied to the candidates that came up: | Candidate | Verdict | |---|---| -| `BlockKind` registry | **Yes.** Turns "a new block kind" from an edit to a method branching on string literals into adding a tuple. Its `node_types` field is what keeps the `nodes.comment` requirement reviewable rather than buried in a `findall` call | +| `BlockKind` registry | **Yes.** Turns "a new block kind" from an edit to a method branching on string literals into adding a tuple. Which docutils node classes a kind arrives as stays in `markup/`, not on `BlockKind`, so the leaf keeps its stdlib-only contract | | Output checker injection | **Yes.** The highest-demand seam, and the one Sybil closed entirely by hard-coding `checker=OutputChecker()` — adding one there requires subclassing two classes | | `Frontend` protocol | **Yes.** Four implementations ship on day one (rST, MyST, text, Python docstrings) | -| Compile-policy callable | **No.** One implementation, and a second would have to reproduce the `` filename shape or break `linecache` | +| `ExecutionProfile` (private) | **Yes, private.** [PR #59](https://github.com/git-pull/gp-libs/pull/59) adds top-level `await` — a second execution policy a `Literal["single", "exec"]` cannot express. Stays private until an out-of-tree caller exists | | A per-example observer protocol | **No.** pytest gets failures through `report_*` and `MultipleDoctestFailures`; the CLI uses `summarize()`. No third consumer | | A registry *object* with builders, digests and manifests | **No.** Two module-level dicts and two `register_*` functions do the same work | | Entry-point plugin discovery | **No.** Not until a caller outside the package exists. It also breaks xdist's collection-purity requirement, since discovery is an import side effect rather than a function of (files, argv, ini) | diff --git a/notes/analyses/23-namespace-scope-and-test-identity.md b/notes/analyses/23-namespace-scope-and-test-identity.md index 70bcb75..e284608 100644 --- a/notes/analyses/23-namespace-scope-and-test-identity.md +++ b/notes/analyses/23-namespace-scope-and-test-identity.md @@ -69,12 +69,18 @@ structural, not incidental: Every one of these evaporates when the item is the sharing unit. -## Test identity: never positional +## Test identity: never source-coordinate-derived -Two of the surveyed projects derive node ids from position — Sybil's +Two of the surveyed projects derive node ids from **source coordinates** — +Sybil's [`line:{line},column:{column}`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/sybil.py#L155-L157) and pytest-examples' `path:start-end`. +An **ordinal among the extracted blocks** is a different thing and is fine. It is +not a source coordinate: `page.md[3]` is unchanged by a paragraph inserted above +it, which is exactly the edit that renames a `line:N` id. Released +`doctest_docutils` names its tests that way already, and the rule preserves it. + For a *documentation* test runner this is indefensible, because prose above examples is the thing that changes most often. Adding a sentence renames every downstream test, which breaks `--lf`, `--nf`, checked-in deselect files, xfail diff --git a/notes/analyses/90-bibliography.md b/notes/analyses/90-bibliography.md index 5f6daed..07bcc0a 100644 --- a/notes/analyses/90-bibliography.md +++ b/notes/analyses/90-bibliography.md @@ -102,9 +102,9 @@ Documentation: [`Doc/library/doctest.rst`](https://github.com/python/cpython/blo | `_get_asyncio_mode` | [`:222`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L222) | resolve once, query once | | `pytest_configure` | [`:295-301`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L295-L301) | detecting an unset default via the sentinel | -## Sphinx — `v9.1.0` +## Sphinx — `v8.2.3` -[`sphinx-doc/sphinx @ v9.1.0`](https://github.com/sphinx-doc/sphinx/tree/v8.2.3) · +[`sphinx-doc/sphinx @ v8.2.3`](https://github.com/sphinx-doc/sphinx/tree/v8.2.3) · [`sphinx/ext/doctest.py`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py) | Symbol | Anchor | Cited for | From 1eb642ab44bbb2272e6674c70deffd4edc58f441 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 17:17:18 -0500 Subject: [PATCH 16/31] docs(adrs[0001]): Take metadata off Example and design the seams MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit why: A third review found the surrounding model incomplete or self-contradictory even though the invariant holds. The Example subclass was the worst of it: restoring equality with an isinstance check makes two UNRELATED subclasses compare equal, and equal to any third party's bare subclass. Verified by execution on 3.10 through 3.15. what: - Move execution policy off doctest.Example onto ProjectedBlock. The runner establishes an active execution request before delegating to stdlib run() and clears it in a finally, so stock Example objects stay stock and nothing in the kernel is subclassed for metadata - Replace the type set with the lifecycle it actually has: ParsedBlock/ParsedOutput (inert) -> ProjectedBlock (stock DocTest, phase, profile, gate, gateable ExpectedOutput) -> GroupPlan (immutable) -> RunContext (live globs) -> BlockResult/GroupResult - Drop want from the parsed layer: neither of its owners is the parsed block - Give BlockKind a profile NAME so a public type stops holding a private one - Add an error outcome and primary/secondary failures, with the precedence rule stated and pytest classification left to the adapter - Design Settings as three facets with a stated precedence chain, and resolve the frozen registry alongside them rather than pointing at ADR 0006, which defines no freeze - Record that entry-point discovery is not what breaks xdist determinism; nondeterministic contribution is - Split DocumentParser from extract_blocks so a Sphinx extension can pass its own resolved doctree, and say why that is not a builder - Move Python object discovery out of markup/: it takes an object, not text, and is DocTestFinder-shaped - Replace the share axis with ungrouped = default | block, which is the question actually being answered - Correct __lt__: it compares (name, filename, lineno, id), with name leading — the hazard holds, the old wording did not - Correct the Sphinx precedent: setup blocks are combined into one simulated DocTest, likewise cleanup; only test blocks are per-block --- docs/adrs/0001-typed-vanilla-doctest-core.md | 236 ++++++++++++++----- 1 file changed, 177 insertions(+), 59 deletions(-) diff --git a/docs/adrs/0001-typed-vanilla-doctest-core.md b/docs/adrs/0001-typed-vanilla-doctest-core.md index c306597..8c83a3e 100644 --- a/docs/adrs/0001-typed-vanilla-doctest-core.md +++ b/docs/adrs/0001-typed-vanilla-doctest-core.md @@ -98,12 +98,19 @@ One {class}`pytest.Item` per (document, group). Inside it, a tuple of per-block `DocTest`s run in phase order against one live `globs` mapping that never leaves the item. -The execution shape is not new: `sphinx.ext.doctest` already runs several -`DocTest`s against one shared group namespace. What is new is giving that shape a -**pytest identity**. Sphinx produces no selectable, reportable unit for a group — -every block in it shares one `DocTest.name`, which is why `SphinxDocTestRunner` -overrides a private stdlib method to swallow the resulting `IndexError`. Mapping -the group onto one {class}`pytest.Item` is the contribution. +The execution shape is partly precedented. `sphinx.ext.doctest` runs several +`DocTest`s against one shared group namespace — but only for the *test* phase. +All of a group's `testsetup` blocks are combined into a **single** simulated +`DocTest` named `f"{group.name} (setup code)"`, and likewise cleanup; only +`group.tests` is one `DocTest` per block +([`sphinx/ext/doctest.py:525-556`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L525-L556)). + +So this design is per-block in all three phases where Sphinx is per-block in one, +and — more importantly — Sphinx produces no selectable, reportable unit for any +of them: every ordinary test block in a group shares one `DocTest.name`, which is +why `SphinxDocTestRunner` overrides a private stdlib method to swallow the +resulting `IndexError`. Mapping the group onto one {class}`pytest.Item` while +each block keeps its own identity is the contribution. What that buys, for free and with no override of `repr_failure` or `reportinfo`: per-block failure locations, per-block gutters, and per-block "location unknown". @@ -250,7 +257,8 @@ settings one frozen Settings, resolved once <- leaf; everything rea | blocks inert data: Block, BlockKind, Phase, Diagnostic, Example | (stdlib imports only) -markup/ _rst, _myst, _text, _python -> (blocks, diagnostics) +markup/ DocumentParser: _rst, _myst -> doctree + | extract_blocks(doctree) -> (parsed blocks, diagnostics) | project grouping, pairing, phase order, naming -> GroupPlan | @@ -264,8 +272,8 @@ pytest_doctest_docutils collection, items, globs lifetime, reporting | Layer | Owns | Must not know | |---|---|---| -| `settings` | One frozen `Settings` resolved once, with `None` sentinels at the resolve boundary so a future default change is announceable | pytest's `Config`, argparse, ini format, Sphinx's `app`. The host extracts; this resolves | -| `blocks` | `Block`, `BlockKind` registry, `Phase`, `Diagnostic`, `PlannedBlock`, `GroupPlan`, `ExecutionProfile`, `Example(doctest.Example)` | docutils, MyST, Sphinx, pytest, xdist, the filesystem. Stdlib imports only, enforced by an `import-linter` contract | +| `settings` | Three immutable facets — `ParseSettings`, `ProjectionSettings`, `RunSettings` — plus the **frozen registry**, resolved together exactly once per session. `None` sentinels at the resolve boundary make a future default change announceable | pytest's `Config`, argparse, ini format, Sphinx's `app`. The host extracts; this resolves | +| `blocks` | `ParsedBlock`, `ParsedOutput`, `BlockKind`, `Phase`, `Diagnostic`, `ProjectedBlock`, `GroupPlan`, `RunContext`, `ExecutionProfile`, the result types. **No stdlib subclasses.** | docutils, MyST, Sphinx, pytest, xdist, the filesystem. Stdlib imports only, enforced by an `import-linter` contract | | `markup/` | Text → `(blocks, diagnostics)`. **The whole docutils vocabulary**: which node classes each kind may arrive as, the `BlockAttributes` stamp, line-number recovery and its per-front-end meaning, `.. include::` attribution, `nodes.comment` traversal, reporter capture, idempotent directive registration, per-kind `option_spec` | Groups as a runtime concept, `DocTest`, pytest, pairing | | `project` | The **only** place grouping exists: `*` expansion, anonymous naming, phase order, `testcode`/`testoutput` pairing, option defaults, name minting. A pure function | docutils, pytest, the filesystem, whether anything will run. Evaluates no user code | | `runner` | `_DocTestRunner__run`; option merge, `SKIP`-after-merge, `FAIL_FAST`, `report_*` dispatch, version shims. `run_group()` owns phase sequencing, run-time `:skipif:`, the profile's context manager, and the `try`/`finally` guaranteeing `testcleanup` | docutils, markup, pytest. Never overrides `run()` | @@ -275,6 +283,41 @@ pytest_doctest_docutils collection, items, globs lifetime, reporting `_pytest.doctest`, behind a pinned support matrix. See {doc}`0006-pytest-private-api-compatibility`. +### Settings and the frozen registry + +Settings are **faceted**, not one object every layer imports — a single shared +`Settings` becomes a dependency magnet, and the facets have genuinely different +consumers: + +| Facet | Owns | Read by | +|---|---|---| +| `ParseSettings` | encoding, MyST extensions, front-matter policy, diagnostic promotion and suppression | `markup/` | +| `ProjectionSettings` | `ungrouped` (`"default"` or `"block"`), wildcard resolution, name minting | `project` | +| `RunSettings` | optionflags, `continue_on_failure`, checker selection, report style | `runner` | + +Precedence is fixed and stated once: **per-block directive option → per-document +front matter → host configuration (pytest ini/CLI, Sphinx `conf.py`, argparse) → +built-in default.** Each adapter translates its host's configuration into the +facets exactly once, at session start; no layer reads a host config object. + +The **registry** is resolved and frozen alongside them, not as a separate object +with its own lifecycle. It holds the contributed block kinds, front ends and +execution profiles. "Frozen" means an immutable mapping built once before +collection begins; registering afterwards is an error, not a silent late +addition. + +This is what {ref}`the collection contract ` means by +*frozen registry*. Two consequences worth stating: + +- **Entry-point discovery is not inherently unsafe** under pytest-xdist. Workers + are fresh processes that re-run plugin loading with the same argv, so + entry-point resolution is identical in the controller and every worker. What + breaks determinism is *nondeterministic* contribution — a conftest that + registers conditionally on the environment — not discovery itself. +- **Nothing here is public in v1.** Contribution happens through the host's own + extension point (a pytest hook, Sphinx's `setup(app)`), and the registry type + stays private until a caller outside the package needs it. + ### Item lifecycle The custom item is load-bearing, and half-reusing {class}`pytest.DoctestItem` @@ -319,7 +362,7 @@ referents. Each term below is decided once and used only that way. | `globs` | the dict examples exec in; [`DocTest.__init__` stores a **copy**](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L565) | — | assigned to `test.globs` after construction, run with `clear_globs=False` | Keep `globs` for the mapping | | namespace | — | [`doctest_namespace`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L721) means *injected names* | — | **Not** used for the sharing unit; pytest owns the word | | group | — | `xdist_group` is a *scheduling* affinity marker ([`remote.py:245-254`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/remote.py#L245-L254)) | the author-facing bucket: `.. doctest:: intro`, `default`, `*` | Adopt `group` for the sharing unit. The xdist affinity key is *derived*, never the group name | -| scope | — | the fixture-lifetime ladder | — | Reserved for pytest. The block-vs-document axis becomes `share` | +| scope | — | the fixture-lifetime ladder | — | Reserved for pytest. The real question is what an *unlabelled* block joins, so the setting is `ungrouped = "default" | "block"`, not a `share` axis | | test / item / block | `DocTest`, `Example` | `Item` | [`TestCode`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L235) is the parsed unit | Three nouns: `Example` (stdlib), `Block` (parsed), `DocTest` (runnable) | | skip | the `SKIP` flag, short-circuiting before `report_start` | a reported outcome with a reason, **at item granularity** | [drops the node entirely](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L449-L450) | doctest's mechanism (set `SKIP`, never drop the node); pytest's outcome where the granularity allows it — see [](#the-outcome-contract). Sphinx's drop is deliberately rejected | | directive | inline `# doctest: +FLAG` | — | a docutils directive with an `option_spec` | Reserved for the docutils meaning. doctest's form is "inline flags" | @@ -336,10 +379,12 @@ class Phase(enum.IntEnum): CLEANUP = 2 -class Block(t.NamedTuple): +# --- parsed: inert, produced by extraction, owns no semantics ------------- + + +class ParsedBlock(t.NamedTuple): kind: str # registered BlockKind name - source: str # dedented body, verbatim - want: str | None # from a paired testoutput + source: str # dedented body, verbatim author text path: pathlib.Path # the file the text lives in, not the collected document line: int | None # None when docutils could not recover one position: int # 0-based document pre-order index; the naming key @@ -349,45 +394,91 @@ class Block(t.NamedTuple): hidden: bool +class ParsedOutput(t.NamedTuple): + """A `testoutput` body. Not a block: it never runs.""" + + text: str + path: pathlib.Path + line: int | None + position: int + groups: tuple[str, ...] + options: t.Mapping[int, bool] + skipif: str | None # a gated output means its testcode expects nothing + + class BlockKind(t.NamedTuple): name: str phase: Phase - profile: ExecutionProfile # how a body of this kind is compiled and run + profile_name: str # resolved against the frozen registry, not held here pairs_with: str | None grouped: bool -class PlannedBlock(t.NamedTuple): +# --- projected: one per block, with everything the runner needs ------------ + + +class ExpectedOutput(t.NamedTuple): + text: str + options: t.Mapping[int, bool] + skipif: str | None # when truthy at run time, `want` becomes "" + + +class ProjectedBlock(t.NamedTuple): phase: Phase - skipif: str | None # UNEVALUATED; gated in run_group(), not at collection - test: doctest.DocTest # one block, its own filename/lineno/docstring + test: doctest.DocTest # stock objects; its own filename/lineno/docstring + profile: ExecutionProfile # uniform for the block, resolved at projection + skipif: str | None # UNEVALUATED; gated in run_group() + expected: ExpectedOutput | None # paired testoutput, itself gateable class GroupPlan(t.NamedTuple): group: str - blocks: tuple[PlannedBlock, ...] # in phase order; IMMUTABLE + blocks: tuple[ProjectedBlock, ...] # in phase order; IMMUTABLE seed: t.Mapping[str, t.Any] # initial namespace; copied per attempt -class GroupRun: +# --- run: one attempt ------------------------------------------------------ + + +class RunContext: """One execution attempt. Owns the live mapping; a plan never does.""" plan: GroupPlan - globs: dict[str, t.Any] # cleared and reseeded per attempt + globs: dict[str, t.Any] # cleared in place and reseeded per attempt class BlockResult(t.NamedTuple): - block: PlannedBlock - outcome: t.Literal["passed", "failed", "skipped"] - reason: str | None # the gate expression, when skipped + block: ProjectedBlock + outcome: t.Literal["passed", "failed", "skipped", "error"] + reason: str | None # gate expression when skipped; message when error + error: BaseException | None # gate or profile failure, not a doctest one class GroupResult(t.NamedTuple): group: str blocks: tuple[BlockResult, ...] failures: tuple[doctest.DocTestFailure | doctest.UnexpectedException, ...] + primary: BaseException | None # what runtest() re-raises + secondary: tuple[BaseException, ...] # e.g. a cleanup failure after a body failure ``` +`ParsedBlock` carries no `want`, because neither owner of a `want` is the parsed +block: for a prompt-form block it is *inside* `source` and +{class}`doctest.DocTestParser` extracts it at projection, and for a +`testcode`/`testoutput` pair it is a separate `ParsedOutput`. Conflating the two +was what made "projection owns pairing" untrue. + +`BlockKind` names a profile rather than holding one, so a public type never +contains a private one. The name resolves against the frozen registry. + +`BlockResult.outcome` needs `"error"` because a gate that raises, or a profile +that fails to establish, is neither a pass, a fail, nor a skip. +`GroupResult.primary` and `.secondary` make the precedence explicit: **a body +failure outranks a cleanup failure**, a control-flow exception +({exc}`pytest.skip`, `xfail`, exit) outranks both, and the loser is recorded +rather than dropped. Classifying which exceptions are control-flow is the pytest +adapter's job, not the core's. + `Block.line` being nullable is load-bearing, not defensive. A bare `>>>` block nested in a `.. note::`, a list item or a block quote reports `line=None, source=None` from docutils, and an `.. include::`-ed block numbers against the @@ -444,8 +535,44 @@ docutils into a layer declared stdlib-only. The runtime objects are stdlib's, unconditionally. Precision lives in a parallel layer that never changes what is constructed. +- **Parsing and extraction are two seams, not one.** A single + `Frontend.parse(text, path)` cannot serve Sphinx, because a Sphinx extension + already *has* a doctree and a raw re-parse is not the same tree. So: + + ```python + class DocumentParser(t.Protocol): + """Text -> doctree. Plural implementations: _rst, _myst, third-party.""" + + suffixes: t.ClassVar[frozenset[str]] + + def parse( + self, text: str, path: pathlib.Path, *, settings: ParseSettings + ) -> tuple[nodes.document, tuple[Diagnostic, ...]]: ... + + + def extract_blocks( + doctree: nodes.document, *, settings: ParseSettings + ) -> ParseResult: ... + ``` + + `extract_blocks` is deliberately a plain function, not a `Protocol`: exactly + one extractor is the *point* of the split, and a second implementation would + reintroduce the standalone-versus-Sphinx divergence it exists to prevent. + Standalone reST and MyST use both halves; a Sphinx extension calls only the + extractor, on the doctree it already resolved. + + This is not a Sphinx builder. A builder owns discovery, an `env`, an `outdir` + and a reporting format; an extractor is a pure function from a doctree to + blocks, and {doc}`0001-typed-vanilla-doctest-core`'s rejection of a builder + stands. + +- **Python object discovery is not a front end.** Finding doctests in a module's + docstrings takes an *object*, not `(text, path)`, and stdlib already has the + right shape for it. It is a {class}`doctest.DocTestFinder`-shaped adapter, and + putting a `_python` module in `markup/` was a category error. + - **`Protocol` for the seams, nominal subclassing for type-checkability.** - `Frontend` is a `Protocol` so a third party can supply one structurally. The + `DocumentParser` is a `Protocol` so a third party can supply one structurally. The shipped parser *also* subclasses {class}`doctest.DocTestParser` — not because the interpreter requires it (stdlib `doctest` performs no `isinstance` check on `parser` or `test_finder`; a duck-typed object works at runtime) but because @@ -483,7 +610,7 @@ cited. The full derivation is in `notes/analyses/`. |---|---| | A failure's file line is `test.lineno + example.lineno + 1`; `Example.lineno` is 0-based within the containing string | [`doctest.py:1344`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1344) | | `DocTest.__init__` **copies** the globs mapping, so a shared mapping must be assigned after construction and run with `clear_globs=False` | [`doctest.py:565`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L565) | -| Never sort collected tests by name: `__lt__` compares names as text, and a name carries its position as text, so `page.md[10]` sorts before `page.md[1]` | [`doctest.py:596`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L596) | +| Never sort collected tests. `__lt__` compares `(name, filename, lineno, id(self))`; `name` leads, so a name carrying its position as text sorts `page.md[10]` before `page.md[1]` however correct `lineno` is. `filename` and `lineno` only break ties among equal names | [`doctest.py:596-603`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L596-L603) | | The per-example loop is name-mangled; the supported in-loop seams are the four `report_*` methods and the injected checker | [`doctest.py:1286-1314`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1286-L1314) | | `run()` mutates global interpreter state for its duration and restores in `finally`; it is neither reentrant nor thread-safe | [`doctest.py:1534-1573`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1534-L1573) | | Each example compiles under `""` in `"single"` mode with `dont_inherit=True`; `"exec"` suppresses expression echo, emptying every `want` | [`doctest.py:1400`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1400) | @@ -560,41 +687,31 @@ that rebuild reuses the same `Example` objects. Metadata the runner does *not* need — groups, wildcards, pairing — never touches a stdlib object and dies in the projection layer. -*Price:* the subclass must restore equality, because -{meth}`doctest.Example.__eq__` gates on **exact type identity** -([`Lib/doctest.py:518`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L518)) — -a bare subclass compares unequal to a stock `Example` with identical fields, in -both directions, while hashing the same. So the subclass overrides `__eq__` with -an {func}`isinstance` check and re-binds `__hash__` explicitly; Python's -reflected-operand rule then makes equality symmetric again: +*Price:* **none of it goes on {class}`doctest.Example`.** An earlier draft put +compile mode on an `Example` subclass, which fails twice. +{meth}`doctest.Example.__eq__` gates on exact type identity +([`Lib/doctest.py:518`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L518)), +so a bare subclass is unequal to a stock `Example` with identical fields in both +directions while hashing the same. Restoring equality with an {func}`isinstance` +override then over-corrects: two *unrelated* subclasses compare equal to each +other, and to any third party's bare subclass. ```{doctest} >>> import doctest ->>> class Tagged(doctest.Example): -... def __eq__(self, other): -... if not isinstance(other, doctest.Example): -... return NotImplemented -... return (self.source, self.want, self.lineno, self.indent, -... self.options, self.exc_msg) == ( -... other.source, other.want, other.lineno, other.indent, -... other.options, other.exc_msg) -... __hash__ = doctest.Example.__hash__ ->>> plain, tagged = doctest.Example("1\n", "1\n"), Tagged("1\n", "1\n") ->>> tagged == plain, plain == tagged, tagged in [plain] -(True, True, True) - -Without the override, a bare subclass is unequal both ways: - ->>> class Bare(doctest.Example): pass ->>> bare = Bare("1\n", "1\n") ->>> bare == plain, plain == bare -(False, False) +>>> def tagged(name): +... ns = {"__eq__": lambda s, o: isinstance(o, doctest.Example) +... and s.source == o.source, "__hash__": doctest.Example.__hash__} +... return type(name, (doctest.Example,), ns) +>>> Exec, Await = tagged("Exec"), tagged("Await") +>>> Exec("1\n", "1\n") == Await("1\n", "1\n") +True ``` -The alternative — setting the attribute on a *stock* `Example` with no subclass -at all — is equality-safe by construction and equally durable. It is rejected -only because it forfeits the typed surface; if the subclass ever proves -troublesome, that is the fallback. +A block's execution policy is **uniform across its examples**, so it belongs on +`ProjectedBlock`, not on the examples. The runner establishes an *active +execution request* immediately before delegating to stdlib `run()` and clears it +in a `finally`; `_DocTestRunner__run` reads it. Stock `Example` objects stay +stock, and nothing in the compatibility kernel is subclassed for metadata at all. **Node-id granularity versus shared state.** *Position:* decouple them — N `DocTest`s under one node id. *Price:* selecting a group runs all its blocks; @@ -613,15 +730,16 @@ collection evaluates no *author-supplied* Python — `:skipif:` is carried throu unevaluated and run in `runtest()`. *Price:* `--collect-only` no longer shows which blocks will skip. +(the-collection-contract)= + Collection is **not** a pure function of (bytes, argv, ini), and claiming so would be wrong on four counts: `.. include::` reads transitive files, docutils directive implementations execute during parsing, the directive registry is process-global, and MyST plugins change the tree. The defensible contract is determinism over **complete source closure + normalized settings + frozen -registry**. Deferring the author's gate removes the largest divergence risk; it -does not make xdist divergence structurally impossible, and the -{doc}`registry freeze <0006-pytest-private-api-compatibility>` is what closes the -rest. +registry** — all three defined above. Deferring the author's gate removes the +largest divergence risk; it does not make xdist divergence structurally +impossible. **Sphinx compatibility versus silent-loss behaviours.** Sphinx silently discards an orphan `testoutput`, silently discards a `testoutput` following a `doctest` From 12ac9eb2109b63531c9e63be0665a5cdaa7adefb Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 17:19:08 -0500 Subject: [PATCH 17/31] notes(analyses): Correct a third round of drift why: Review found remaining inaccuracies, and the ADR 0001 model changed under the notes. Each correction below was verified against a pinned tag or by execution. what: - __lt__ compares (name, filename, lineno, id), with name leading. The hazard survives, and it is LIVE: find() sorts blocks named page.md[k], so an eleven-block page runs its eleventh block second - parse() covers the NORMALIZED input, not the input exactly: tabs are expanded, common indent stripped, and a comment-only example is dropped outright - pytest_xdist_make_scheduler is a broader affinity seam than _split_scope; the honest claim is scoped to the shipped schedulers - myst_fence_as_directive runs a fence through the directive of the SAME name; python does not become testcode without an alias - pytest-asyncio's _get_asyncio_mode is called from several sites, so the lesson is one conversion site, not one resolution - pytest-examples stores Python string indices, and one indent scalar does not invert a dedent in general - Reconcile with the corrected ADR model: ParsedBlock/ParsedOutput, compile mode on the projected block, ungrouped instead of share - Replace note 22's reason for deferring entry points: discovery is identical across xdist workers, so nondeterministic contribution is the hazard, not discovery. Record that the freeze is real even though the registry object is not --- notes/analyses/10-cpython-doctest.md | 13 ++++++++----- notes/analyses/12-pytest-xdist.md | 10 +++++++++- notes/analyses/13-pytest-asyncio.md | 15 ++++++++------- notes/analyses/16-docutils-myst.md | 7 ++++--- notes/analyses/17-prior-art.md | 10 ++++++---- notes/analyses/20-data-structures.md | 24 +++++++++++++----------- notes/analyses/22-extension-seams.md | 16 ++++++++-------- 7 files changed, 56 insertions(+), 39 deletions(-) diff --git a/notes/analyses/10-cpython-doctest.md b/notes/analyses/10-cpython-doctest.md index 8d942d6..6fcb4f8 100644 --- a/notes/analyses/10-cpython-doctest.md +++ b/notes/analyses/10-cpython-doctest.md @@ -28,10 +28,13 @@ Passing a shared dict through the constructor has no effect whatsoever. A shared mapping must be assigned to `test.globs` after construction, and the runner must be given `clear_globs=False` or it empties the mapping in its `finally`. -**`__lt__` compares names as text** -([`:596`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L596)). -Any name carrying a position as text sorts `[10]` before `[1]`. This fails -silently: every test passes, in the wrong order. +**`__lt__` compares `(name, filename, lineno, id(self))`** +([`:596-603`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L596-L603)). +`name` leads, so any name carrying a position as text sorts `[10]` before `[1]` +however correct `lineno` is; the later terms only break ties among equal names. +This fails silently: every test passes, in the wrong order. Released gp-libs hits +it — `find()` calls `tests.sort()` over blocks named `page.md[k]`, so an +eleven-block page runs its eleventh block second. **`TestResults` carries `skipped` outside the tuple** ([`:114`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L114)), @@ -44,7 +47,7 @@ unpack in the ecosystem, including `doctest._test()` itself. ```text source string | DocTestParser.parse -> list[str | Example], alternating, - | reconstructing the input exactly + | covering the NORMALIZED input | DocTestParser.get_doctest -> DocTest v DocTestFinder.find(obj) -> list[DocTest] diff --git a/notes/analyses/12-pytest-xdist.md b/notes/analyses/12-pytest-xdist.md index 2f7cf74..7fc3852 100644 --- a/notes/analyses/12-pytest-xdist.md +++ b/notes/analyses/12-pytest-xdist.md @@ -71,9 +71,17 @@ cause. Worker restarts are on by default. | `pytest_xdist_node_collection_finished(node, ids)` | hook — observe the agreed id list | | `pytest_xdist_setupnodes(config, specs)` | hook — receives the already-expanded spec list; never raises | | `pytest_xdist_auto_num_workers(config)` | hook | -| `LoadScopeScheduling._split_scope(nodeid) -> str` ([`loadscope.py:284`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L284)) | subclass hook — **the only affinity primitive in the codebase** | +| `LoadScopeScheduling._split_scope(nodeid) -> str` ([`loadscope.py:284`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L284)) | subclass hook — the only affinity primitive *inside the shipped schedulers* | | `@pytest.mark.xdist_group(name)` | marker, honoured only under `--dist loadgroup` | +`pytest_xdist_make_scheduler` is the broader seam: a plugin may substitute an +entire `Scheduling` implementation, which is strictly more control than +`_split_scope` alone. That does not rescue shared state, because the substitution +happens controller-side and the controller only ever sees node-id strings — but +"the only affinity seam in the codebase" overstates it, and the honest claim is +narrower: *within the shipped schedulers*, `_split_scope` is the only affinity +primitive. + `_split_scope` is worth stating plainly: it is a pure function from a node-id string to a scope string, and both shipped grouping modes are two-line overrides of it — [`loadfile.py:35`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadfile.py#L35) diff --git a/notes/analyses/13-pytest-asyncio.md b/notes/analyses/13-pytest-asyncio.md index 3ba4400..fdb76ea 100644 --- a/notes/analyses/13-pytest-asyncio.md +++ b/notes/analyses/13-pytest-asyncio.md @@ -36,8 +36,9 @@ drift would occur: `_get_asyncio_mode` ([`:222-232`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L222-L232)) reads the CLI value, falls back to the ini value, and calls `Mode(val)` inside a `try`, translating a `ValueError` into a {exc}`pytest.UsageError` that lists the -valid modes. The lesson is not "no conversion" — it is that the conversion -happens **once**, in one named function, with a good error. +valid modes. It is *called* from several sites, so the value is not literally +resolved once per session — but the conversion and its error message live in one +named function, and that is the transferable part. Declaring its own `HookspecMarker("pytest")` namespace is the interesting one. A third party extends pytest-asyncio by implementing a hook, not by subclassing @@ -89,9 +90,9 @@ the second group is warned. `pytest_configure` does exactly that for an unset `asyncio_default_fixture_loop_scope` ([`:296-301`](https://github.com/pytest-dev/pytest-asyncio/blob/v1.4.0/pytest_asyncio/plugin.py#L296-L301)). -This project has the same problem coming: ADR 0001's vocabulary change renames -`namespace_scope` to `share`, and any future move of the block-vs-document default -needs the same mechanism. +This project has the same problem coming: ADR 0001 settles the vocabulary as +`ungrouped = "default" | "block"`, and any future move of that default needs the +same mechanism. **Normalize, then query once.** In `AUTO` mode the plugin literally adds the marker it would otherwise have to special-case, so downstream code has one @@ -101,8 +102,8 @@ class of bug. **Reuse the host's vocabulary rather than inventing a parallel one.** Loop scope uses pytest's own `function`/`class`/`module`/`package`/`session` ladder and its -scope names verbatim. It does not invent a third word for lifetime. Compare the -current `namespace_scope`/`namespace_items` naming, which collides with two pytest +scope names verbatim. It does not invent a third word for lifetime. Compare +PR #87's `namespace_scope`/`namespace_items`, which collides with two pytest concepts at once. **Session-wide errors raise {exc}`pytest.UsageError`.** An invalid `asyncio_mode` diff --git a/notes/analyses/16-docutils-myst.md b/notes/analyses/16-docutils-myst.md index f1b002c..8021966 100644 --- a/notes/analyses/16-docutils-myst.md +++ b/notes/analyses/16-docutils-myst.md @@ -98,9 +98,10 @@ plain ```` ```python ```` fence is only picked up by a prompt sniff, and the line-length guard and MyST transforms never run. Those omissions become a decision rather than an accident once the front-end owns its own configuration. -`myst_fence_as_directive` is also the answer to a real user request: it maps a -bare language fence onto a directive name, so a project that prefers not to write -`{testcode}` can still have its ```` ```python ```` blocks collected. +`myst_fence_as_directive` is narrower than it sounds. It runs the fence through +the directive of the **same name**, so a ```` ```python ```` fence looks for a +directive called `python`. It does not rename `python` to `testcode`; a project +wanting that must register a `python` directive or alias itself. ## Reporter behaviour diff --git a/notes/analyses/17-prior-art.md b/notes/analyses/17-prior-art.md index 4933274..ef55ecf 100644 --- a/notes/analyses/17-prior-art.md +++ b/notes/analyses/17-prior-art.md @@ -115,10 +115,12 @@ output is rendered back into the source file. - **Emitting the canonical form instead of parsing it collapses check-mode and update-mode into one code path.** That is a real structural insight. -- **Absolute byte offsets plus an invertible dedent scalar** are exactly what a - data model needs to rewrite source. Read-only tools discard the indent; keeping - it is the difference between "we could add `--update-examples` later" and "we - would have to redesign the data model first". Two int fields. +- **Absolute source offsets plus a recorded indent** are most of what a data model + needs to rewrite source. They are Python *string* indices, not byte offsets, and + a single indent scalar does not invert a dedent in general — a block whose lines + carry differing leading whitespace, or a tab, does not round-trip. Keeping the + offsets is still the difference between "we could add `--update-examples` later" + and "we would have to redesign the data model first". - **It composes with pytest by contributing no collector at all.** Examples are `parametrize` params, so marks, `-k`, fixtures and xfail all work unmodified. This is the cheapest correct integration in the entire survey. diff --git a/notes/analyses/20-data-structures.md b/notes/analyses/20-data-structures.md index dea809d..d8bdeb0 100644 --- a/notes/analyses/20-data-structures.md +++ b/notes/analyses/20-data-structures.md @@ -14,7 +14,7 @@ The disagreements are entirely about which of those four are the same object. | Sybil | `Region` | `Example` = one `Item` | `Document.namespace` | truthy return or exception | | xdoctest | `DoctestPart` | own `DocTest` | `global_namespace` | own report objects | | pytest-examples | `CodeExample` | the block, exec'd once | explicit `module_globals=` | captured output, or a rewrite | -| ADR 0001 | `Block` | `DocTest` per block | one `globs` per **group**, on the `Item` | stdlib's, per block | +| ADR 0001 | `ParsedBlock` | `DocTest` per block | one `globs` per **group**, on the `Item` | `BlockResult`/`GroupResult`, projected to stdlib's | Reading across the "shared-state unit" column against the "execution unit" column is the whole design problem. Sphinx and Sybil both put the shared state at a @@ -29,10 +29,10 @@ is six process-wide integers. ## Field-by-field: what a source unit carries -| Field | `Example` | `TestCode` | `Region` | `CodeExample` | `Block` (ADR 0001) | +| Field | `Example` | `TestCode` | `Region` | `CodeExample` | `ParsedBlock` (ADR 0001) | |---|---|---|---|---|---| | source text | `source` | `code` | via `lexemes` | `source` | `source` | -| expected output | `want` | paired separately | — | written, not read | `want` | +| expected output | `want` | paired separately | — | written, not read | a separate `ParsedOutput` | | line | `lineno` (0-based, string-relative) | `lineno` | computed from span | `start_line` | `line` (nullable) | | byte offsets | — | — | `start`, `end` | `start_index`, `end_index` | — (deferred) | | dedent scalar | `indent` | — | `Lexeme.offset` | `indent` | — (deferred) | @@ -41,20 +41,22 @@ is six process-wide integers. | options | `options` | `options` | — | — | `options` | | gate | — | `skipif` on the node | — | — | `skipif` (unevaluated) | | file | on the `DocTest` | `filename` | on the `Document` | `path` | `path` | -| compile mode | — | on the *builder*, mutable | — | always exec | on the `Example` subclass | +| compile mode | — | on the *builder*, mutable | — | always exec | on `ProjectedBlock` | Three observations. -**Only pytest-examples carries byte offsets and an invertible dedent.** Those two -fields are the entire difference between a read-only tool and one that can rewrite -expected output later. ADR 0001 defers them, which is a decision to be revisited -rather than a decision made. +**Only pytest-examples carries source offsets and a recorded dedent.** They are +Python string indices rather than byte offsets, and one indent scalar does not +generally invert a dedent — but carrying them at all is the difference between a +read-only tool and one that can rewrite expected output later. ADR 0001 defers +them, which is a decision to be revisited rather than a decision made. **Compile mode is on the wrong object everywhere except ADR 0001.** Sphinx keeps it as mutable builder state read through a process-global patch; stdlib hard-codes -it in the loop. Putting it on the example data is what lets it survive -`copy.copy`, merging and any reordering, without a flag some runner has to be -holding at the right moment. +it in the loop. ADR 0001 puts it on the projected *block*, which is where it +actually belongs — a block's execution policy is uniform across its examples, so +putting it on an `Example` subclass would both over-specify and drag the +compatibility kernel into carrying metadata. **Nobody but ADR 0001 makes the line nullable.** Every other system either always has a line (because it computed the span itself) or fabricates one. With a real diff --git a/notes/analyses/22-extension-seams.md b/notes/analyses/22-extension-seams.md index 05de168..57a1bc7 100644 --- a/notes/analyses/22-extension-seams.md +++ b/notes/analyses/22-extension-seams.md @@ -90,14 +90,14 @@ needs it. Applied to the candidates that came up: | `Frontend` protocol | **Yes.** Four implementations ship on day one (rST, MyST, text, Python docstrings) | | `ExecutionProfile` (private) | **Yes, private.** [PR #59](https://github.com/git-pull/gp-libs/pull/59) adds top-level `await` — a second execution policy a `Literal["single", "exec"]` cannot express. Stays private until an out-of-tree caller exists | | A per-example observer protocol | **No.** pytest gets failures through `report_*` and `MultipleDoctestFailures`; the CLI uses `summarize()`. No third consumer | -| A registry *object* with builders, digests and manifests | **No.** Two module-level dicts and two `register_*` functions do the same work | -| Entry-point plugin discovery | **No.** Not until a caller outside the package exists. It also breaks xdist's collection-purity requirement, since discovery is an import side effect rather than a function of (files, argv, ini) | - -That last row is worth keeping in mind generally: **any registry populated by -import side effects is in tension with -[`12-pytest-xdist.md`](12-pytest-xdist.md)'s requirement** that every worker -collect identical ids in identical order. A registry whose contents depend on -which conftest happened to be imported is not a pure function of the inputs. +| A registry *object* with builders, digests and manifests | **No object, but there is a freeze.** The contributed block kinds, front ends and profiles are resolved into an immutable mapping alongside the frozen `Settings` facets, before collection begins; registering after that is an error. That is what ADR 0001's collection contract means by *frozen registry* — not a class with its own lifecycle | +| Entry-point plugin discovery | **Not in v1**, for want of a caller outside the package — *not* because it breaks xdist. Workers are fresh processes that re-run plugin loading with the same argv, so entry-point resolution is identical everywhere. What breaks determinism is nondeterministic *contribution*, such as a conftest registering conditionally on the environment | + +The distinction those two rows turn on is worth stating once: **discovery is not +the hazard, nondeterminism is.** A registry populated identically in every process +satisfies [`12-pytest-xdist.md`](12-pytest-xdist.md)'s identical-collection +requirement no matter how it was populated; a registry whose contents depend on +which conftest happened to run, or on the environment, does not. ## Anchors From b23e1f2f7e3d041d9cff6b6bbe2f0f10ad4e8e62 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 18:08:40 -0500 Subject: [PATCH 18/31] docs(adrs[0001]): Make the plan a recipe and name the missing contracts why: A fourth review found the surrounding contracts did not compose. GroupPlan was called immutable while holding a mutable DocTest whose globs a run reassigns, so reruns would operate on last attempt's state. DocumentParser was said to subclass doctest.DocTestParser, whose parse(string, name) signature it cannot satisfy. And the record promised a pluggable core while declaring nothing public. what: - Hold ingredients, not a DocTest: ProjectedBlock is a recipe and RunContext materializes fresh stock Example and DocTest objects per attempt, so a plan can never carry run state - A gated testoutput makes its output ABSENT, not empty-with-options, which is what Sphinx does - Replace the nullable result record with a discriminated union, so a passed result carrying an exception is unrepresentable - Give exception precedence explicitly: control-flow, then body, then cleanup, with the loser recorded - Split ExecutionProfile (immutable factory) from ExecutionRuntime (per attempt, one per profile a group uses), so a group may mix prompt, exec and async blocks and an async runtime owns one loop - Run ordinary prompt blocks on CPython's untouched loop and reserve the owned __run for extended profiles, making the common lane compatible by construction - Give the parser three lanes, since DocumentParser cannot be a DocTestParser: strings, markup, and DocTestFinder-shaped objects - Define self.dtest as a synthetic zero-example group DocTest whose globs is the live mapping, and keep the darwin capture guard - Add the report-attribute channel, since the controller sees serialized reports rather than items - Give settings lifetimes (session, document, block) rather than one precedence sentence; move encoding to the loader and report style to the host; make wildcard and naming invariants, not knobs - Publish a contributor protocol while keeping the registry private, and record the conftest-timing and heterogeneous-worker hazards --- docs/adrs/0001-typed-vanilla-doctest-core.md | 224 ++++++++++++++----- 1 file changed, 165 insertions(+), 59 deletions(-) diff --git a/docs/adrs/0001-typed-vanilla-doctest-core.md b/docs/adrs/0001-typed-vanilla-doctest-core.md index 8c83a3e..7e5a67e 100644 --- a/docs/adrs/0001-typed-vanilla-doctest-core.md +++ b/docs/adrs/0001-typed-vanilla-doctest-core.md @@ -285,38 +285,49 @@ pytest_doctest_docutils collection, items, globs lifetime, reporting ### Settings and the frozen registry -Settings are **faceted**, not one object every layer imports — a single shared -`Settings` becomes a dependency magnet, and the facets have genuinely different -consumers: +Settings have **lifetimes**, not just precedence. "Resolve exactly once at session +start" cannot be true of document front matter, which does not exist until that +document is parsed. Three scopes: -| Facet | Owns | Read by | +| Scope | Owns | Resolved | |---|---|---| -| `ParseSettings` | encoding, MyST extensions, front-matter policy, diagnostic promotion and suppression | `markup/` | -| `ProjectionSettings` | `ungrouped` (`"default"` or `"block"`), wildcard resolution, name minting | `project` | -| `RunSettings` | optionflags, `continue_on_failure`, checker selection, report style | `runner` | - -Precedence is fixed and stated once: **per-block directive option → per-document -front matter → host configuration (pytest ini/CLI, Sphinx `conf.py`, argparse) → -built-in default.** Each adapter translates its host's configuration into the -facets exactly once, at session start; no layer reads a host config object. - -The **registry** is resolved and frozen alongside them, not as a separate object -with its own lifecycle. It holds the contributed block kinds, front ends and -execution profiles. "Frozen" means an immutable mapping built once before -collection begins; registering afterwards is an error, not a silent late -addition. - -This is what {ref}`the collection contract ` means by -*frozen registry*. Two consequences worth stating: - -- **Entry-point discovery is not inherently unsafe** under pytest-xdist. Workers - are fresh processes that re-run plugin loading with the same argv, so - entry-point resolution is identical in the controller and every worker. What - breaks determinism is *nondeterministic* contribution — a conftest that - registers conditionally on the environment — not discovery itself. -- **Nothing here is public in v1.** Contribution happens through the host's own - extension point (a pytest hook, Sphinx's `setup(app)`), and the registry type - stays private until a caller outside the package needs it. +| `SessionSettings` | defaults plus normalized host configuration; the frozen registry | once, at session start | +| `DocumentSettings` | the front-matter overlay a page is permitted to set | per document, after parsing | +| block / example policy | directive options, then inline `# doctest:` flags | per block, at projection and run | + +Not every field shares one ladder, so the precedence is stated per axis. For +option flags it follows Sphinx: **runner defaults → directive or output +`:options:` → inline flags.** + +Two fields move out of the core entirely. **Encoding** belongs to the source +loader, because `DocumentParser` already receives `str`. **Report style** belongs +to the host adapter. And wildcard resolution and name minting are *invariants*, +not user-configurable knobs — exposing them would let a project produce node ids +no other project can read. + +The **registry** is resolved and frozen alongside `SessionSettings`. "Frozen" +means an immutable mapping built once; registering afterwards is an error. + +**Contribution is public; the registry is not.** The stated goal is an +extendable, pluggable core, so a small host-neutral contributor protocol ships in +v1 — what may be contributed (front ends, block kinds, execution profiles, +checkers), when registration closes, how duplicate names are resolved, and what +ordering guarantees exist. Its *implementation* stays private. pytest hooks, +Sphinx's `setup(app)`, direct library calls and — optionally — entry points all +feed that one surface. + +Two hazards to name rather than assume away: + +- **Nested `conftest.py` files load during collection**, so "freeze before + collection" is too simple. Either the boundary moves later, or conftest-level + contribution is explicitly excluded. That is an open question in + {doc}`0006-pytest-private-api-compatibility`. +- **Entry-point discovery is not identical across workers in general.** xdist + supports SSH and socket workers and heterogeneous environments, so "same argv, + same plugins" holds only for a homogeneous local run. Either state a + homogeneous-plugin requirement or compare a deterministic registry manifest + across workers. What breaks determinism is *nondeterministic contribution*, not + discovery as such — but heterogeneity makes that distinction load-bearing. ### Item lifecycle @@ -324,6 +335,14 @@ The custom item is load-bearing, and half-reusing {class}`pytest.DoctestItem` reintroduces the exact bug this design exists to avoid. The contract, stated so an implementer cannot get it wrong by omission: +0. **The carrier.** {class}`pytest.DoctestItem` reads `self.dtest` in + `setup()`, `reportinfo()` and `_check_all_skipped()`, so the subclass must + define it even though a group holds many tests. `self.dtest` is a synthetic + **zero-example** `DocTest` for the group, and its `globs` **is** the canonical + live mapping — the same object every `RunnableBlock`'s test is given. That + makes the inherited `setup()` inject fixtures into exactly the right place + with no override of the injection itself. + 1. **Collection** builds one `GroupPlan` per (document, group) and one item per plan. An empty plan yields no item. 2. **`setup()`** starts an attempt. In order: clear the live mapping **in place**; @@ -348,8 +367,21 @@ an implementer cannot get it wrong by omission: `PytestDoctestRunner` is nested inside a factory and cannot be imported. 4. **Outcome** follows [](#the-outcome-contract): skip the item only when every runnable block is skipped. -5. **`repr_failure`** is inherited unchanged. It already reads each failure's own - `DocTest`. + `runtest()` must also keep `_disable_output_capturing_for_darwin()`, which + the inherited implementation calls before running and which has nothing to do + with grouping. + +5. **`repr_failure`** is inherited unchanged for *failures* — it already reads + each failure's own `DocTest`. It does **not** render `GroupResult`, and + claiming it does would conflate two channels. Partial-skip detail goes to a + report section and the terminal summary; see [](#the-outcome-contract). + +6. **Reporting across processes.** The controller never sees the item — it + receives serialized `TestReport` dictionaries. So `pytest_runtest_makereport` + copies a **versioned, JSON-safe** block summary onto the report; pytest + serializes arbitrary report attributes and xdist reconstructs them + controller-side. The rich `GroupResult`, and every exception in it, stays + worker-local. ### Vocabulary @@ -424,20 +456,33 @@ class ExpectedOutput(t.NamedTuple): class ProjectedBlock(t.NamedTuple): + """A RECIPE. Holds no `DocTest`, because a `DocTest` is mutable.""" + phase: Phase - test: doctest.DocTest # stock objects; its own filename/lineno/docstring - profile: ExecutionProfile # uniform for the block, resolved at projection + name: str # the minted test name + source: str # normalized executable body + filename: str + lineno: int | None + profile_name: str # resolved against the frozen registry per attempt skipif: str | None # UNEVALUATED; gated in run_group() expected: ExpectedOutput | None # paired testoutput, itself gateable class GroupPlan(t.NamedTuple): group: str - blocks: tuple[ProjectedBlock, ...] # in phase order; IMMUTABLE + blocks: tuple[ProjectedBlock, ...] # in phase order; genuinely immutable seed: t.Mapping[str, t.Any] # initial namespace; copied per attempt -# --- run: one attempt ------------------------------------------------------ +# --- run: one attempt materializes fresh stdlib objects -------------------- + + +class RunnableBlock(t.NamedTuple): + """Materialized per attempt. Its `DocTest` is never retained by a plan.""" + + recipe: ProjectedBlock + test: doctest.DocTest # fresh stock objects, built this attempt + runtime: ExecutionRuntime # from the profile factory, this attempt class RunContext: @@ -445,21 +490,39 @@ class RunContext: plan: GroupPlan globs: dict[str, t.Any] # cleared in place and reseeded per attempt + runtimes: dict[str, ExecutionRuntime] # one per profile used by the group + + +# --- results: a discriminated union, so invalid states cannot be built ----- + + +class Passed(t.NamedTuple): + block: ProjectedBlock + + +class Failed(t.NamedTuple): + block: ProjectedBlock + failure: doctest.DocTestFailure | doctest.UnexpectedException + + +class Skipped(t.NamedTuple): + block: ProjectedBlock + reason: str # the gate expression -class BlockResult(t.NamedTuple): +class Errored(t.NamedTuple): block: ProjectedBlock - outcome: t.Literal["passed", "failed", "skipped", "error"] - reason: str | None # gate expression when skipped; message when error - error: BaseException | None # gate or profile failure, not a doctest one + error: BaseException # a gate that raised, or a runtime that would not start + + +BlockResult: t.TypeAlias = Passed | Failed | Skipped | Errored class GroupResult(t.NamedTuple): group: str blocks: tuple[BlockResult, ...] - failures: tuple[doctest.DocTestFailure | doctest.UnexpectedException, ...] primary: BaseException | None # what runtest() re-raises - secondary: tuple[BaseException, ...] # e.g. a cleanup failure after a body failure + secondary: tuple[BaseException, ...] # e.g. cleanup failing after the body ``` `ParsedBlock` carries no `want`, because neither owner of a `want` is the parsed @@ -471,13 +534,27 @@ was what made "projection owns pairing" untrue. `BlockKind` names a profile rather than holding one, so a public type never contains a private one. The name resolves against the frozen registry. -`BlockResult.outcome` needs `"error"` because a gate that raises, or a profile -that fails to establish, is neither a pass, a fail, nor a skip. -`GroupResult.primary` and `.secondary` make the precedence explicit: **a body -failure outranks a cleanup failure**, a control-flow exception -({exc}`pytest.skip`, `xfail`, exit) outranks both, and the loser is recorded -rather than dropped. Classifying which exceptions are control-flow is the pytest -adapter's job, not the core's. +**A plan holds no `DocTest`.** {class}`doctest.DocTest` is mutable — the design +assigns `globs` to it after construction, and a run mutates that mapping — so a +plan retaining one would not be a recipe, it would be last attempt's state. Under +`--reruns` that is the false-green this design exists to prevent. `ProjectedBlock` +therefore carries the *ingredients*, and `RunContext` materializes fresh stock +`Example` and `DocTest` objects for every attempt. + +The same applies to a gated `testoutput`: when its gate is truthy the output is +**absent**, not empty. Its text *and* its output-specific options both disappear, +which is what Sphinx does, and which a pre-built `want=""` with retained options +would get wrong. + +**Results are a discriminated union**, not one record with nullable fields, so +"passed with an exception attached" is unrepresentable rather than merely +unlikely. `Errored` exists because a gate that raises, or a runtime that will not +start, is none of pass, fail or skip. + +`GroupResult.primary` and `.secondary` make precedence explicit: **a control-flow +exception outranks a body failure, which outranks a cleanup failure**, and the +loser is recorded rather than dropped. Classifying which exceptions are +control-flow is the pytest adapter's job, not the core's. `Block.line` being nullable is load-bearing, not defensive. A bare `>>>` block nested in a `.. note::`, a list item or a block quote reports `line=None, @@ -510,16 +587,27 @@ evaluates it against the live group mapping, after fixture injection and after earlier blocks have run. That is more useful — a gate can consult a fixture — and it is not what `sphinx-build` does. Recorded rather than hidden. -`ExecutionProfile` is a **private** protocol carrying compile mode, extra compile -flags, and an optional per-group context manager. It is not a `Literal["single", -"exec"]`, because a second execution policy already exists in this repository: -[PR #59](https://github.com/git-pull/gp-libs/pull/59) adds top-level `await`, -which needs `ast.PyCF_ALLOW_TOP_LEVEL_AWAIT` and an event-loop lifetime that a -mode string cannot express. The profile's context manager is what `run_group()` -enters, so that lifetime is served without overriding `run()` and stdlib's -save-and-restore `finally` stays inherited. It stays private until a caller -outside the package needs it; PR #59 is the in-repo caller that justifies it -existing at all. +**`ExecutionProfile` is an immutable factory; `ExecutionRuntime` is per attempt.** +A group can mix prompt, `exec` and async blocks, so there is no single per-group +profile. The profile is chosen per block and names a factory; `RunContext` creates +one runtime *per distinct profile* the group uses, and blocks sharing a profile +share its runtime. An async runtime therefore owns one event loop for the whole +group, which is what lets awaited state cross block boundaries. + +It is not a `Literal["single", "exec"]`, because a second execution policy already +exists in this repository: [PR #59](https://github.com/git-pull/gp-libs/pull/59) +adds top-level `await`, which needs `ast.PyCF_ALLOW_TOP_LEVEL_AWAIT` and an +event-loop lifetime a mode string cannot express. The runtime's context manager is +what `run_group()` enters, so that lifetime is served without overriding `run()` +and stdlib's save-and-restore `finally` stays inherited. + +**The ordinary lane does not use the owned loop at all.** For prompt-form blocks — +the overwhelming majority — the runner is a plain reporter subclass over CPython's +*untouched* per-example loop. `_DocTestRunner__run` is invoked only for extended +profiles: `exec` bodies, top-level await, and whatever comes next. Ordinary +doctests are then compatible **by construction** rather than by differential +testing, and {doc}`0002-runner-conformance-across-cpython`'s harness shrinks to +guarding the extended lane. **Which docutils node classes a kind may arrive as is a front-end concern, not a `BlockKind` field.** `testsetup`, `testcleanup` and any `:hide:` block are @@ -561,6 +649,24 @@ layer that never changes what is constructed. Standalone reST and MyST use both halves; a Sphinx extension calls only the extractor, on the doctree it already resolved. + **`DocumentParser` is not a {class}`doctest.DocTestParser`.** The two + signatures are incompatible — stdlib's is `parse(self, string, name='')` + returning alternating `str` and `Example`, and `get_doctest` depends on exactly + that + ([`Lib/doctest.py:657`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L657), + [`:696`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L696)). + So there are three lanes, not one: + + | Lane | Contract | + |---|---| + | plain text and strings | the exact `DocTestParser` contract, unmodified | + | reST / MyST | `DocumentParser` → doctree → `extract_blocks` | + | Python objects | a `DocTestFinder`-shaped adapter | + + Anything promising `DocFileSuite(parser=...)` compatibility is a separate + stdlib-shaped façade over the first lane, not the markup lane wearing a + stdlib name. + This is not a Sphinx builder. A builder owns discovery, an `env`, an `outdir` and a reporting format; an extractor is a pure function from a doctree to blocks, and {doc}`0001-typed-vanilla-doctest-core`'s rejection of a builder From e042bad603e461443f824c03d6e79d2eef4f6264 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 18:10:32 -0500 Subject: [PATCH 19/31] docs,notes: Narrow 0002, reframe 0005, and fix a fourth round of drift MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit why: ADR 0001 now runs ordinary prompt blocks on CPython's untouched loop, which shrinks what 0002 must prove and makes 0005 orthogonal to the architecture. Review also found the notes still generalized Sphinx's per-block execution and conflated xdist's two channels. what: - 0002: scope the harness to the extended lane; ordinary blocks are the reference rather than something to differentially prove - 0005: state plainly that the floor is support policy, not core architecture, and that no part of 0001 depends on the answer - 0005: correct the Sphinx version — the group fallback for a bare, unstamped doctest_block changed in 9.0, not 9.1, and directives always stamp groups so unargumented directives are unaffected - Notes: Sphinx is per-block for its TEST phase only; setup blocks are combined into one simulated DocTest and cleanup into another - Notes: separate xdist's scheduling channel from its reporting channel — the controller sees node-id strings when scheduling and serialized reports afterwards, and reports carry arbitrary extra attributes - Notes: a nullable line is not unique to this design; Sphinx's get_line_number returns None too. What is new is per-block propagation - Notes: narrow the affinity claim to the shipped schedulers, since pytest_xdist_make_scheduler substitutes a whole Scheduling - Notes: PR #87's per-block mode is proposed, not shipped - Notes: carry the block/runtime split into the data-flow diagram --- .../0002-runner-conformance-across-cpython.md | 7 ++++ .../0005-line-recovery-for-nested-blocks.md | 15 +++++--- notes/analyses/00-taxonomy.md | 12 +++--- notes/analyses/12-pytest-xdist.md | 18 ++++++--- notes/analyses/20-data-structures.md | 11 +++--- notes/analyses/21-data-flows.md | 5 ++- .../23-namespace-scope-and-test-identity.md | 38 ++++++++++--------- 7 files changed, 67 insertions(+), 39 deletions(-) diff --git a/docs/adrs/0002-runner-conformance-across-cpython.md b/docs/adrs/0002-runner-conformance-across-cpython.md index 90bc45c..afd1889 100644 --- a/docs/adrs/0002-runner-conformance-across-cpython.md +++ b/docs/adrs/0002-runner-conformance-across-cpython.md @@ -48,6 +48,13 @@ continuously, without a `sys.version_info` ladder? A differential conformance harness, run in CI on every supported interpreter, gating the build step that lands the runner. +**Scoped to the extended lane.** {doc}`0001-typed-vanilla-doctest-core` runs +ordinary prompt blocks on CPython's untouched per-example loop, so those need no +differential proof — they *are* the reference. The owned `__run` is invoked only +for `exec` bodies, top-level await and future profiles, and that is what this +harness guards. It is a smaller obligation than an unconditionally owned loop, +and it is the reason owning the loop is affordable at all. + A fixed case matrix — pass, fail, unexpected exception, `SyntaxError`, all examples skipped, partially skipped, `FAIL_FAST`, `REPORT_ONLY_FIRST_FAILURE`, `IGNORE_EXCEPTION_DETAIL`, and an exec-mode body — is run through both this diff --git a/docs/adrs/0005-line-recovery-for-nested-blocks.md b/docs/adrs/0005-line-recovery-for-nested-blocks.md index 99cd1f8..7a43251 100644 --- a/docs/adrs/0005-line-recovery-for-nested-blocks.md +++ b/docs/adrs/0005-line-recovery-for-nested-blocks.md @@ -46,8 +46,11 @@ mutation" is wrong. ## Direction -**Raise the docutils floor instead** — but that is a support-matrix decision, not -a pin change, and it is not this record's to make alone. +**Raise the docutils floor instead** — but that is **support policy, not core +architecture**. Nothing in {doc}`0001-typed-vanilla-doctest-core` depends on the +answer: a nullable line is the honest representation either way, and the floor +only decides how often it is `None`. This record can stay open indefinitely +without blocking the design. docutils 0.22 fixed the underlying defect upstream: a nested block reports a real line, and top-level and nested blocks agree on reporting the **first** line @@ -104,9 +107,11 @@ statement. documenting two behaviours for the same page. - Sequencing with the `gp-sphinx` cap. That release has to land first, and this repository does not control it. -- Whether the Sphinx move belongs in this record or its own. Sphinx 9.1 changes - the default group an unargumented block joins, which is a semantics change - beyond line numbers. +- Whether the Sphinx move belongs in this record or its own. Sphinx **9.0** + changed the fallback group for a bare, unstamped `doctest_block` from + `['default']` to `[doctest_test_doctest_blocks]`; directives always stamp + `groups`, so unargumented *directives* are unaffected. 9.x also differs in + fail-fast and result propagation. All of that is semantics beyond line numbers. - Whether tests should pin exact `(path, line)` for a bare block nested in a `.. note::`, a list item, a block quote and a `{tab}` directive. They should — they are the regression net for the floor, and this repository already has diff --git a/notes/analyses/00-taxonomy.md b/notes/analyses/00-taxonomy.md index dbc6d12..233184a 100644 --- a/notes/analyses/00-taxonomy.md +++ b/notes/analyses/00-taxonomy.md @@ -24,7 +24,7 @@ a different philosophy. |---|---|---|---|---| | CPython `doctest` | same (one `DocTest` per docstring) | dotted symbol path | *is* the model | none — line regex over a string | | `_pytest.doctest` | same (one item per `DocTest`) | `path::module.qualname` | stdlib, unchanged | none — delegates | -| `sphinx.ext.doctest` | same (one group, one runner pass) | group name, shared by every block | stdlib, constructed per group | real doctree | +| `sphinx.ext.doctest` | same (one group, no selectable unit) | group name, shared by every test block | stdlib; per test block, combined for setup and cleanup | real doctree | | Sybil | **different, unacknowledged** | positional `line:N,column:N` | stdlib `Example`, one-line `DocTest` fork | flat character spans | | xdoctest | same (one `DocTest` per docstring) | `Callname:N` | own, with a late bridge back | none for `.rst`/`.txt` | | pytest-examples | none — no implicit sharing | positional `path:start-end` | none | regex over fences | @@ -57,16 +57,18 @@ product space is four cells: | | one node id | N node ids | |---|---|---| | **one `DocTest`** | PR #87's `merged` | — (incoherent) | -| **N `DocTest`s** | `sphinx.ext.doctest`, but with *no* ids; ADR 0001 adds the pytest identity | Sybil, PR #87's `per-block`, released `doctest_docutils` (no sharing) | +| **N `DocTest`s** | `sphinx.ext.doctest` for its *test* phase only, and with *no* ids; ADR 0001 adds the pytest identity | Sybil, PR #87's `per-block`, released `doctest_docutils` (no sharing) | The bottom-right cell is where the silent failure lives — but only when the blocks share state. Released `doctest_docutils` sits there safely because its blocks share nothing: each gets its own copied `globs`. The bottom-left cell is where the design goes. Sphinx already *executes* that -shape — N `DocTest`s over one group namespace — but produces no addressable unit -for any of them, since every block shares one `DocTest.name`. Giving that shape a -pytest identity is the contribution; the execution shape is not new. +shape — but only for ordinary test blocks: all of a group's setup blocks are +combined into one simulated `DocTest`, and likewise cleanup. It also produces no +addressable unit for any of them, since every test block shares one +`DocTest.name`. So the design is per-block in three phases where Sphinx is +per-block in one, and the pytest identity is new either way. **Axis 3 has an empirical answer.** xdoctest is the controlled experiment for abandoning the stdlib object model, and it is now building the bridge back. The diff --git a/notes/analyses/12-pytest-xdist.md b/notes/analyses/12-pytest-xdist.md index 7fc3852..40e82ea 100644 --- a/notes/analyses/12-pytest-xdist.md +++ b/notes/analyses/12-pytest-xdist.md @@ -20,11 +20,19 @@ controller worker (one per process) collection: list[str] <- the agreed id list ``` -The controller's entire model of the suite is a list of node-id strings that -arrived from a worker. It has no items, no marks, no fixtures and no knowledge of -what any test does. A plugin that needs "these tests share state" therefore cannot -tell the controller so directly — it can only encode the fact *into the node id* -or infer it from string shape. +The controller's entire model of the suite **during scheduling** is a list of +node-id strings that arrived from a worker. It has no items, no marks, no fixtures +and no knowledge of what any test does. A plugin that needs "these tests share +state" therefore cannot tell the controller so directly — it can only encode the +fact *into the node id* or infer it from string shape. + +**Reporting is a separate channel with a different shape.** After execution the +controller receives serialized `TestReport` dictionaries, and pytest serializes +arbitrary extra attributes on a report, which xdist reconstructs controller-side. +So a worker *can* ship structured per-block detail to the controller — as +JSON-safe data on the report, never as an object hanging off the item. Confusing +the two channels is what makes a controller-side summary look impossible when it +is not. ## Data flow diff --git a/notes/analyses/20-data-structures.md b/notes/analyses/20-data-structures.md index d8bdeb0..3aae0e9 100644 --- a/notes/analyses/20-data-structures.md +++ b/notes/analyses/20-data-structures.md @@ -58,11 +58,12 @@ actually belongs — a block's execution policy is uniform across its examples, putting it on an `Example` subclass would both over-specify and drag the compatibility kernel into carrying metadata. -**Nobody but ADR 0001 makes the line nullable.** Every other system either always -has a line (because it computed the span itself) or fabricates one. With a real -doctree there are constructs that genuinely have no recoverable line, and pytest -has a branch for exactly that — `EXAMPLE LOCATION UNKNOWN` — which is unreachable -unless the model can express it. +**A nullable line is not unique to ADR 0001.** Sphinx's own `get_line_number` +returns `None` — its docstring says "get the real line number or admit we don't +know" — for a block whose source is a stripped docstring. What ADR 0001 adds is +not the nullability but the *propagation*: `lineno=None` reaches pytest's +`EXAMPLE LOCATION UNKNOWN` branch per block, without a sibling's known line +masking it. ## What a result unit carries diff --git a/notes/analyses/21-data-flows.md b/notes/analyses/21-data-flows.md index f6b1f40..e2ff665 100644 --- a/notes/analyses/21-data-flows.md +++ b/notes/analyses/21-data-flows.md @@ -64,8 +64,9 @@ two. **At the compile call.** stdlib hard-codes `"single"`. Sphinx flips a mutable builder attribute and reads it through a process-global rebinding of `doctest.compile` that is never restored. PR #87 proposes cloning the -mangled loop's code object to get a private version of that rebinding. ADR 0001 -carries the mode on the example data and reads it in a loop it owns — the only one +mangled loop's code object to get a private version of that rebinding. ADR 0001 puts the policy on the projected *block*, materializes a runtime per +profile, and reads it in a loop it owns — and only for extended profiles, since +ordinary prompt blocks run on CPython's untouched loop — the only one of the four that neither mutates process state nor copies a code object. **At the location.** stdlib computes `test.lineno + example.lineno + 1`. Sphinx diff --git a/notes/analyses/23-namespace-scope-and-test-identity.md b/notes/analyses/23-namespace-scope-and-test-identity.md index e284608..e457d20 100644 --- a/notes/analyses/23-namespace-scope-and-test-identity.md +++ b/notes/analyses/23-namespace-scope-and-test-identity.md @@ -22,11 +22,13 @@ narrative page needs depends on the first. | **one `DocTest`** | PR #87's `merged` | incoherent | | **N `DocTest`s** | `sphinx.ext.doctest`, but with *no* ids at all; ADR 0001 gives the shape a pytest identity | Sybil; PR #87's `per-block` | -Sphinx belongs in the bottom row: it already builds one `DocTest` per block -against one shared group namespace. Its "one node id" is really *no* id — every -block in a group shares one `DocTest.name`, which is why `SphinxDocTestRunner` -overrides a private stdlib method to swallow the resulting `IndexError`. So the -execution shape is well-trodden; the contribution is making it addressable. +Sphinx belongs in the bottom row, with a qualification: it builds one `DocTest` +per *ordinary test* block against one shared group namespace, while combining all +setup blocks into one simulated `DocTest` and all cleanup into another. Its "one +node id" is really *no* id — every test block in a group shares one +`DocTest.name`, which is why `SphinxDocTestRunner` overrides a private stdlib +method to swallow the resulting `IndexError`. So the execution shape is partly +well-trodden; the contribution is making it addressable. The bottom-right cell is where the silent failure lives, and two shipped projects are in it. @@ -35,9 +37,9 @@ Sybil is there **unacknowledged**: one `Document.namespace` shared by reference, one pytest item per region. `pytest -k` on an example whose predecessor bound a name raises `NameError`, and nothing in its documentation says so. -`doctest_docutils`'s `per-block` mode is there **acknowledged and guarded** — the -guards are the xdist scheduler substitution, the scheduler refusal, and the -run-twice refusal. Those guards are the reason `_worker_count`, `_shared_page`, +PR #87's proposed `per-block` mode would sit there **acknowledged and guarded** — +the guards being an xdist scheduler substitution, a scheduler refusal and a +run-twice refusal. It has not shipped; {doc}`ADR 0003 <../../docs/adrs/0003-rejecting-per-block-items>` rejects the shape. Those guards are the reason `_worker_count`, `_shared_page`, `_is_page` and `_splitting_scheduler` exist at all. The bottom-left cell gives per-block reporting *and* an unsplittable sharing unit, @@ -53,10 +55,12 @@ structural, not incidental: - The controller never collects, so it cannot ask "which items share state" — it sees node-id strings and nothing else. Any protection must be *inferred from string shape* or *encoded into the id*. -- The only affinity primitive is +- The only affinity primitive *inside the shipped schedulers* is [`_split_scope(nodeid) -> str`](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/scheduler/loadscope.py#L284). - `load` and `worksteal` have no scope concept at any layer, so under a user-typed - `--dist load` the options are refuse, substitute, or do not need protecting. + A plugin may substitute a whole `Scheduling` via `pytest_xdist_make_scheduler`, + but that still reasons only over node-id strings. `load` and `worksteal` have no + scope concept at any layer, so under a user-typed `--dist load` the options are + refuse, substitute, or do not need protecting. - `xdist_group` is applied [worker-side](https://github.com/pytest-dev/pytest-xdist/blob/v3.8.0/src/xdist/remote.py#L245-L254) and only when the worker's own `--dist` is literally `loadgroup`. A @@ -76,12 +80,7 @@ Sybil's [`line:{line},column:{column}`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/sybil.py#L155-L157) and pytest-examples' `path:start-end`. -An **ordinal among the extracted blocks** is a different thing and is fine. It is -not a source coordinate: `page.md[3]` is unchanged by a paragraph inserted above -it, which is exactly the edit that renames a `line:N` id. Released -`doctest_docutils` names its tests that way already, and the rule preserves it. - -For a *documentation* test runner this is indefensible, because prose above +For a *documentation* test runner that is indefensible, because prose above examples is the thing that changes most often. Adding a sentence renames every downstream test, which breaks `--lf`, `--nf`, checked-in deselect files, xfail lists and CI flake history. pytest-examples compounds it: the same string is the @@ -91,6 +90,11 @@ The rule: **author-declared name first, stable ordinal as fallback, and the fallback shape invariant across configuration.** The test to apply is concrete — adding a sentence to a page must rename zero tests. +An **ordinal among the extracted blocks** is not a source coordinate and is fine: +`page.md[3]` is unchanged by a paragraph inserted above it, which is exactly the +edit that renames a `line:N` id. Released `doctest_docutils` names its tests that +way already, and the rule preserves it. + Two corollaries: - `DocTest.name` must be machine-independent. Embedding an absolute path makes a From f4f8d4dc1648323ea22b11a111f3af8527ba4d60 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 19:33:06 -0500 Subject: [PATCH 20/31] docs(adrs[0001]): Make the recipe reconstructible and the results honest MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit why: Making the plan a recipe last round dropped information the recipe must carry. One prompt block yields SEVERAL doctest.Example objects — three for a block ending in a traceback — each with its own source, want, exc_msg, lineno, indent and options, and pytest's failure renderer needs the docstring besides. A single source string and one lineno cannot rebuild that. The result types lost outcomes for the same reason: continue_on_failure produces several failures from one block, and a PASSING block can report attempted=2 skipped=1. what: - Add ExampleRecipe and carry examples plus docstring on ProjectedBlock, so materialization reproduces get_doctest exactly rather than approximating it. Pinned by a doctest asserting the three-example case - Make Failed.failures plural, give Passed and Failed counts, and type SkipReason, since a skip may come from :skipif:, an inline flag, :pyversion: or a profile declining - Replace the single precedence ladder with a phase-aware table: process aborts always win; a cleanup failure — including a pytest.skip raised in cleanup — never outranks a body failure - Enter profile runtimes through an ExitStack so partial startup unwinds in reverse - Rename position to source_ordinal and define it as the ordinal among runnable candidates before gating, filtering and group expansion, so inserting prose renames nothing - Call GroupPlan structurally immutable and say the seed is shallow-copied per attempt, matching doctest's namespace semantics - Retire the last PlannedBlock and plan-held-DocTest language, and restate wildcard membership as per-group materialization --- docs/adrs/0001-typed-vanilla-doctest-core.md | 129 +++++++++++++++---- 1 file changed, 105 insertions(+), 24 deletions(-) diff --git a/docs/adrs/0001-typed-vanilla-doctest-core.md b/docs/adrs/0001-typed-vanilla-doctest-core.md index 7e5a67e..633e4c5 100644 --- a/docs/adrs/0001-typed-vanilla-doctest-core.md +++ b/docs/adrs/0001-typed-vanilla-doctest-core.md @@ -350,12 +350,14 @@ an implementer cannot get it wrong by omission: `super().setup()` so fixtures inject into that same object. Clearing in place rather than rebinding is what keeps `item.globs is run.globs` true for every block, and what stops attempt two of a `--reruns` run from reading attempt - one's mutations. Every block's `DocTest.globs` is assigned this object *after* - construction, because `DocTest.__init__` copies. + one's mutations. Each `RunnableBlock` is materialized against this object — + its `DocTest.globs` assigned *after* construction, because `DocTest.__init__` + copies. 3. **`runtest()`** is overridden. It must not delegate to `DoctestItem.runtest`, which runs a single `dtest` with `clear_globs` defaulting to `True` — that would empty the shared mapping after the first - block. It calls `run_group()`, which runs each `PlannedBlock` in phase order + block. It calls `run_group()`, which materializes and runs each + `RunnableBlock` in phase order with `clear_globs=False`, evaluates `:skipif:` against the live mapping, finalizes each paired `want` from its gated `ExpectedOutput`, and wraps the body in a `try`/`finally` so cleanup runs whether or not the body raised. When @@ -419,7 +421,8 @@ class ParsedBlock(t.NamedTuple): source: str # dedented body, verbatim author text path: pathlib.Path # the file the text lives in, not the collected document line: int | None # None when docutils could not recover one - position: int # 0-based document pre-order index; the naming key + source_ordinal: int # ordinal among runnable candidates, BEFORE gating, + # filtering and group expansion; the naming key groups: tuple[str, ...] # declared verbatim; () and ("*",) unresolved here options: t.Mapping[int, bool] # plain int keys, exactly as doctest produces skipif: str | None # UNEVALUATED @@ -432,7 +435,7 @@ class ParsedOutput(t.NamedTuple): text: str path: pathlib.Path line: int | None - position: int + source_ordinal: int groups: tuple[str, ...] options: t.Mapping[int, bool] skipif: str | None # a gated output means its testcode expects nothing @@ -455,14 +458,27 @@ class ExpectedOutput(t.NamedTuple): skipif: str | None # when truthy at run time, `want` becomes "" +class ExampleRecipe(t.NamedTuple): + """Everything needed to rebuild one stock `doctest.Example`.""" + + source: str + want: str + exc_msg: str | None + lineno: int # 0-based, relative to the block's docstring + indent: int + options: t.Mapping[int, bool] + + class ProjectedBlock(t.NamedTuple): """A RECIPE. Holds no `DocTest`, because a `DocTest` is mutable.""" phase: Phase name: str # the minted test name - source: str # normalized executable body + examples: tuple[ExampleRecipe, ...] # a prompt block yields SEVERAL + docstring: str # what pytest's failure renderer slices filename: str - lineno: int | None + lineno: int | None # the block's own line; examples are relative to it + options: t.Mapping[int, bool] # block-level directive :options: profile_name: str # resolved against the frozen registry per attempt skipif: str | None # UNEVALUATED; gated in run_group() expected: ExpectedOutput | None # paired testoutput, itself gateable @@ -470,7 +486,7 @@ class ProjectedBlock(t.NamedTuple): class GroupPlan(t.NamedTuple): group: str - blocks: tuple[ProjectedBlock, ...] # in phase order; genuinely immutable + blocks: tuple[ProjectedBlock, ...] # in phase order; STRUCTURALLY immutable seed: t.Mapping[str, t.Any] # initial namespace; copied per attempt @@ -496,18 +512,31 @@ class RunContext: # --- results: a discriminated union, so invalid states cannot be built ----- +class Counts(t.NamedTuple): + attempted: int + skipped: int # a PASSING block can still carry skipped examples + + +class SkipReason(t.NamedTuple): + kind: t.Literal["skipif", "inline-flag", "pyversion", "profile"] + detail: str # the gate expression, the flag, the specifier + + class Passed(t.NamedTuple): block: ProjectedBlock + counts: Counts class Failed(t.NamedTuple): block: ProjectedBlock - failure: doctest.DocTestFailure | doctest.UnexpectedException + counts: Counts + # PLURAL: continue_on_failure yields several from one block + failures: tuple[doctest.DocTestFailure | doctest.UnexpectedException, ...] class Skipped(t.NamedTuple): block: ProjectedBlock - reason: str # the gate expression + reason: SkipReason class Errored(t.NamedTuple): @@ -541,20 +570,70 @@ plan retaining one would not be a recipe, it would be last attempt's state. Unde therefore carries the *ingredients*, and `RunContext` materializes fresh stock `Example` and `DocTest` objects for every attempt. +**The ingredients are per example, not per block.** One prompt block routinely +yields several {class}`doctest.Example` objects, each with its own `source`, +`want`, `exc_msg`, `lineno`, `indent` and `options` — three, for a block whose +last statement raises: + +```{doctest} +>>> import doctest +>>> src = ">>> x = 1\n>>> x + 1\n2\n>>> int('z')\nTraceback (most recent call last):\nValueError: bad\n" +>>> test = doctest.DocTestParser().get_doctest(src, {}, "blk", "p.md", 0) +>>> len(test.examples) +3 +>>> [(e.lineno, e.want.strip()) for e in test.examples] +[(0, ''), (1, '2'), (3, 'Traceback (most recent call last):...')] +``` + +A single `source` and one `lineno` cannot represent that, and `docstring` is +separately required by pytest's known-location failure renderer. Hence +`ExampleRecipe` and `ProjectedBlock.docstring`: the recipe reproduces exactly what +{meth}`doctest.DocTestParser.get_doctest` produced, rather than approximating it. + The same applies to a gated `testoutput`: when its gate is truthy the output is **absent**, not empty. Its text *and* its output-specific options both disappear, which is what Sphinx does, and which a pre-built `want=""` with retained options would get wrong. +`GroupPlan` is **structurally** immutable, not deeply so. Its tuples cannot be +rebound, but `seed` is a `Mapping` whose *values* are arbitrary user objects. Each +attempt shallow-copies it into the live mapping, which is exactly what +`DocTest.__init__` does with `globs` — matching doctest's own namespace semantics +rather than inventing a deeper guarantee the ecosystem does not provide. + **Results are a discriminated union**, not one record with nullable fields, so "passed with an exception attached" is unrepresentable rather than merely unlikely. `Errored` exists because a gate that raises, or a runtime that will not start, is none of pass, fail or skip. -`GroupResult.primary` and `.secondary` make precedence explicit: **a control-flow -exception outranks a body failure, which outranks a cleanup failure**, and the -loser is recorded rather than dropped. Classifying which exceptions are -control-flow is the pytest adapter's job, not the core's. +Three details the first draft got wrong, each checked by execution: + +- **`Failed.failures` is plural.** Under `continue_on_failure` one block reports + several failures; a singular field silently keeps the first. +- **`Passed` carries counts.** A block can pass *and* have skipped examples — + `failed=0 attempted=2 skipped=1` — and a result type without counts loses the + skip entirely, which is the same information ADR 0001's outcome contract + promises to surface. +- **`SkipReason` is typed.** A skip originates from `:skipif:`, an inline + `# doctest: +SKIP`, `:pyversion:`, or a profile declining to run — and "the gate + expression" describes only the first. + +**Exception precedence is phase-aware, not a single ladder.** Grouping +{exc}`KeyboardInterrupt`, a debugger quit, {exc}`pytest.skip` and `xfail` into one +"control-flow" tier is unsafe: a `pytest.skip()` raised *in cleanup* must not +erase a real body failure. + +| Tier | Examples | Rule | +|---|---|---| +| 1. process abort | {exc}`KeyboardInterrupt`, `SystemExit`, `bdb.BdbQuit` | always wins, from any phase | +| 2. pytest outcome from the **body** | `skip`, `xfail`, `exit` | wins over doctest failures | +| 3. doctest failure from the **body** | `DocTestFailure`, `UnexpectedException` | | +| 4. runtime startup failure | a profile that would not start | fails the block as `Errored` | +| 5. cleanup failure, of any kind | including a `pytest.skip()` in cleanup | never outranks 2–4; always recorded in `secondary` | + +Profile runtimes are entered through {class}`contextlib.ExitStack`, so a partial +startup unwinds deterministically in reverse. Classifying which exceptions are +pytest outcomes is the pytest adapter's job; the core knows only the phase. `Block.line` being nullable is load-bearing, not defensive. A bare `>>>` block nested in a `.. note::`, a list item or a block quote reports `line=None, @@ -562,13 +641,13 @@ source=None` from docutils, and an `.. include::`-ed block numbers against the *included* file. Both propagate to `DocTest.lineno=None` and pytest's honest "location unknown", rather than to a fabricated number. -`PlannedBlock` exists because `run_group()` owns phase sequencing, run-time -`:skipif:` evaluation and a cleanup `finally` — and cannot do any of the three -from a bare tuple of `DocTest`s. A `DocTest` carries no phase and no gate, so the -plan has to. Tagging each entry also makes the ordering self-describing rather -than a convention a comment asserts. +`ProjectedBlock` carries phase and gate because `run_group()` owns phase +sequencing, run-time `:skipif:` evaluation and a cleanup `finally` — and cannot do +any of the three from a bare tuple of `DocTest`s. A `DocTest` carries no phase and +no gate, so the recipe has to. Tagging each entry also makes the ordering +self-describing rather than a convention a comment asserts. -**A `PlannedBlock.test` cannot always be fully preconstructed.** Sphinx accepts +**A paired `want` is not known until run time.** Sphinx accepts `:skipif:` on a `testoutput`, and when that output is gated away its `testcode` still runs — expecting *empty* output. So the `want` of a paired block is a function of a gate evaluated at run time, and the plan must carry the paired @@ -576,10 +655,12 @@ output as data (`ExpectedOutput`, itself gated) with the `DocTest` finalized in `run_group()`. Freezing `want` at projection time silently runs the wrong assertion. -**A wildcard block needs a distinct `DocTest` per group it joins.** `DocTest.globs` -is a plain mutable attribute, so one object shared across two `GroupPlan`s would -have the second group's assignment win and both groups would execute against one -mapping. `*` membership therefore materializes a separate `DocTest` per group. +**A wildcard block is materialized separately per group it joins.** Under the +recipe model this costs nothing — the same `ProjectedBlock` appears in each +group's plan and each `RunContext` builds its own `DocTest` from it, with its own +minted name. What would be wrong is *sharing* a materialized `DocTest` across two +groups: `DocTest.globs` is a plain mutable attribute, so the second group's +assignment would win and both would execute against one mapping. **The gate's evaluation namespace is a deliberate divergence.** Sphinx evaluates each `:skipif:` in a fresh context seeded with `doctest_global_setup`; this design From 3fb8fd232e641918305ea6978d3db9d6ab5cc02f Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 19:45:41 -0500 Subject: [PATCH 21/31] docs,notes: Close the registry, compatibility and diagnostics gaps MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit why: The remaining review items were the contracts named but never written: a registry that ADR 0001 depended on and ADR 0006 did not define, a compatibility promise stated as one phrase covering surfaces of very different strength, and an unknown-directive default that could swallow a body holding doctests. what: - 0006: define the freeze lifecycle — Contributor and Registrar signatures, build_registry, duplicate names as an error unless replace=True, deterministic contribution order, and a freeze point per host - 0006: nested conftests may not contribute, since they load after any freeze that precedes collection; their fixtures and hooks are unaffected - 0006: workers compare a registry manifest rather than node ids, since two workers can collect identical ids while resolving one profile name to different code — which is what makes SSH and socket workers safe rather than assumed homogeneous - 0001: decompose "vanilla-compatible" into a matrix, and narrow the Sphinx promise to consuming a resolved doctree, since no execution lifecycle or result channel is specified - 0001: name the registry as a fifth input collection is not pure over - 0001: correct the layer diagram, which still said the runner always owns the example loop - 0004: split unknown roles from unknown body-owning directives. A role cannot swallow a code block; a container can, so it is a collection error by default and a project registers legitimate containers explicitly - Notes: the execution profile is contributable, not private, matching 0001's public contributor protocol; entry-point discovery is optional behind it rather than out of scope - Notes: narrow the node-id claim to per-block ids over shared MUTABLE state, since ids over isolated state do keep the promise --- docs/adrs/0001-typed-vanilla-doctest-core.md | 53 ++++++++++++++---- docs/adrs/0004-diagnostics-as-data.md | 39 ++++++++----- .../0006-pytest-private-api-compatibility.md | 56 +++++++++++++++++++ notes/analyses/22-extension-seams.md | 4 +- .../23-namespace-scope-and-test-identity.md | 9 +-- 5 files changed, 129 insertions(+), 32 deletions(-) diff --git a/docs/adrs/0001-typed-vanilla-doctest-core.md b/docs/adrs/0001-typed-vanilla-doctest-core.md index 633e4c5..5c5c36a 100644 --- a/docs/adrs/0001-typed-vanilla-doctest-core.md +++ b/docs/adrs/0001-typed-vanilla-doctest-core.md @@ -262,7 +262,8 @@ markup/ DocumentParser: _rst, _myst -> doctree | project grouping, pairing, phase order, naming -> GroupPlan | -runner owns the per-example loop; phase sequencing +runner stock loop for prompt blocks; owned loop for extended + | profiles only. Phase sequencing, gates, cleanup finally | pytest_doctest_docutils collection, items, globs lifetime, reporting ``` @@ -316,18 +317,18 @@ ordering guarantees exist. Its *implementation* stays private. pytest hooks, Sphinx's `setup(app)`, direct library calls and — optionally — entry points all feed that one surface. -Two hazards to name rather than assume away: +Two hazards decide the lifecycle, and +{doc}`0006-pytest-private-api-compatibility` settles both: -- **Nested `conftest.py` files load during collection**, so "freeze before - collection" is too simple. Either the boundary moves later, or conftest-level - contribution is explicitly excluded. That is an open question in - {doc}`0006-pytest-private-api-compatibility`. +- **Nested `conftest.py` files load during collection**, after any freeze that + precedes it. They therefore may not contribute block kinds, parsers, profiles or + checkers; attempting to is an error naming the file. Their fixtures and ordinary + hooks are unaffected. - **Entry-point discovery is not identical across workers in general.** xdist supports SSH and socket workers and heterogeneous environments, so "same argv, - same plugins" holds only for a homogeneous local run. Either state a - homogeneous-plugin requirement or compare a deterministic registry manifest - across workers. What breaks determinism is *nondeterministic contribution*, not - discovery as such — but heterogeneity makes that distinction load-bearing. + same plugins" holds only for a homogeneous local run. Workers therefore compare + a deterministic registry **manifest**, not node ids — two workers can collect + identical ids while resolving one profile name to different code. ### Item lifecycle @@ -786,6 +787,32 @@ layer that never changes what is constructed. `tests`, and the marker file does not exist, so every consumer sees the package as untyped. This is a packaging defect independent of the rest of this ADR. +### What "vanilla-compatible" promises + +The phrase is worth decomposing, because it covers several different promises of +several different strengths. + +| Surface | Promise | +|---|---| +| `Example` / `DocTest` runtime types | **Exact.** Stock instances, never subclassed for metadata | +| plain-text parsing | **Exact.** The stdlib lane uses `DocTestParser` unmodified | +| option flags and checkers | **Exact.** `register_optionflag` and the stdlib checker contract, with pytest's `ALLOW_UNICODE`/`ALLOW_BYTES`/`NUMBER` reachable | +| prompt-block execution | **Exact.** CPython's own per-example loop, unmodified | +| `DocTestFinder`-shaped Python-object discovery | **Shaped**, as a separate adapter | +| `DocFileSuite` / `DocTestSuite` | **Façade only**, over the stdlib lane. A group plan cannot be expressed through an API that returns one `DocTest` per parser call | +| `{testcode}`, async, groups, phases, Sphinx gates | **Deliberate extension.** No stdlib equivalent to be compatible with | +| pytest collection, fixtures, reporting | pytest's own contracts, composed with rather than replaced | +| Sphinx **node** vocabulary | **Exact.** The `BlockAttributes` stamp is byte-compatible | +| Sphinx **execution** | **Not promised.** See below | + +**The Sphinx promise is narrow, and this record narrows it deliberately.** What is +offered is an *extractor over a Sphinx-resolved doctree* — a pure function from a +doctree to blocks, callable from an extension. A full Sphinx host would need an +execution lifecycle and a result channel that neither this record nor +{doc}`0006-pytest-private-api-compatibility` defines, and inventing one would be +the builder that [](#alternatives-rejected) turns down. Until someone specifies +that adapter, the record promises doctree consumption and nothing more. + ## Constraints The design is pinned by facts about three upstreams. Each was verified at the tag @@ -920,9 +947,11 @@ which blocks will skip. (the-collection-contract)= Collection is **not** a pure function of (bytes, argv, ini), and claiming so -would be wrong on four counts: `.. include::` reads transitive files, docutils +would be wrong on five counts: `.. include::` reads transitive files, docutils directive implementations execute during parsing, the directive registry is -process-global, and MyST plugins change the tree. The defensible contract is +process-global, MyST plugins change the tree, and the frozen registry is itself +an input — assembled from installed plugins and conftests, neither of which is +argv or ini. The defensible contract is determinism over **complete source closure + normalized settings + frozen registry** — all three defined above. Deferring the author's gate removes the largest divergence risk; it does not make xdist divergence structurally diff --git a/docs/adrs/0004-diagnostics-as-data.md b/docs/adrs/0004-diagnostics-as-data.md index aceb8cb..45ed820 100644 --- a/docs/adrs/0004-diagnostics-as-data.md +++ b/docs/adrs/0004-diagnostics-as-data.md @@ -55,10 +55,25 @@ surface. ## Direction -Suppress narrowly, and only for the two classes a bare-docutils parse cannot -judge: unknown roles and unknown directives. "By code" is the intent; the -classifier that assigns a code to a docutils message is unsettled, so this -direction is not yet implementable. +**Treat unknown roles and unknown directives differently.** They are not the same +risk, and the first draft's symmetric treatment was the mistake. + +An **unknown role** is inline markup. It cannot swallow a code block, so a +bare-docutils parse seeing `:mod:` in a Sphinx project is noise and is suppressed +by default. + +An **unknown body-owning directive** is a collection error. It swallows its body +unparsed, so a page whose doctests live inside one collects zero tests and exits +green — which is the failure diagnostics-as-data exists to prevent. Suppressing it +by default trades a loud, correct error for a silent wrong answer. A project with +legitimate foreign containers registers them as known vocabulary; that is an +explicit act, not a default. + +For a Sphinx project the question does not arise: the extractor consumes an +already-resolved doctree, in which every registered directive has run. + +"By code" remains the intent for everything else; the classifier that assigns a +code to a docutils message is unsettled, which is why this record stays `Draft`. Every diagnostic raised by this project's own layers defaults to visible, and `level="error"` from those layers fails collection with the file and line named. @@ -80,15 +95,11 @@ global on/off switch. pre-empting at the source by overriding the directive-dispatch path so the unknown case never becomes a reporter message at all. This is the decision ADR 0004 cannot ship without. -- **What promotes an unknown-directive message back to visible.** The first - draft proposed "when the name is one this project registers", but that is - inverted for the typo case — a misspelled `.. doctset::` is precisely *not* a - registered name — and never fires for a foreign container whose body was - swallowed unparsed. The rule that covers both is **body content**: promote when - the swallowed body matches `DocTestParser._EXAMPLE_RE`. Near-miss-to-a-registered - -name is a useful additional rule for the typo case, where the body check does - not help. -- The body-content rule is reST-only as stated, because myst-parser discards the - fence body. The Markdown equivalent is unsolved. +- **How a project registers a legitimate foreign container**, so that an unknown + body-owning directive it genuinely does not care about stops erroring. This is + the escape hatch the default requires, and it needs a spelling. +- Whether a near-miss to a registered name (`.. doctset::` for `.. doctest::`) + earns a distinct, more helpful message than the generic unknown-directive + error. Cheap, and the typo is the common case. - Whether the CLI (`python -m doctest_docutils`) and the pytest plugin share one formatter or two. diff --git a/docs/adrs/0006-pytest-private-api-compatibility.md b/docs/adrs/0006-pytest-private-api-compatibility.md index 8cae260..e72b56d 100644 --- a/docs/adrs/0006-pytest-private-api-compatibility.md +++ b/docs/adrs/0006-pytest-private-api-compatibility.md @@ -99,6 +99,62 @@ symbol, and it names the document that triggered it. CI carries a job pinned to the minimum supported pytest and one tracking its prerelease. +### The registry freeze lifecycle + +{doc}`0001-typed-vanilla-doctest-core` depends on a frozen registry and leaves the +lifecycle here. This is it. + +```python +class Contributor(t.Protocol): + """What a plugin implements to extend the core.""" + + def contribute(self, registrar: Registrar) -> None: ... + + +class Registrar(t.Protocol): + def add_block_kind(self, kind: BlockKind, *, replace: bool = False) -> None: ... + def add_document_parser( + self, parser: DocumentParser, *, replace: bool = False + ) -> None: ... + def add_execution_profile( + self, profile: ExecutionProfile, *, replace: bool = False + ) -> None: ... + def add_output_checker( + self, name: str, factory: CheckerFactory, *, replace: bool = False + ) -> None: ... + + +def build_registry(contributors: t.Sequence[Contributor]) -> Registry: ... +``` + +**Duplicate names are an error unless `replace=True`.** Silent last-writer-wins is +what makes the docutils directive table a recurring bug source, and this registry +does not repeat it. Contribution order is: built-ins, then installed plugins in +`pluginmanager` registration order, then explicit contributors — deterministic +under a given plugin set. + +**Freeze points, per host:** + +| Host | Contributors accepted | Frozen at | +|---|---|---| +| direct API | whatever the caller passes to `build_registry` | on return | +| pytest | built-ins, installed plugins, explicit contributors, and **initial/root conftests only** | `pytest_sessionstart` | +| Sphinx | extensions, via `setup(app)` | after extension setup, before reading | + +**Nested conftests may not contribute.** They load during collection, after the +freeze, and a registry that grows while collection runs cannot be the same in +every worker. Registration from a nested conftest is an error naming the file — +not a silent late addition. Fixtures and ordinary pytest hooks in nested conftests +are unaffected; this is only about block kinds, parsers, profiles and checkers. + +**Workers compare a manifest, not node ids.** Matching collection is not +sufficient: two workers can collect identical ids while resolving the same profile +name to different code. So the controller ships a deterministic registry +manifest — sorted `(kind, name, provider, version)` tuples — through worker +configuration, and a worker whose manifest differs fails the session with both +manifests in the message. This is what makes heterogeneous xdist (SSH, socket, +mixed environments) safe rather than assumed-homogeneous. + ## Open - Whether requiring the built-in plugin should be stated as a hard dependency. diff --git a/notes/analyses/22-extension-seams.md b/notes/analyses/22-extension-seams.md index 57a1bc7..ecf02c1 100644 --- a/notes/analyses/22-extension-seams.md +++ b/notes/analyses/22-extension-seams.md @@ -88,10 +88,10 @@ needs it. Applied to the candidates that came up: | `BlockKind` registry | **Yes.** Turns "a new block kind" from an edit to a method branching on string literals into adding a tuple. Which docutils node classes a kind arrives as stays in `markup/`, not on `BlockKind`, so the leaf keeps its stdlib-only contract | | Output checker injection | **Yes.** The highest-demand seam, and the one Sybil closed entirely by hard-coding `checker=OutputChecker()` — adding one there requires subclassing two classes | | `Frontend` protocol | **Yes.** Four implementations ship on day one (rST, MyST, text, Python docstrings) | -| `ExecutionProfile` (private) | **Yes, private.** [PR #59](https://github.com/git-pull/gp-libs/pull/59) adds top-level `await` — a second execution policy a `Literal["single", "exec"]` cannot express. Stays private until an out-of-tree caller exists | +| `ExecutionProfile` | **Yes, and contributable.** [PR #59](https://github.com/git-pull/gp-libs/pull/59) adds top-level `await` — a second execution policy a `Literal["single", "exec"]` cannot express. The *type* is part of the public contributor protocol, since a pluggable core that cannot accept a new execution policy is not pluggable; the registry holding them stays private | | A per-example observer protocol | **No.** pytest gets failures through `report_*` and `MultipleDoctestFailures`; the CLI uses `summarize()`. No third consumer | | A registry *object* with builders, digests and manifests | **No object, but there is a freeze.** The contributed block kinds, front ends and profiles are resolved into an immutable mapping alongside the frozen `Settings` facets, before collection begins; registering after that is an error. That is what ADR 0001's collection contract means by *frozen registry* — not a class with its own lifecycle | -| Entry-point plugin discovery | **Not in v1**, for want of a caller outside the package — *not* because it breaks xdist. Workers are fresh processes that re-run plugin loading with the same argv, so entry-point resolution is identical everywhere. What breaks determinism is nondeterministic *contribution*, such as a conftest registering conditionally on the environment | +| Entry-point plugin discovery | **Optional, behind the contributor protocol.** Not the hazard it was once called: what breaks determinism is nondeterministic *contribution*, not discovery. But "identical across workers" holds only for a homogeneous local run — xdist supports SSH and socket workers — so a heterogeneous session needs a registry manifest comparison or an explicit homogeneous-plugin requirement | The distinction those two rows turn on is worth stating once: **discovery is not the hazard, nondeterminism is.** A registry populated identically in every process diff --git a/notes/analyses/23-namespace-scope-and-test-identity.md b/notes/analyses/23-namespace-scope-and-test-identity.md index e457d20..0167fb6 100644 --- a/notes/analyses/23-namespace-scope-and-test-identity.md +++ b/notes/analyses/23-namespace-scope-and-test-identity.md @@ -106,10 +106,11 @@ Two corollaries: ## What a node id does *not* promise Worth stating plainly, because it is the honest limit of the recommended design: -**no surveyed implementation makes a node id a promise of independent -runnability.** Selecting block two of a stateful page raises `NameError` under -Sybil, under `per-block`, and under any scheme that hands out per-block ids over -shared state. +**a per-block node id over shared mutable state cannot truthfully promise +independent execution.** Selecting block two of a stateful page raises +`NameError` under Sybil, under PR #87's `per-block`, and under any scheme of that +shape. Ids over *isolated* state — released `doctest_docutils`, or a page whose +blocks declare no group — promise independence and keep the promise. The choice is therefore not between "selectable blocks" and "unselectable blocks". It is between an id that *claims* to be selectable and is not, and an id whose From d7af005f7f5cf6dee8484c85d74a8c92ae4747b1 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 20:55:24 -0500 Subject: [PATCH 22/31] doctest_core(docs[model]): Complete contracts why: The proposed core mixed settings with discovered capabilities and left source ordering, host outcomes, checker rendering, and result projection underspecified. what: - Separate contracts, settings, models, and registry snapshots - Define stable source identity and complete gate/result records - Specify pytest failure projection and phase-aware exceptions --- docs/adrs/0001-typed-vanilla-doctest-core.md | 294 +++++++++++-------- 1 file changed, 177 insertions(+), 117 deletions(-) diff --git a/docs/adrs/0001-typed-vanilla-doctest-core.md b/docs/adrs/0001-typed-vanilla-doctest-core.md index 5c5c36a..1925caa 100644 --- a/docs/adrs/0001-typed-vanilla-doctest-core.md +++ b/docs/adrs/0001-typed-vanilla-doctest-core.md @@ -112,8 +112,9 @@ why `SphinxDocTestRunner` overrides a private stdlib method to swallow the resulting `IndexError`. Mapping the group onto one {class}`pytest.Item` while each block keeps its own identity is the contribution. -What that buys, for free and with no override of `repr_failure` or `reportinfo`: -per-block failure locations, per-block gutters, and per-block "location unknown". +With the default checker that buys, for free and with no override of +`repr_failure` or `reportinfo`: per-block failure locations, per-block gutters, +and per-block "location unknown". Meanwhile `-k`, `--lf`, `-x`, `--reruns` and every `--dist` mode are structurally incapable of splitting the shared state, because there is only one item to schedule. @@ -152,7 +153,8 @@ With one `DocTest` per block, every failure therefore carries its own `filename` and `lineno` for free. A block reached through `.. include::` reports the *included* file. A block docutils could not locate carries `lineno=None` and takes pytest's honest `EXAMPLE LOCATION UNKNOWN` branch **without poisoning its -siblings**. No override of `repr_failure` or `reportinfo` is required. +siblings**. The default-checker path requires no override of `repr_failure` or +`reportinfo`. This is what makes merging unnecessary: the synthetic page, its blank-line padding and its clamp exist only to reconstruct locations from a single spliced @@ -199,9 +201,9 @@ under any `--dist` mode. No affinity primitive, no scheduler substitution, no `parse_tx_spec_config` fork, no node-id string sniffing. The identical-collection requirement is untouched by this and still binds at any -granularity. It is satisfied separately, by collection being a pure function of -(bytes, argv, ini) — which is why `:skipif:` is carried through collection -unevaluated. +granularity. It is approached separately through deterministic projection over +the complete source closure, normalized settings and a frozen registry — which +is why `:skipif:` is carried through collection unevaluated. (the-outcome-contract)= @@ -212,7 +214,7 @@ stated here rather than discovered later. | Signal | Granularity | Notes | |---|---|---| -| Failure location, `want`/`got`, gutter | **per block** | `repr_failure` iterates failures and reads each one's own `DocTest` | +| Failure location, `want`/`got`, gutter | **per block** | default `repr_failure` iterates failures and reads each one's own `DocTest`; custom checkers use the same locations in the narrow rendering branch | | `EXAMPLE LOCATION UNKNOWN` | **per block** | a block with `lineno=None` does not affect its siblings | | `passed` / `failed` / `skipped` | **per item** | `TestReport.outcome` is one scalar | | JUnit `` | **per item** | node reporters are keyed by node id | @@ -225,9 +227,10 @@ block's skip, a group with one all-`SKIP` block and one passing block reports though a sibling passed. pytest's own doctest plugin takes the second horn only when *every* example is skipped, via `_check_all_skipped`. -This design takes the same position: **skip the item when every runnable block is -skipped; otherwise report partial skips as typed block detail, not as a pytest -outcome.** No extra reports are synthesized. +This design takes the same position over the test phase: **skip the item when +every `Phase.TEST` block is skipped; otherwise report partial skips as typed block +detail, not as a pytest outcome.** Setup and cleanup are infrastructure and do +not contribute a passed or skipped test. No extra reports are synthesized. "Typed block detail" needs a channel, or implementers will re-invent skip lifting or write to stderr. The channel is a `GroupResult` — one `BlockResult` per block, @@ -249,32 +252,29 @@ surface. Revisit if it stabilizes. ### Layers -Dependencies flow one way, from the leaf toward the hosts. No layer may import a -layer above it. +Dependencies flow from the hosts toward small foundations. No foundational +layer imports a host, and configuration never owns discovered capabilities. ```text -settings one frozen Settings, resolved once <- leaf; everything reads it - | -blocks inert data: Block, BlockKind, Phase, Diagnostic, Example - | (stdlib imports only) -markup/ DocumentParser: _rst, _myst -> doctree - | extract_blocks(doctree) -> (parsed blocks, diagnostics) - | -project grouping, pairing, phase order, naming -> GroupPlan - | -runner stock loop for prompt blocks; owned loop for extended - | profiles only. Phase sequencing, gates, cleanup finally - | -pytest_doctest_docutils collection, items, globs lifetime, reporting +contracts settings model + \ | / + +--------- registry --------+ + | + markup + | + project + | + runner + | + direct / pytest / Sphinx hosts ``` -`settings` sits at the bottom, not beside the host, because `Frontend.parse` and -`project()` both take a `Settings`. A layer every other layer reads is a leaf. - | Layer | Owns | Must not know | |---|---|---| -| `settings` | Three immutable facets — `ParseSettings`, `ProjectionSettings`, `RunSettings` — plus the **frozen registry**, resolved together exactly once per session. `None` sentinels at the resolve boundary make a future default change announceable | pytest's `Config`, argparse, ini format, Sphinx's `app`. The host extracts; this resolves | -| `blocks` | `ParsedBlock`, `ParsedOutput`, `BlockKind`, `Phase`, `Diagnostic`, `ProjectedBlock`, `GroupPlan`, `RunContext`, `ExecutionProfile`, the result types. **No stdlib subclasses.** | docutils, MyST, Sphinx, pytest, xdist, the filesystem. Stdlib imports only, enforced by an `import-linter` contract | +| `contracts` | Public protocols and immutable contribution records: `DocumentParser`, `ExecutionProfile`, `ExecutionRuntime`, `CheckerFactory`, `Contributor` and `Registrar` | Sphinx, pytest, xdist and host lifecycle objects. Only stdlib and public parser types cross this boundary | +| `settings` | Three immutable facets — `ParseSettings`, `ProjectionSettings`, `RunSettings`. `None` sentinels at the resolve boundary make a future default change announceable | registries, pytest's `Config`, argparse, ini format, Sphinx's `app`. The host extracts; this resolves | +| `model` | `ParsedBlock`, `ParsedOutput`, `BlockKind`, `Phase`, `Diagnostic`, `ProjectedBlock`, `GroupPlan`, `RunContext` and the result types. **No stdlib subclasses.** | docutils, MyST, Sphinx, pytest, xdist, the filesystem. Stdlib imports only | +| `registry` | A private mutable builder and the public immutable `RegistrySnapshot` consumed by every later layer | host lifecycle objects after the snapshot is frozen | | `markup/` | Text → `(blocks, diagnostics)`. **The whole docutils vocabulary**: which node classes each kind may arrive as, the `BlockAttributes` stamp, line-number recovery and its per-front-end meaning, `.. include::` attribution, `nodes.comment` traversal, reporter capture, idempotent directive registration, per-kind `option_spec` | Groups as a runtime concept, `DocTest`, pytest, pairing | | `project` | The **only** place grouping exists: `*` expansion, anonymous naming, phase order, `testcode`/`testoutput` pairing, option defaults, name minting. A pure function | docutils, pytest, the filesystem, whether anything will run. Evaluates no user code | | `runner` | `_DocTestRunner__run`; option merge, `SKIP`-after-merge, `FAIL_FAST`, `report_*` dispatch, version shims. `run_group()` owns phase sequencing, run-time `:skipif:`, the profile's context manager, and the `try`/`finally` guaranteeing `testcleanup` | docutils, markup, pytest. Never overrides `run()` | @@ -292,7 +292,7 @@ document is parsed. Three scopes: | Scope | Owns | Resolved | |---|---|---| -| `SessionSettings` | defaults plus normalized host configuration; the frozen registry | once, at session start | +| `SessionSettings` | defaults plus normalized host configuration | once, before collection or direct execution | | `DocumentSettings` | the front-matter overlay a page is permitted to set | per document, after parsing | | block / example policy | directive options, then inline `# doctest:` flags | per block, at projection and run | @@ -306,29 +306,28 @@ to the host adapter. And wildcard resolution and name minting are *invariants*, not user-configurable knobs — exposing them would let a project produce node ids no other project can read. -The **registry** is resolved and frozen alongside `SessionSettings`. "Frozen" -means an immutable mapping built once; registering afterwards is an error. - -**Contribution is public; the registry is not.** The stated goal is an -extendable, pluggable core, so a small host-neutral contributor protocol ships in -v1 — what may be contributed (front ends, block kinds, execution profiles, -checkers), when registration closes, how duplicate names are resolved, and what -ordering guarantees exist. Its *implementation* stays private. pytest hooks, -Sphinx's `setup(app)`, direct library calls and — optionally — entry points all -feed that one surface. - -Two hazards decide the lifecycle, and -{doc}`0006-pytest-private-api-compatibility` settles both: - -- **Nested `conftest.py` files load during collection**, after any freeze that - precedes it. They therefore may not contribute block kinds, parsers, profiles or - checkers; attempting to is an error naming the file. Their fixtures and ordinary - hooks are unaffected. -- **Entry-point discovery is not identical across workers in general.** xdist - supports SSH and socket workers and heterogeneous environments, so "same argv, - same plugins" holds only for a homogeneous local run. Workers therefore compare - a deterministic registry **manifest**, not node ids — two workers can collect - identical ids while resolving one profile name to different code. +`ProjectionSettings.ungrouped` defaults to `"default"`. An unlabelled runnable +block therefore joins the page's `default` group unless the user explicitly asks +for block isolation. This clean-slate default follows Sphinx's author vocabulary +and makes state-sharing opt-out rather than a project-specific surprise. + +The **registry** is a separate input resolved after `SessionSettings`. "Frozen" +means the public `RegistrySnapshot` contains immutable mappings and records; the +mutable builder is private and discarded. Registering after the host freezes its +snapshot is an error. Keeping these values separate matters under xdist: settings +are normalized user input, while the snapshot is the capability set discovered in +that process. + +**Contribution and the snapshot are public; mutation is not.** The stated goal is +an extendable, pluggable core, so a small host-neutral contributor protocol ships +in v1. Front ends, block kinds, execution profiles and checkers all feed one +builder and every consumer receives the same `RegistrySnapshot`. Host-specific +registration timing is a separate decision; the core contract does not import a +pytest hook or a Sphinx application. + +The direct, pytest and Sphinx lifecycles, including the xdist consistency check, +are specified separately so this record does not mistake a host bootstrap policy +for a core dependency. ### Item lifecycle @@ -358,8 +357,8 @@ an implementer cannot get it wrong by omission: `DoctestItem.runtest`, which runs a single `dtest` with `clear_globs` defaulting to `True` — that would empty the shared mapping after the first block. It calls `run_group()`, which materializes and runs each - `RunnableBlock` in phase order - with `clear_globs=False`, evaluates `:skipif:` against the live mapping, + `RunnableBlock` in phase order with `clear_globs=False`, evaluates `:skipif:` + and `:pyversion:` against the live mapping and interpreter, finalizes each paired `want` from its gated `ExpectedOutput`, and wraps the body in a `try`/`finally` so cleanup runs whether or not the body raised. When cleanup *also* fails, the body's failure is the one raised; cleanup's is @@ -368,20 +367,29 @@ an implementer cannot get it wrong by omission: The `OutcomeException` re-raise, the `bdb.BdbQuit` → `outcomes.exit` conversion and `continue_on_failure` are reimplemented here, because `PytestDoctestRunner` is nested inside a factory and cannot be imported. -4. **Outcome** follows [](#the-outcome-contract): skip the item only when every - runnable block is skipped. +4. **Outcome** follows [](#the-outcome-contract): only `Phase.TEST` blocks + determine pass versus skip. A plan with no test block yields no item. Setup and + cleanup are infrastructure: an error there may fail or abort the item, but a + successful setup is not a passed test and a cleanup skip cannot erase the test + result. Skip the item only when every test block is skipped. `runtest()` must also keep `_disable_output_capturing_for_darwin()`, which the inherited implementation calls before running and which has nothing to do with grouping. -5. **`repr_failure`** is inherited unchanged for *failures* — it already reads - each failure's own `DocTest`. It does **not** render `GroupResult`, and - claiming it does would conflate two channels. Partial-skip detail goes to a - report section and the terminal summary; see [](#the-outcome-contract). +5. **Failure projection** flattens every `Failed.failures` tuple in block order + and raises pytest's `MultipleDoctestFailures`. With the default pytest checker, + `repr_failure` is inherited unchanged and therefore preserves pytest's exact + output. A contributed checker must supply both `check_output` and + `output_difference`; the item takes one small custom-rendering branch so the + checker that decided the failure also explains it. `repr_failure` does not + render `GroupResult`. Partial-skip detail goes to a report section and the + terminal summary; see [](#the-outcome-contract). 6. **Reporting across processes.** The controller never sees the item — it receives serialized `TestReport` dictionaries. So `pytest_runtest_makereport` - copies a **versioned, JSON-safe** block summary onto the report; pytest + copies a **versioned, JSON-safe** block summary onto the report. The summary + contains only the schema version, group name, block name, phase, outcome, + counts and structured skip reason — never exceptions or live objects. pytest serializes arbitrary report attributes and xdist reconstructs them controller-side. The rich `GroupResult`, and every exception in it, stays worker-local. @@ -414,6 +422,37 @@ class Phase(enum.IntEnum): CLEANUP = 2 +# --- contracts: public, host-neutral extension seams ---------------------- + + +Failure: t.TypeAlias = doctest.DocTestFailure | doctest.UnexpectedException + + +class RuntimeOutcome(t.NamedTuple): + results: doctest.TestResults + failures: tuple[Failure, ...] + + +class RuntimeSettings(t.NamedTuple): + optionflags: int + continue_on_failure: bool + checker: doctest.OutputChecker + + +class CheckerFactory(t.Protocol): + def __call__(self) -> doctest.OutputChecker: ... + + +class ExecutionRuntime(t.Protocol): + def run(self, test: doctest.DocTest) -> RuntimeOutcome: ... + + +class ExecutionProfile(t.Protocol): + def open( + self, settings: RuntimeSettings + ) -> contextlib.AbstractContextManager[ExecutionRuntime]: ... + + # --- parsed: inert, produced by extraction, owns no semantics ------------- @@ -422,11 +461,12 @@ class ParsedBlock(t.NamedTuple): source: str # dedented body, verbatim author text path: pathlib.Path # the file the text lives in, not the collected document line: int | None # None when docutils could not recover one - source_ordinal: int # ordinal among runnable candidates, BEFORE gating, - # filtering and group expansion; the naming key + document_order: int # position among blocks AND outputs; the pairing key + block_ordinal: int # position among runnable blocks; the identity key groups: tuple[str, ...] # declared verbatim; () and ("*",) unresolved here options: t.Mapping[int, bool] # plain int keys, exactly as doctest produces skipif: str | None # UNEVALUATED + pyversion: str | None # UNEVALUATED PEP 440 specifier hidden: bool @@ -436,10 +476,11 @@ class ParsedOutput(t.NamedTuple): text: str path: pathlib.Path line: int | None - source_ordinal: int + document_order: int # shares one sequence with ParsedBlock for pairing groups: tuple[str, ...] options: t.Mapping[int, bool] skipif: str | None # a gated output means its testcode expects nothing + pyversion: str | None class BlockKind(t.NamedTuple): @@ -457,6 +498,7 @@ class ExpectedOutput(t.NamedTuple): text: str options: t.Mapping[int, bool] skipif: str | None # when truthy at run time, `want` becomes "" + pyversion: str | None # when disallowed at run time, `want` becomes "" class ExampleRecipe(t.NamedTuple): @@ -475,6 +517,7 @@ class ProjectedBlock(t.NamedTuple): phase: Phase name: str # the minted test name + block_ordinal: int # stable among runnable blocks before filtering examples: tuple[ExampleRecipe, ...] # a prompt block yields SEVERAL docstring: str # what pytest's failure renderer slices filename: str @@ -482,6 +525,7 @@ class ProjectedBlock(t.NamedTuple): options: t.Mapping[int, bool] # block-level directive :options: profile_name: str # resolved against the frozen registry per attempt skipif: str | None # UNEVALUATED; gated in run_group() + pyversion: str | None # UNEVALUATED; gated in run_group() expected: ExpectedOutput | None # paired testoutput, itself gateable @@ -532,11 +576,12 @@ class Failed(t.NamedTuple): block: ProjectedBlock counts: Counts # PLURAL: continue_on_failure yields several from one block - failures: tuple[doctest.DocTestFailure | doctest.UnexpectedException, ...] + failures: tuple[Failure, ...] class Skipped(t.NamedTuple): block: ProjectedBlock + counts: Counts reason: SkipReason @@ -561,6 +606,14 @@ block: for a prompt-form block it is *inside* `source` and `testcode`/`testoutput` pair it is a separate `ParsedOutput`. Conflating the two was what made "projection owns pairing" untrue. +`document_order` is one monotonic sequence shared by runnable blocks and +`ParsedOutput` records. Pairing therefore follows the source stream even when an +output sits between two runnable candidates. `block_ordinal` counts runnable +blocks only and survives gating, filtering and wildcard expansion, so adding or +removing expected output cannot rename every later test. Both `:skipif:` and +`:pyversion:` remain data until the run boundary; collection never evaluates +either gate. + `BlockKind` names a profile rather than holding one, so a public type never contains a private one. The name resolves against the frozen registry. @@ -607,7 +660,7 @@ rather than inventing a deeper guarantee the ecosystem does not provide. unlikely. `Errored` exists because a gate that raises, or a runtime that will not start, is none of pass, fail or skip. -Three details the first draft got wrong, each checked by execution: +Four result details are load-bearing: - **`Failed.failures` is plural.** Under `continue_on_failure` one block reports several failures; a singular field silently keeps the first. @@ -615,28 +668,31 @@ Three details the first draft got wrong, each checked by execution: `failed=0 attempted=2 skipped=1` — and a result type without counts loses the skip entirely, which is the same information ADR 0001's outcome contract promises to surface. +- **`Skipped` carries counts too.** A whole-block gate attempts zero examples, + while an all-`SKIP` doctest has parsed examples and reports them skipped. The + reason alone cannot distinguish those cases. - **`SkipReason` is typed.** A skip originates from `:skipif:`, an inline `# doctest: +SKIP`, `:pyversion:`, or a profile declining to run — and "the gate expression" describes only the first. **Exception precedence is phase-aware, not a single ladder.** Grouping -{exc}`KeyboardInterrupt`, a debugger quit, {exc}`pytest.skip` and `xfail` into one -"control-flow" tier is unsafe: a `pytest.skip()` raised *in cleanup* must not -erase a real body failure. +{exc}`KeyboardInterrupt`, a debugger quit, {exc}`pytest.skip`, `xfail` and +`pytest.exit` into one "control-flow" tier is unsafe. A cleanup skip must not erase +a real test failure, while a session exit must never be converted into block data. -| Tier | Examples | Rule | +| Class | Examples | Rule | |---|---|---| -| 1. process abort | {exc}`KeyboardInterrupt`, `SystemExit`, `bdb.BdbQuit` | always wins, from any phase | -| 2. pytest outcome from the **body** | `skip`, `xfail`, `exit` | wins over doctest failures | -| 3. doctest failure from the **body** | `DocTestFailure`, `UnexpectedException` | | -| 4. runtime startup failure | a profile that would not start | fails the block as `Errored` | -| 5. cleanup failure, of any kind | including a `pytest.skip()` in cleanup | never outranks 2–4; always recorded in `secondary` | +| process, debugger or session abort | {exc}`KeyboardInterrupt`, `SystemExit`, `bdb.BdbQuit`, `pytest.exit` | always propagates from every phase; cleanup still runs, but cannot replace it | +| host outcome from setup or test | `pytest.skip`, `pytest.xfail` | propagates as the host outcome after cleanup | +| doctest mismatch from test | `DocTestFailure`, `UnexpectedException` | retained in source order and projected to the host after cleanup | +| runtime error from setup or test | a gate that raises, or a profile that cannot start | recorded as `Errored`; becomes the primary failure when no abort or host outcome exists | +| cleanup exception or cleanup host outcome | any exception, including `pytest.skip` or `pytest.xfail` | recorded as `secondary` when a primary exists; otherwise becomes the item failure, never a skip or xfail | Profile runtimes are entered through {class}`contextlib.ExitStack`, so a partial startup unwinds deterministically in reverse. Classifying which exceptions are pytest outcomes is the pytest adapter's job; the core knows only the phase. -`Block.line` being nullable is load-bearing, not defensive. A bare `>>>` block +`ParsedBlock.line` being nullable is load-bearing, not defensive. A bare `>>>` block nested in a `.. note::`, a list item or a block quote reports `line=None, source=None` from docutils, and an `.. include::`-ed block numbers against the *included* file. Both propagate to `DocTest.lineno=None` and pytest's honest @@ -656,12 +712,12 @@ output as data (`ExpectedOutput`, itself gated) with the `DocTest` finalized in `run_group()`. Freezing `want` at projection time silently runs the wrong assertion. -**A wildcard block is materialized separately per group it joins.** Under the -recipe model this costs nothing — the same `ProjectedBlock` appears in each -group's plan and each `RunContext` builds its own `DocTest` from it, with its own -minted name. What would be wrong is *sharing* a materialized `DocTest` across two -groups: `DocTest.globs` is a plain mutable attribute, so the second group's -assignment would win and both would execute against one mapping. +**A wildcard block is projected separately per group it joins.** Projection +clones the recipe and mints a group-qualified name for each destination. Each +`RunContext` then builds its own `DocTest` from its own recipe. Reusing one +`ProjectedBlock` would make its name ambiguous; sharing one materialized +`DocTest` would be worse, because `DocTest.globs` is mutable and the second +group's assignment would win. **The gate's evaluation namespace is a deliberate divergence.** Sphinx evaluates each `:skipif:` in a fresh context seeded with `doctest_global_setup`; this design @@ -691,6 +747,14 @@ doctests are then compatible **by construction** rather than by differential testing, and {doc}`0002-runner-conformance-across-cpython`'s harness shrinks to guarding the extended lane. +**A checker owns both comparison and explanation.** The default pytest +registration constructs pytest's checker, preserving `ALLOW_UNICODE`, +`ALLOW_BYTES`, `NUMBER` and its inherited failure representation exactly. A +contributed `CheckerFactory` constructs a fresh checker for each runtime. The +same instance performs `check_output()` and `output_difference()`; using pytest's +private `_get_checker()` only at rendering time would let one checker reject the +example and another explain why. + **Which docutils node classes a kind may arrive as is a front-end concern, not a `BlockKind` field.** `testsetup`, `testcleanup` and any `:hide:` block are emitted as {class}`docutils.nodes.comment`, not `literal_block` @@ -706,7 +770,7 @@ The runtime objects are stdlib's, unconditionally. Precision lives in a parallel layer that never changes what is constructed. - **Parsing and extraction are two seams, not one.** A single - `Frontend.parse(text, path)` cannot serve Sphinx, because a Sphinx extension + `DocumentParser.parse(text, path)` cannot serve Sphinx, because a Sphinx extension already *has* a doctree and a raw re-parse is not the same tree. So: ```python @@ -759,16 +823,15 @@ layer that never changes what is constructed. right shape for it. It is a {class}`doctest.DocTestFinder`-shaped adapter, and putting a `_python` module in `markup/` was a category error. -- **`Protocol` for the seams, nominal subclassing for type-checkability.** - `DocumentParser` is a `Protocol` so a third party can supply one structurally. The - shipped parser *also* subclasses {class}`doctest.DocTestParser` — not because - the interpreter requires it (stdlib `doctest` performs no `isinstance` check on - `parser` or `test_finder`; a duck-typed object works at runtime) but because - typeshed's signatures name the class, so a checker rejects what the interpreter - accepts. The subclass buys type-checkability, not passability. Passability is a - matter of matching the call signature: a finder whose `find()` takes a string - first cannot be handed to `DocTestSuite`, which passes a module — and - subclassing does not fix that. +- **`Protocol` for markup seams; nominal classes only for stdlib façades.** + `DocumentParser` is a `Protocol` so a third party can supply one structurally. + It must not subclass {class}`doctest.DocTestParser`: their `parse()` signatures + and return types are incompatible, so the apparent typeshed accommodation is + itself an invalid override. The optional `DocFileSuite` façade instead owns a + separate nominal `DocTestParser` adapter with the exact stdlib signature. + Runtime passability still comes from matching the called method: a finder whose + `find()` takes a string first cannot be handed to `DocTestSuite`, which passes a + module, and subclassing does not fix that. One genuine nominal edge does exist: `DocTestSuite` sorts its results, and `DocTest.__lt__` returns `NotImplemented` for a non-`DocTest`, so a custom @@ -892,17 +955,13 @@ Each is a genuine conflict where satisfying one goal costs another. "Both" is no an answer; the position taken and its price are recorded. **A vanilla `DocTest` cannot carry the front-end's metadata.** It has exactly -`(examples, globs, name, filename, lineno, docstring)`. *Position:* metadata the -runner needs at run time rides on a {class}`doctest.Example` subclass — -`doctest.Example` has no `__slots__`, so attributes survive {func}`copy.copy`, -{mod}`pickle`, and a third party's naive -`DocTest(examples, globs, name, filename, lineno, docstring)` rebuild, because -that rebuild reuses the same `Example` objects. Metadata the runner does *not* -need — groups, wildcards, pairing — never touches a stdlib object and dies in the -projection layer. - -*Price:* **none of it goes on {class}`doctest.Example`.** An earlier draft put -compile mode on an `Example` subclass, which fails twice. +`(examples, globs, name, filename, lineno, docstring)`. *Position:* all extension +metadata stays on `ProjectedBlock`, `RunContext` and the result records. Stock +`DocTest` and `Example` objects remain exact compatibility objects, not metadata +carriers. *Price:* a consumer holding only the stdlib object sees only stdlib +semantics; it must retain the core recipe to inspect groups, profiles or gates. + +Putting metadata on an `Example` subclass is rejected because {meth}`doctest.Example.__eq__` gates on exact type identity ([`Lib/doctest.py:518`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L518)), so a bare subclass is unequal to a stock `Example` with identical fields in both @@ -931,7 +990,8 @@ stock, and nothing in the compatibility kernel is subclassed for metadata at all `DocTest`s under one node id. *Price:* selecting a group runs all its blocks; there is no id that names block three alone. That is honest: no surveyed implementation makes a node id a promise of independent runnability, and the -current `per-block` mode ships ids that raise `NameError` when selected. +proposed shared per-block shape would expose ids that raise `NameError` when +selected without their predecessors. **Sphinx's skip versus pytest's skip.** *Position:* pytest's meaning, doctest's mechanism — set `SKIP`, never drop the node. *Price:* a page carrying a gated @@ -1051,8 +1111,8 @@ the boundaries the change was supposed to clean. mode in the same user-writable namespace as `ELLIPSIS`. A `doctest_optionflags = EXEC` in any ini, or a stray `# doctest: +EXEC`, compiles in exec mode, which suppresses expression echo so every `want` compares against empty output and the -suite passes vacuously. Compile mode is an attribute on the `Example` subclass, -unreachable from user configuration. +suite passes vacuously. Execution policy is selected by +`ProjectedBlock.profile_name`, outside the user-writable optionflag namespace. **Sybil's non-overlap-raises invariant.** Two `.. include::` directives naming the same file produce blocks over identical source spans. That is a legitimate @@ -1083,8 +1143,8 @@ conformance test in CI. ### Positive -- Failure locations are correct by construction, including through `.. include::`, - with no `repr_failure` or `reportinfo` override. +- Failure locations are correct by construction, including through `.. include::`; + the default-checker path needs no `repr_failure` or `reportinfo` override. - A block docutils cannot locate degrades to an honest disclaimer instead of a fabricated line, and does not affect its siblings. - Every `--dist` mode works, because there is no shared state to split. @@ -1108,7 +1168,6 @@ conformance test in CI. runner, and a test asserts it. - The plugin now *requires* the built-in doctest plugin rather than blocking it, so `-p no:doctest` is an error rather than a degraded mode. -- Retiring `namespace_items = per-block` is a real feature removal. - The line count does not fall. ### Risks @@ -1137,8 +1196,9 @@ the narrow default set in {doc}`0004-diagnostics-as-data`. This ADR fixes the architecture. Five decisions it defers get their own records: {doc}`0002-runner-conformance-across-cpython` (how the owned loop is proven -equivalent), {doc}`0003-rejecting-per-block-items` (the deprecation -path), {doc}`0004-diagnostics-as-data` (what is reported and what is suppressed), +equivalent), {doc}`0003-rejecting-per-block-items` (why shared per-block items +are rejected), {doc}`0004-diagnostics-as-data` (what is reported and what is +suppressed), {doc}`0005-line-recovery-for-nested-blocks` (the optional last step), and {doc}`0006-pytest-private-api-compatibility` (the quarantine and its matrix). @@ -1148,9 +1208,9 @@ The core produces real {class}`doctest.DocTest` objects holding real {class}`doctest.Example` objects. `Example.source` is the stdlib-normalized executable body — prompts and indentation stripped, trailing newline added, the stripped column recorded in `Example.indent` — not a synthesized wrapper and not -the author's verbatim text, which is what `Block.source` holds. Everything else — -groups, phases, pairing, diagnostics, distribution — is a layer above that fact, -and no layer reaches around another. +the author's verbatim text, which is what `ParsedBlock.source` holds. Everything +else — groups, phases, pairing, diagnostics, distribution — is a layer above that +fact, and no layer reaches around another. The unit that shares a `globs` mapping is the unit pytest schedules. That is the one invariant every other property in this document follows from, and it is not From 31b600c712f8c41070bddb469edc161b85c76992 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 20:59:58 -0500 Subject: [PATCH 23/31] doctest_core(docs[plugins]): Define lifecycle why: Registry construction is a host-neutral extension contract, not a pytest private-API detail, and each supported host freezes contributors at a different point. what: - Add typed contributor, registrar, and snapshot contracts - Define direct, pytest, Sphinx, and xdist lifecycles - Keep mutable construction private and late registration explicit --- docs/adrs/0001-typed-vanilla-doctest-core.md | 28 +-- .../0006-pytest-private-api-compatibility.md | 58 +----- ...0007-host-plugin-registration-lifecycle.md | 194 ++++++++++++++++++ docs/adrs/index.md | 1 + 4 files changed, 213 insertions(+), 68 deletions(-) create mode 100644 docs/adrs/0007-host-plugin-registration-lifecycle.md diff --git a/docs/adrs/0001-typed-vanilla-doctest-core.md b/docs/adrs/0001-typed-vanilla-doctest-core.md index 1925caa..a2b9de1 100644 --- a/docs/adrs/0001-typed-vanilla-doctest-core.md +++ b/docs/adrs/0001-typed-vanilla-doctest-core.md @@ -326,8 +326,8 @@ registration timing is a separate decision; the core contract does not import a pytest hook or a Sphinx application. The direct, pytest and Sphinx lifecycles, including the xdist consistency check, -are specified separately so this record does not mistake a host bootstrap policy -for a core dependency. +are specified in {doc}`0007-host-plugin-registration-lifecycle` so this record +does not mistake a host bootstrap policy for a core dependency. ### Item lifecycle @@ -484,7 +484,6 @@ class ParsedOutput(t.NamedTuple): class BlockKind(t.NamedTuple): - name: str phase: Phase profile_name: str # resolved against the frozen registry, not held here pairs_with: str | None @@ -615,7 +614,8 @@ removing expected output cannot rename every later test. Both `:skipif:` and either gate. `BlockKind` names a profile rather than holding one, so a public type never -contains a private one. The name resolves against the frozen registry. +contains a private implementation. The profile name and the block kind's own +registration name resolve against the frozen registry. **A plan holds no `DocTest`.** {class}`doctest.DocTest` is mutable — the design assigns `globs` to it after construction, and a run mutates that mapping — so a @@ -870,11 +870,11 @@ several different strengths. **The Sphinx promise is narrow, and this record narrows it deliberately.** What is offered is an *extractor over a Sphinx-resolved doctree* — a pure function from a -doctree to blocks, callable from an extension. A full Sphinx host would need an -execution lifecycle and a result channel that neither this record nor -{doc}`0006-pytest-private-api-compatibility` defines, and inventing one would be -the builder that [](#alternatives-rejected) turns down. Until someone specifies -that adapter, the record promises doctree consumption and nothing more. +doctree to blocks, callable from an extension. +{doc}`0007-host-plugin-registration-lifecycle` defines how Sphinx extensions +contribute capabilities, but not a Sphinx execution lifecycle or result channel. +Inventing those would be the builder that [](#alternatives-rejected) turns down. +The record promises doctree consumption and nothing more. ## Constraints @@ -1194,13 +1194,15 @@ the narrow default set in {doc}`0004-diagnostics-as-data`. ## Relationship to other ADRs -This ADR fixes the architecture. Five decisions it defers get their own records: +This ADR fixes the architecture. Six decisions it defers get their own records: {doc}`0002-runner-conformance-across-cpython` (how the owned loop is proven equivalent), {doc}`0003-rejecting-per-block-items` (why shared per-block items are rejected), {doc}`0004-diagnostics-as-data` (what is reported and what is -suppressed), -{doc}`0005-line-recovery-for-nested-blocks` (the optional last step), and -{doc}`0006-pytest-private-api-compatibility` (the quarantine and its matrix). +suppressed), {doc}`0005-line-recovery-for-nested-blocks` (the optional last +step), {doc}`0006-pytest-private-api-compatibility` (the quarantine and its +matrix), and +{doc}`0007-host-plugin-registration-lifecycle` (host registration and freeze +points). ## Final position diff --git a/docs/adrs/0006-pytest-private-api-compatibility.md b/docs/adrs/0006-pytest-private-api-compatibility.md index e72b56d..175bba4 100644 --- a/docs/adrs/0006-pytest-private-api-compatibility.md +++ b/docs/adrs/0006-pytest-private-api-compatibility.md @@ -99,61 +99,9 @@ symbol, and it names the document that triggered it. CI carries a job pinned to the minimum supported pytest and one tracking its prerelease. -### The registry freeze lifecycle - -{doc}`0001-typed-vanilla-doctest-core` depends on a frozen registry and leaves the -lifecycle here. This is it. - -```python -class Contributor(t.Protocol): - """What a plugin implements to extend the core.""" - - def contribute(self, registrar: Registrar) -> None: ... - - -class Registrar(t.Protocol): - def add_block_kind(self, kind: BlockKind, *, replace: bool = False) -> None: ... - def add_document_parser( - self, parser: DocumentParser, *, replace: bool = False - ) -> None: ... - def add_execution_profile( - self, profile: ExecutionProfile, *, replace: bool = False - ) -> None: ... - def add_output_checker( - self, name: str, factory: CheckerFactory, *, replace: bool = False - ) -> None: ... - - -def build_registry(contributors: t.Sequence[Contributor]) -> Registry: ... -``` - -**Duplicate names are an error unless `replace=True`.** Silent last-writer-wins is -what makes the docutils directive table a recurring bug source, and this registry -does not repeat it. Contribution order is: built-ins, then installed plugins in -`pluginmanager` registration order, then explicit contributors — deterministic -under a given plugin set. - -**Freeze points, per host:** - -| Host | Contributors accepted | Frozen at | -|---|---|---| -| direct API | whatever the caller passes to `build_registry` | on return | -| pytest | built-ins, installed plugins, explicit contributors, and **initial/root conftests only** | `pytest_sessionstart` | -| Sphinx | extensions, via `setup(app)` | after extension setup, before reading | - -**Nested conftests may not contribute.** They load during collection, after the -freeze, and a registry that grows while collection runs cannot be the same in -every worker. Registration from a nested conftest is an error naming the file — -not a silent late addition. Fixtures and ordinary pytest hooks in nested conftests -are unaffected; this is only about block kinds, parsers, profiles and checkers. - -**Workers compare a manifest, not node ids.** Matching collection is not -sufficient: two workers can collect identical ids while resolving the same profile -name to different code. So the controller ships a deterministic registry -manifest — sorted `(kind, name, provider, version)` tuples — through worker -configuration, and a worker whose manifest differs fails the session with both -manifests in the message. This is what makes heterogeneous xdist (SSH, socket, -mixed environments) safe rather than assumed-homogeneous. +Registry construction is not a pytest-private-API concern. The host-neutral +contract, pytest hookspec, Sphinx adapter and xdist manifest are specified in +{doc}`0007-host-plugin-registration-lifecycle`. ## Open diff --git a/docs/adrs/0007-host-plugin-registration-lifecycle.md b/docs/adrs/0007-host-plugin-registration-lifecycle.md new file mode 100644 index 0000000..be3b515 --- /dev/null +++ b/docs/adrs/0007-host-plugin-registration-lifecycle.md @@ -0,0 +1,194 @@ +(adr-0007-host-plugin-registration-lifecycle)= + +# ADR 0007: Host plugin registration lifecycle + +Status: Draft +Date: 2026-08-02 + +## Context + +{doc}`0001-typed-vanilla-doctest-core` makes block kinds, document parsers, +execution profiles and output checkers extensible. The core needs one typed +contribution contract, but its hosts discover contributors at different times: +direct callers already have an explicit iterable, pytest loads installed plugins +and conftests in stages, and Sphinx loads extensions before it reads doctrees. + +Treating settings and registrations as one object obscures that difference. +Settings are normalized user input. Registrations are discovered capabilities, +and xdist may discover them in more than one process. The mutable construction +mechanism must not leak into parsing or execution. + +## Decision + +The public boundary consists of `Contributor`, `Registrar`, immutable registration +records and `RegistrySnapshot`. A private builder is the only mutable object. It +accepts contributions, validates them and produces a snapshot; every parser, +projector and runner receives that snapshot explicitly. + +```python +T = t.TypeVar("T") + + +class Provider(t.NamedTuple): + name: str + version: str | None + + +class Registration(t.NamedTuple, t.Generic[T]): + name: str + value: T + provider: Provider + + +class Contributor(t.Protocol): + provider: Provider + + def contribute(self, registrar: Registrar) -> None: ... + + +class Registrar(t.Protocol): + def add_block_kind( + self, name: str, kind: BlockKind, *, replace: bool = False + ) -> None: ... + + def add_document_parser( + self, name: str, parser: DocumentParser, *, replace: bool = False + ) -> None: ... + + def add_execution_profile( + self, name: str, profile: ExecutionProfile, *, replace: bool = False + ) -> None: ... + + def add_output_checker( + self, name: str, factory: CheckerFactory, *, replace: bool = False + ) -> None: ... + + +class RegistrySnapshot(t.NamedTuple): + block_kinds: t.Mapping[str, Registration[BlockKind]] + document_parsers: t.Mapping[str, Registration[DocumentParser]] + execution_profiles: t.Mapping[str, Registration[ExecutionProfile]] + output_checkers: t.Mapping[str, Registration[CheckerFactory]] + + +def build_registry( + contributors: t.Iterable[Contributor] = (), +) -> RegistrySnapshot: ... +``` + +The snapshot fields are read-only `MappingProxyType` views over private copies, +not mutable dictionaries typed as `Mapping`. The builder is discarded after +`freeze()`. While applying each `Contributor`, the builder gives it a registrar +bound to that contributor's `Provider`; registrations cannot claim a different +origin. A contributor retaining that registrar cannot retain a mutation path: +every method raises `RegistryClosedError` after the snapshot is made. + +### Names, collisions and order + +Registration names are case-sensitive ASCII identifiers matching +`[a-z][a-z0-9_.-]*`. The same name may exist in different categories. Within one +category a duplicate is an error naming the category, incumbent provider and +challenger provider unless the challenger passes `replace=True`. Replacement +retains the incumbent's insertion position, so an explicit override cannot +silently reorder parser or profile selection. + +Built-ins register first. Contributor order is then the order supplied by the +host, and calls within a contributor retain program order. The snapshot preserves +that order. Any selection rule that needs precedence uses this declared sequence; +it never sorts by an implementation object's representation or module path. + +For document parsers, overlapping suffix claims are also collisions. They are +accepted only when the challenger uses the incumbent parser's name and passes +`replace=True`; two differently named parsers cannot both win `.md` by incidental +plugin load order. + +## Host adapters + +### Direct API + +Direct callers pass contributors to `build_registry()`. The function registers +built-ins, applies the iterable once, freezes and returns `RegistrySnapshot`. +There is no entry-point scan or process-global default in the core API. + +### pytest + +The pytest adapter publishes its hookspec in `pytest_addhooks`: + +```python +class DoctestCoreHooks: + @pytest.hookspec + def pytest_doctest_core_contributors( + self, + ) -> Contributor | t.Iterable[Contributor] | None: + """Return doctest-core contributors before collection.""" +``` + +Its `pytest_configure(trylast=True)` implementation invokes the hook, flattens +its non-`None` results in pluggy's hook-call order, builds the registry and stores +the snapshot on pytest's stash. Installed plugins and initial conftests are +already registered at that point. The snapshot is therefore ready before +`pytest_sessionstart`, when xdist starts controller nodes, and before collection. + +Nested conftests load during collection and are outside this lifecycle. A +`pytest_plugin_registered` guard detects a late plugin implementing the hookspec +and raises `pytest.UsageError` naming that plugin and the closed registration +phase. Fixtures and unrelated hooks in nested conftests remain valid. + +### Sphinx + +The Sphinx adapter exposes +`add_doctest_core_contributor(app, contributor)`. Extensions call it from their +`setup(app)` function. At `config-inited`, after extension setup and before any +document is read, the adapter emits a `doctest-core-contributors` event, appends +the `Contributor` objects returned by its listeners to the queued contributors, +builds the registry and freezes the snapshot on the application. + +The adapter function is the order-independent path. An extension that connects +directly to the custom event must list the doctest-core extension before itself, +because Sphinx cannot connect a listener to an event that has not been declared. +Calling the adapter after `config-inited` raises `RegistryClosedError` with the +extension name. + +This lifecycle makes the extractor usable on Sphinx-resolved doctrees. It does +not add a builder or claim parity with `sphinx-build -b doctest` execution. + +## xdist consistency + +The controller sends a JSON-safe manifest through `workerinput` from +`pytest_configure_node`. Each worker builds its own snapshot during +`pytest_configure` and compares before collection. The manifest has a schema +version and contains: + +- a JSON-safe projection of normalized `SessionSettings` +- every registry category, name, provider and provider version in declared order +- `doctest.OPTIONFLAGS_BY_NAME`, sorted by flag name + +A mismatch aborts the session with the controller and worker manifests. This is +an extension-set consistency check, not proof that two workers are semantically +identical. Equal provider names and versions do not prove equal source code, and +the manifest does not hash included documents, directive implementations or MyST +plugins. + +Version 1 therefore supports homogeneous worker environments. Equal source +closure and equal installed provider code are preconditions, while xdist's own +identical-collection check remains authoritative for node ids. Stronger support +for deliberately heterogeneous SSH or socket workers would require content or +environment attestation and is deferred. + +## Consequences + +- Core extension authors implement one `Contributor` regardless of host. +- Settings remain serializable inputs; discovered objects remain in the registry. +- Parse and execution code cannot mutate capabilities after collection starts. +- pytest and Sphinx own timing and diagnostics in their native idioms without + leaking their lifecycle types into the core. +- Replacement is possible but visible, attributed and deterministic. +- Supporting heterogeneous xdist workers is explicitly outside the first + contract rather than implied by a weak manifest. + +## Open + +- Whether a later release should add opt-in entry-point discovery to the direct + adapter. The core function remains explicit either way. +- Whether provider code hashes are useful enough to justify the packaging and + editable-install edge cases they introduce. diff --git a/docs/adrs/index.md b/docs/adrs/index.md index 02af2f3..be6a504 100644 --- a/docs/adrs/index.md +++ b/docs/adrs/index.md @@ -35,4 +35,5 @@ the anchor lands on unrelated code while still resolving. 0004-diagnostics-as-data 0005-line-recovery-for-nested-blocks 0006-pytest-private-api-compatibility +0007-host-plugin-registration-lifecycle ``` From a47d33202ae1ad9483c7f9682af183b952e49ce0 Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sun, 2 Aug 2026 21:05:48 -0500 Subject: [PATCH 24/31] doctest_core(docs[analysis]): Correct evidence why: Several supporting claims overstated Sphinx block granularity, xdist determinism, stdlib subclass compatibility, and source offset semantics. what: - Correct Sphinx 9.0 grouping and phase execution claims - Separate markup protocols from stdlib-shaped facades - Qualify xdist manifests and source rewrite metadata --- docs/adrs/0001-typed-vanilla-doctest-core.md | 12 ++-- docs/adrs/0003-rejecting-per-block-items.md | 4 +- .../0005-line-recovery-for-nested-blocks.md | 13 ++-- notes/analyses/10-cpython-doctest.md | 6 +- notes/analyses/12-pytest-xdist.md | 13 ++-- notes/analyses/14-asyncio.md | 8 ++- notes/analyses/15-sphinx-ext-doctest.md | 20 ++++-- notes/analyses/16-docutils-myst.md | 4 +- notes/analyses/20-data-structures.md | 12 ++-- notes/analyses/22-extension-seams.md | 64 ++++++++++--------- .../23-namespace-scope-and-test-identity.md | 10 +-- notes/analyses/90-bibliography.md | 2 +- notes/analyses/README.md | 2 +- 13 files changed, 96 insertions(+), 74 deletions(-) diff --git a/docs/adrs/0001-typed-vanilla-doctest-core.md b/docs/adrs/0001-typed-vanilla-doctest-core.md index a2b9de1..44e813f 100644 --- a/docs/adrs/0001-typed-vanilla-doctest-core.md +++ b/docs/adrs/0001-typed-vanilla-doctest-core.md @@ -945,9 +945,13 @@ against an empty mapping. Worker restarts are on by default. | `is_allowed_version(spec, version)` takes the specifier **first** | [`doctest.py:45`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L45) | | `DocTestBuilder` flips a mutable `self.type` between `"single"` and `"exec"` and reads it through a process-global `doctest.compile` patch | [`doctest.py:310`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L310), [`:549`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L549) | -Sphinx v9.1.0 changes the default group an unargumented block joins. Any -group-naming behaviour matched against "Sphinx" has to say which Sphinx, and -{doc}`0005-line-recovery-for-nested-blocks` proposes moving this floor. +Sphinx 9.0 changed the fallback only for a bare doctest node with no `groups` +attribute: it now uses `doctest_test_doctest_blocks` +([`v9.0.0:463`](https://github.com/sphinx-doc/sphinx/blob/v9.0.0/sphinx/ext/doctest.py#L463)). +An unargumented directive still stamps `groups=["default"]` +([`v9.0.0:94-98`](https://github.com/sphinx-doc/sphinx/blob/v9.0.0/sphinx/ext/doctest.py#L94-L98)). +Compatibility therefore distinguishes directive-produced nodes from bare +`doctest_block` nodes instead of claiming the author-facing default changed. ## Tensions @@ -1069,7 +1073,7 @@ ceiling, and the `import-linter` contract on the leaf. |---|---|---| | [Sybil 10.0.1](https://github.com/simplistix/sybil/tree/10.0.1) | A document is a flat sequence of non-overlapping character spans; every format is a regex lexer; zero runtime dependencies | Format independence at no dependency cost, and a non-overlap invariant that raises on double collection. But one mutable namespace per document with [one independently selectable item per span](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/integration/pytest.py), so `-k` on a later example raises `NameError`. [Node ids are positional](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/sybil.py#L155-L157) (`line:4,column:1`), so adding a paragraph renames every downstream test. No group support at all — a regex cannot see directive options | | [xdoctest v1.3.2](https://github.com/Erotemic/xdoctest/tree/v1.3.2) | Abandon stdlib compatibility; own the parser via `ast`/`tokenize`; make directives structured objects | `ast` parsing and structured directives are real advances — [`REQUIRES`](https://github.com/Erotemic/xdoctest/blob/v1.3.2/src/xdoctest/directive.py#L58) carries *why* a block skipped, which a bool cannot. But it is now building compatibility back, and its permissive got/want defaults silently change tests users wrote for stdlib. It unregisters pytest's doctest plugin outright | -| [pytest-examples v0.0.18](https://github.com/pydantic/pytest-examples/tree/v0.0.18) | Emit the canonical form rather than parse it; rewrite expected output in place | Check-mode and update-mode collapse into one path, and byte offsets plus an invertible dedent scalar are what a data model needs to rewrite source. Composes with pytest by contributing no collector at all — the cheapest correct integration in the survey | +| [pytest-examples v0.0.18](https://github.com/pydantic/pytest-examples/tree/v0.0.18) | Emit the canonical form rather than parse it; rewrite expected output in place | Check-mode and update-mode collapse into one path. Its absolute Python string indices enable source rewriting, although one indent scalar does not invert dedent in general. It composes with pytest by contributing no collector at all — the cheapest correct integration in the survey | | [typeshed](https://github.com/python/typeshed/blob/8c7256c/stdlib/doctest.pyi) | Annotate the 2001 API faithfully | Hands back `Any` at exactly the three extensible points — `globs`, `**options`, `optionflags: int`. Declares `DocTestRunner.test: DocTest` unconditionally although runtime assigns it only inside `run()`, so the stub type-checks a crash | The collective lesson: **namespace scope is not test identity, markup parsing is diff --git a/docs/adrs/0003-rejecting-per-block-items.md b/docs/adrs/0003-rejecting-per-block-items.md index f3c5ebc..2a273dd 100644 --- a/docs/adrs/0003-rejecting-per-block-items.md +++ b/docs/adrs/0003-rejecting-per-block-items.md @@ -66,9 +66,7 @@ outcome contract, rather than papered over with an id that raises when used. ## Consequences Because nothing shipped, there is no migration path to write, no deprecation -warning to add and no downstream grep to run. If PR #87 lands before this -architecture does, this record converts into exactly those things — and that is -the argument for settling the two in order rather than in parallel. +warning to add and no downstream grep to run. ## Open diff --git a/docs/adrs/0005-line-recovery-for-nested-blocks.md b/docs/adrs/0005-line-recovery-for-nested-blocks.md index 7a43251..784f7c7 100644 --- a/docs/adrs/0005-line-recovery-for-nested-blocks.md +++ b/docs/adrs/0005-line-recovery-for-nested-blocks.md @@ -18,9 +18,9 @@ reports its **last** line. A MyST fence reports its **first** line. An `.. include::`-ed block numbers against the *included* file. {doc}`0001-typed-vanilla-doctest-core` handles all of this honestly rather than -approximately: `Block.line` is nullable, the per-front-end meaning is normalized -inside the front-end that knows it, `Block.path` carries the file the text -actually lives in, and a block with no recoverable line propagates +approximately: `ParsedBlock.line` is nullable, the per-front-end meaning is +normalized inside the front-end that knows it, `ParsedBlock.path` carries the +file the text actually lives in, and a block with no recoverable line propagates `DocTest.lineno=None` into pytest's `EXAMPLE LOCATION UNKNOWN` branch. That is correct but not maximal. A nested block's failure says the location is @@ -83,9 +83,10 @@ neither permits docutils 0.22. ## Consequences -The nullable `Block.line` stays. It is not a workaround for this defect; it is -the honest representation of a front-end that may legitimately not know, and -`.. include::` attribution still needs `Block.path` regardless of version. +The nullable `ParsedBlock.line` stays. It is not a workaround for this defect; +it is the honest representation of a front-end that may legitimately not know, +and `.. include::` attribution still needs `ParsedBlock.path` regardless of +version. The line-convention normalization in `markup/` gets *simpler* at the new floor — both reStructuredText and MyST report the first line — but the normalization diff --git a/notes/analyses/10-cpython-doctest.md b/notes/analyses/10-cpython-doctest.md index 6fcb4f8..aa75509 100644 --- a/notes/analyses/10-cpython-doctest.md +++ b/notes/analyses/10-cpython-doctest.md @@ -80,9 +80,9 @@ post-dedent indent shifts every reported column. | Seam | Kind | Documented | |---|---|---| -| `DocTestParser` subclass, injected as `parser=` | nominal | yes | -| `DocTestFinder` subclass, injected as `test_finder=` | nominal | yes | -| `OutputChecker.check_output` / `output_difference`, injected as `checker=` | nominal | yes | +| `parser=` object with the `DocTestParser` methods | structural at runtime, nominal in typeshed | yes | +| `test_finder=` object with the `DocTestFinder` methods | structural at runtime, nominal in typeshed | yes | +| `checker=` object with `check_output` / `output_difference` | structural at runtime, nominal in typeshed | yes | | `report_start`, `report_success`, `report_failure`, `report_unexpected_exception` ([`:1286-1314`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1286-L1314)) | subclass hook | yes | | `register_optionflag` / `OPTIONFLAGS_BY_NAME` ([`:153`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L153)) | process-global registry | yes | | `setUp` / `tearDown` on `DocTestSuite` / `DocFileSuite` | callable param | yes | diff --git a/notes/analyses/12-pytest-xdist.md b/notes/analyses/12-pytest-xdist.md index 40e82ea..d2f6e7b 100644 --- a/notes/analyses/12-pytest-xdist.md +++ b/notes/analyses/12-pytest-xdist.md @@ -61,10 +61,15 @@ Scheduling.schedule() -> send integer index batches to workers The abort path is the constraint that matters most. It is not an exception and it is not loud in the usual sense: the scheduler logs a line, assigns nothing, and -the session ends having run nothing. Any collection-time decision that is not a -pure function of (files on disk, argv, ini) — a timestamp, a PID, a hostname, a -dict iteration order, an evaluated `:skipif:` that depends on the environment — -produces this. +the session ends having run nothing. Collection is not a pure function of files, +argv and ini: included files, directive implementations, MyST plugins and the +discovered registry are inputs too. A timestamp, PID, hostname, unstable iteration +order or evaluated `:skipif:` can make workers diverge when any of those values +affects identity or order. + +A registry manifest can expose differing extension sets before collection. It +cannot prove equal source closure or equal provider code, so it supplements rather +than replaces xdist's identical-node-id check. The crash path is the second. Because only uncompleted items of a work unit are retried, a group whose blocks 1-2 ran before the crash has blocks 3..N re-run diff --git a/notes/analyses/14-asyncio.md b/notes/analyses/14-asyncio.md index a2a679c..685a449 100644 --- a/notes/analyses/14-asyncio.md +++ b/notes/analyses/14-asyncio.md @@ -90,9 +90,11 @@ owner should keep owning the lifecycle. The gap is the first row, and it is the concrete reason `DocutilsDocTestFinder` cannot be handed to `DocTestSuite(test_finder=...)` today -despite exposing a compatible `find()`. ADR 0001's answer — declare `Protocol`s -*and* subclass the stdlib classes nominally — is the cheap way to have both, and -it costs nothing at runtime. +despite exposing a differently shaped `find()`. ADR 0001's answer is to keep the +contracts separate: structural `DocumentParser` implementations for markup, an +exact `DocTestParser` lane for strings, and a `DocTestFinder`-shaped adapter for +Python objects. Nominal subclassing is used only when the replacement preserves +the nominal method signature. ## Anchors diff --git a/notes/analyses/15-sphinx-ext-doctest.md b/notes/analyses/15-sphinx-ext-doctest.md index 538366b..e73f415 100644 --- a/notes/analyses/15-sphinx-ext-doctest.md +++ b/notes/analyses/15-sphinx-ext-doctest.md @@ -1,8 +1,13 @@ # `sphinx.ext.doctest` Pinned at [`v8.2.3`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py), -the version this project resolves. v9.1.0 changes the default group an -unargumented block joins; nothing else below differs between the two. +the version this project resolves. Sphinx 9.0 changed only the fallback for a +bare doctest node with no `groups` attribute: it now uses +`doctest_test_doctest_blocks` +([`v9.0.0:463`](https://github.com/sphinx-doc/sphinx/blob/v9.0.0/sphinx/ext/doctest.py#L463)). +An unargumented Sphinx directive still stamps `groups=["default"]` +([`v9.0.0:94-98`](https://github.com/sphinx-doc/sphinx/blob/v9.0.0/sphinx/ext/doctest.py#L94-L98)), +so its group did not change. ## Classification @@ -66,16 +71,17 @@ DocTestBuilder.test_doc(docname, doctree) [:428] | else groups[name].add_code(code) v per group: ns = {} - | three runners: setup / test / cleanup, sharing one _fakeout - | ONE doctest.DocTest PER BLOCK, each with test.globs = ns - | (assigned AFTER construction, since __init__ copies) + | setup codes -> ONE simulated DocTest containing N Examples + | each ordinary or paired test -> ONE DocTest + | cleanup codes -> ONE simulated DocTest containing N Examples + | all have test.globs = ns after construction, since __init__ copies | runner.run(test, out=..., clear_globs=False) | self.type flipped to "exec" for setup, cleanup, testcode [:549, :608] | to "single" for ordinary doctests [:580] | | if setup fails -> RETURN. Cleanup does not run. [:554-556] v -six integer counters + text streamed to outdir/output.txt +six builder counters + text streamed to outdir/output.txt ``` ## Extension seams @@ -103,7 +109,7 @@ rather than trusting one's own directive classes is the only defence against | `:options:` is accepted only on `doctest` and `testoutput`; on a `testcode` it is an unknown-option error, not a discard | [`:111`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L111), [`:174-180`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L174-L180) | | Cleanup does **not** run when setup fails — the group returns early | [`:554-556`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L554-L556) | | A gated block is dropped during collection — no outcome, id or count | [`:449-450`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L449-L450) | -| `*` means every group the document declares; `default` means no argument was given | [`:428`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L428) onward | +| `*` means every group the document declares; an unargumented directive stamps `default` | [`:94-98`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L94-L98), [`:428`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L428) onward | | Setup runs before tests, cleanup after, whatever order the page writes them | `TestGroup` [`:200-226`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L200-L226) | | `is_allowed_version` takes the **specifier first** | [`:45`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L45) | | `doctest.compile` is rebound process-wide and never restored | [`:310`](https://github.com/sphinx-doc/sphinx/blob/v8.2.3/sphinx/ext/doctest.py#L310) | diff --git a/notes/analyses/16-docutils-myst.md b/notes/analyses/16-docutils-myst.md index 8021966..0ea4cea 100644 --- a/notes/analyses/16-docutils-myst.md +++ b/notes/analyses/16-docutils-myst.md @@ -48,8 +48,8 @@ claim here that does not name a version is a bug in the claim. ADR 0005 All four are real and all four have to be normalized by the front-end that knows which it is. A collector that assumes one convention mis-anchors the other's blocks; a collector that reads `.line` as a number crashes on the nested case. -This is why `Block.line` in ADR 0001 is nullable and `Block.path` is separate from -the collected document. +This is why `ParsedBlock.line` in ADR 0001 is nullable and `ParsedBlock.path` is +separate from the collected document. ## The directive registry diff --git a/notes/analyses/20-data-structures.md b/notes/analyses/20-data-structures.md index 3aae0e9..eeb6a6f 100644 --- a/notes/analyses/20-data-structures.md +++ b/notes/analyses/20-data-structures.md @@ -10,7 +10,7 @@ The disagreements are entirely about which of those four are the same object. |---|---|---|---|---| | CPython `doctest` | `Example` | `DocTest` | `DocTest.globs` | `TestResults` (2 ints + an attribute) | | `_pytest.doctest` | `Example` | `DocTest` = one `Item` | `DocTest.globs`, wiped per item | `MultipleDoctestFailures` → `ReprFailDoctest` | -| `sphinx.ext.doctest` | `TestCode` | one `DocTest` **per block** (`TestGroup` is the batching unit, not the execution unit) | `ns`, assigned post-construction to every block's `DocTest` | six ints + text to a file | +| `sphinx.ext.doctest` | `TestCode` | one `DocTest` per ordinary/paired test; setup and cleanup are each combined into one simulated `DocTest` | `ns`, assigned post-construction to every `DocTest` | six builder counters + text to a file | | Sybil | `Region` | `Example` = one `Item` | `Document.namespace` | truthy return or exception | | xdoctest | `DoctestPart` | own `DocTest` | `global_namespace` | own report objects | | pytest-examples | `CodeExample` | the block, exec'd once | explicit `module_globals=` | captured output, or a rewrite | @@ -24,8 +24,8 @@ nothing for a `-k` to split. Sybil does not: it hands out one pytest item per span over one shared mapping, which is the failure. Three units are easy to conflate for Sphinx specifically, so keep them apart: the -**runner call** is per block, the **shared state** is per group, and the **result** -is six process-wide integers. +**runner call** is per ordinary test but combined per setup or cleanup phase, the +**shared state** is per group, and the **result** is six counters on the builder. ## Field-by-field: what a source unit carries @@ -34,12 +34,14 @@ is six process-wide integers. | source text | `source` | `code` | via `lexemes` | `source` | `source` | | expected output | `want` | paired separately | — | written, not read | a separate `ParsedOutput` | | line | `lineno` (0-based, string-relative) | `lineno` | computed from span | `start_line` | `line` (nullable) | -| byte offsets | — | — | `start`, `end` | `start_index`, `end_index` | — (deferred) | +| string offsets | — | — | `start`, `end` | `start_index`, `end_index` | — (deferred) | | dedent scalar | `indent` | — | `Lexeme.offset` | `indent` | — (deferred) | +| document order | list position | list position | span order | list position | `document_order`, shared with outputs | +| identity order | example index | — | positional node id | parametrization id | `block_ordinal`, runnable blocks only | | kind | — | `type` | inferred from evaluator | `prefix_tags()` | `kind` | | group | — | via `TestGroup` | — | — | `groups` | | options | `options` | `options` | — | — | `options` | -| gate | — | `skipif` on the node | — | — | `skipif` (unevaluated) | +| gate | — | `skipif` on the node | — | — | `skipif` and `pyversion` (unevaluated) | | file | on the `DocTest` | `filename` | on the `Document` | `path` | `path` | | compile mode | — | on the *builder*, mutable | — | always exec | on `ProjectedBlock` | diff --git a/notes/analyses/22-extension-seams.md b/notes/analyses/22-extension-seams.md index ecf02c1..e32a615 100644 --- a/notes/analyses/22-extension-seams.md +++ b/notes/analyses/22-extension-seams.md @@ -7,7 +7,7 @@ ranking is not a matter of taste — each has an observed failure mode. | Mechanism | Where | Failure mode | |---|---|---| -| **Nominal subclassing** — the *type checker* demands the class, the interpreter does not | stdlib `doctest` (via typeshed), `sphinx.ext.doctest` directives | Typed callers are rejected for objects that work fine at runtime. stdlib performs no `isinstance` check on `parser` or `test_finder`; the pressure comes entirely from `doctest.pyi`. One genuine runtime edge: `DocTestSuite` sorts, and `DocTest.__lt__` returns `NotImplemented` for a non-`DocTest`, so a custom finder must return real `DocTest`s | +| **Nominal subclassing** — the *type checker* demands the class, the interpreter does not | stdlib `doctest` (via typeshed), `sphinx.ext.doctest` directives | Typed callers reject structurally valid objects, but subclassing is invalid when the replacement method has a different signature. stdlib performs no `isinstance` check on `parser` or `test_finder`; the pressure comes from `doctest.pyi`. One genuine runtime edge: `DocTestSuite` sorts, and `DocTest.__lt__` returns `NotImplemented` for a non-`DocTest`, so a custom finder must return real `DocTest`s | | **Callable aliases** — `Evaluator = Callable[[Example], str \| None]` | Sybil | Types nothing. You cannot express "this lexer emits `source` and `arguments`", so a mismatched pairing raises `KeyError` at run time. Sybil's own source comments say the payload "could likely be a `TypedDict`" | | **Named registry** — a module dict plus a `register_*` function | `doctest.register_optionflag`, docutils directives, xdoctest's two facades | Process-global mutable state. Order-dependent, unscoped, and silently overwritable | | **`Protocol`** — structural typing | xdoctest's `StdlibExampleLike` | None inherent; but a `Protocol` alone does not satisfy a nominal consumer | @@ -46,37 +46,37 @@ stamps, so either winner is fine. direction and is retiring `set_event_loop_policy` in favour of an explicit `loop_factory=` argument — see [`14-asyncio.md`](14-asyncio.md). -## The nominal/structural trap, and the cheap way out +## The nominal/structural trap requires separate adapters -Typeshed's signatures demand classes: `DocTestFinder.__init__(parser: DocTestParser -= ...)`, `DocTestSuite(test_finder: DocTestFinder | None)`. The interpreter does -not — a duck-typed parser or finder runs fine. So a `Protocol` gives a third party -structural typing, and a nominal subclass keeps you *type-checkable* against the -stub. +Typeshed's signatures demand classes: `DocTestFinder.__init__(parser: +DocTestParser = ...)`, `DocTestSuite(test_finder: DocTestFinder | None)`. The +interpreter does not perform an `isinstance` check, but it still calls the exact +stdlib methods. A markup parser whose `parse(text, path, *, settings)` returns a +doctree cannot override `DocTestParser.parse(string, name)` returning alternating +strings and examples. A mypy probe rejects that override, and subclassing does not +make the runtime calls compatible. -The subclass does not buy passability. That is a matter of matching the call -signature: a finder whose `find()` takes a string first cannot be handed to -`DocTestSuite`, which passes a module — and subclassing does not change that. - -Doing both costs nothing: +The solution is two contracts, not one class wearing two names: ```python -class Frontend(t.Protocol): +class DocumentParser(t.Protocol): suffixes: t.ClassVar[frozenset[str]] def parse( - self, text: str, path: pathlib.Path, *, settings: Settings - ) -> ParseResult: ... + self, text: str, path: pathlib.Path, *, settings: ParseSettings + ) -> tuple[nodes.document, tuple[Diagnostic, ...]]: ... -class DocutilsDocTestParser( - doctest.DocTestParser -): # nominal, for DocFileSuite(parser=...) - ... +class StdlibParserFacade(doctest.DocTestParser): + def parse( + self, string: str, name: str = "" + ) -> list[str | doctest.Example]: ... ``` -Sybil, xdoctest and stdlib each took one half. Taking both is the only reason this -is worth stating as a rule. +The markup `DocumentParser` has reStructuredText and MyST implementations. Plain +strings use the exact stdlib parser lane, and Python objects use a separate +`DocTestFinder`-shaped adapter. A nominal façade is useful only where it preserves +the nominal API's signature. ## What deserves a seam, and what does not @@ -85,19 +85,21 @@ needs it. Applied to the candidates that came up: | Candidate | Verdict | |---|---| -| `BlockKind` registry | **Yes.** Turns "a new block kind" from an edit to a method branching on string literals into adding a tuple. Which docutils node classes a kind arrives as stays in `markup/`, not on `BlockKind`, so the leaf keeps its stdlib-only contract | -| Output checker injection | **Yes.** The highest-demand seam, and the one Sybil closed entirely by hard-coding `checker=OutputChecker()` — adding one there requires subclassing two classes | -| `Frontend` protocol | **Yes.** Four implementations ship on day one (rST, MyST, text, Python docstrings) | -| `ExecutionProfile` | **Yes, and contributable.** [PR #59](https://github.com/git-pull/gp-libs/pull/59) adds top-level `await` — a second execution policy a `Literal["single", "exec"]` cannot express. The *type* is part of the public contributor protocol, since a pluggable core that cannot accept a new execution policy is not pluggable; the registry holding them stays private | +| `BlockKind` registry | **Yes.** Turns "a new block kind" from an edit to a method branching on string literals into adding a record. Which docutils node classes a kind arrives as stays in `markup/`, not on `BlockKind` | +| Output checker injection | **Yes.** The highest-demand seam, and the one Sybil closed entirely by hard-coding `checker=OutputChecker()`. One factory supplies the checker used for both comparison and failure explanation | +| `DocumentParser` protocol | **Yes.** Two markup implementations ship initially: reStructuredText and MyST. Plain text and Python objects use the separate stdlib-shaped lanes | +| stdlib parser and finder façades | **Yes, but separate.** They preserve `DocTestParser` and `DocTestFinder` signatures instead of subclassing them with incompatible markup or object-discovery methods | +| `ExecutionProfile` | **Yes, and contributable.** [PR #59](https://github.com/git-pull/gp-libs/pull/59) adds top-level `await` — a second execution policy a `Literal["single", "exec"]` cannot express. The type and its immutable registration are public; the mutable builder is not | | A per-example observer protocol | **No.** pytest gets failures through `report_*` and `MultipleDoctestFailures`; the CLI uses `summarize()`. No third consumer | -| A registry *object* with builders, digests and manifests | **No object, but there is a freeze.** The contributed block kinds, front ends and profiles are resolved into an immutable mapping alongside the frozen `Settings` facets, before collection begins; registering after that is an error. That is what ADR 0001's collection contract means by *frozen registry* — not a class with its own lifecycle | -| Entry-point plugin discovery | **Optional, behind the contributor protocol.** Not the hazard it was once called: what breaks determinism is nondeterministic *contribution*, not discovery. But "identical across workers" holds only for a homogeneous local run — xdist supports SSH and socket workers — so a heterogeneous session needs a registry manifest comparison or an explicit homogeneous-plugin requirement | +| `RegistrySnapshot` and a builder | **Yes, with asymmetric visibility.** The snapshot and immutable registration records are public inputs; the mutable builder is private. Settings remain a separate value. Direct, pytest and Sphinx adapters own their freeze points in ADR 0007 | +| Entry-point plugin discovery | **Optional, behind the contributor protocol.** Discovery is not itself nondeterministic, but matching manifests only compare declared extension sets. Version 1 requires homogeneous provider code and source closure; heterogeneous-worker attestation is deferred | The distinction those two rows turn on is worth stating once: **discovery is not -the hazard, nondeterminism is.** A registry populated identically in every process -satisfies [`12-pytest-xdist.md`](12-pytest-xdist.md)'s identical-collection -requirement no matter how it was populated; a registry whose contents depend on -which conftest happened to run, or on the environment, does not. +the hazard; divergent inputs are.** A registry populated with matching names and +versions can still resolve to different code, and matching registries can still +parse different included files. ADR 0007's manifest catches declared extension +drift; [`12-pytest-xdist.md`](12-pytest-xdist.md)'s identical-collection check +remains authoritative for node ids. ## Anchors diff --git a/notes/analyses/23-namespace-scope-and-test-identity.md b/notes/analyses/23-namespace-scope-and-test-identity.md index 0167fb6..5ce0aa1 100644 --- a/notes/analyses/23-namespace-scope-and-test-identity.md +++ b/notes/analyses/23-namespace-scope-and-test-identity.md @@ -30,8 +30,8 @@ node id" is really *no* id — every test block in a group shares one method to swallow the resulting `IndexError`. So the execution shape is partly well-trodden; the contribution is making it addressable. -The bottom-right cell is where the silent failure lives, and two shipped projects -are in it. +The bottom-right cell is where the silent failure lives. One surveyed project +ships it, and one open proposal implements it with guards. Sybil is there **unacknowledged**: one `Document.namespace` shared by reference, one pytest item per region. `pytest -k` on an example whose predecessor bound a @@ -39,8 +39,10 @@ name raises `NameError`, and nothing in its documentation says so. PR #87's proposed `per-block` mode would sit there **acknowledged and guarded** — the guards being an xdist scheduler substitution, a scheduler refusal and a -run-twice refusal. It has not shipped; {doc}`ADR 0003 <../../docs/adrs/0003-rejecting-per-block-items>` rejects the shape. Those guards are the reason `_worker_count`, `_shared_page`, -`_is_page` and `_splitting_scheduler` exist at all. +run-twice refusal. It has not shipped; +{doc}`ADR 0003 <../../docs/adrs/0003-rejecting-per-block-items>` rejects the +shape. Those guards are the reason `_worker_count`, `_shared_page`, `_is_page` +and `_splitting_scheduler` exist at all. The bottom-left cell gives per-block reporting *and* an unsplittable sharing unit, and it needs no guards, because there is nothing to split. diff --git a/notes/analyses/90-bibliography.md b/notes/analyses/90-bibliography.md index 07bcc0a..e404049 100644 --- a/notes/analyses/90-bibliography.md +++ b/notes/analyses/90-bibliography.md @@ -159,5 +159,5 @@ Typed surface: [`typeshed stubs/docutils`](https://github.com/python/typeshed/tr |---|---|---| | Sybil | [`10.0.1`](https://github.com/simplistix/sybil/tree/10.0.1) | [`sybil.py:155-157`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/sybil.py#L155-L157) (positional ids) · [`document.py`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/document.py) (one namespace, non-overlap invariant) · [`integration/pytest.py`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/integration/pytest.py) (one item per region) · [`region.py`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/region.py) · [`testing.py`](https://github.com/simplistix/sybil/blob/10.0.1/src/sybil/testing.py) (public extension-test helpers) | | xdoctest | [`v1.3.2`](https://github.com/Erotemic/xdoctest/tree/v1.3.2) | [`directive.py:58`](https://github.com/Erotemic/xdoctest/blob/v1.3.2/src/xdoctest/directive.py#L58) (`REQUIRES` carries its reason) · [`plugin.py`](https://github.com/Erotemic/xdoctest/blob/v1.3.2/src/xdoctest/plugin.py) (unregisters pytest's doctest plugin) | -| pytest-examples | [`v0.0.18`](https://github.com/pydantic/pytest-examples/tree/v0.0.18) | [`find_examples.py`](https://github.com/pydantic/pytest-examples/blob/v0.0.18/pytest_examples/find_examples.py) · [`run_code.py`](https://github.com/pydantic/pytest-examples/blob/v0.0.18/pytest_examples/run_code.py) · [`modify_files.py`](https://github.com/pydantic/pytest-examples/blob/v0.0.18/pytest_examples/modify_files.py) (byte offsets, invertible dedent, unguarded splice) | +| pytest-examples | [`v0.0.18`](https://github.com/pydantic/pytest-examples/tree/v0.0.18) | [`find_examples.py`](https://github.com/pydantic/pytest-examples/blob/v0.0.18/pytest_examples/find_examples.py) · [`run_code.py`](https://github.com/pydantic/pytest-examples/blob/v0.0.18/pytest_examples/run_code.py) · [`modify_files.py`](https://github.com/pydantic/pytest-examples/blob/v0.0.18/pytest_examples/modify_files.py) (Python string offsets, recorded indent, unguarded splice) | | typeshed | [`8c7256c`](https://github.com/python/typeshed/tree/8c7256c) | [`stdlib/doctest.pyi`](https://github.com/python/typeshed/blob/8c7256c/stdlib/doctest.pyi) | diff --git a/notes/analyses/README.md b/notes/analyses/README.md index 74df866..72b5187 100644 --- a/notes/analyses/README.md +++ b/notes/analyses/README.md @@ -31,7 +31,7 @@ What exactly does each of them require, and where do they contradict each other? | pytest | `pytest-dev/pytest` | `9.1.1` | | pytest-xdist | `pytest-dev/pytest-xdist` | `v3.8.0` | | pytest-asyncio | `pytest-dev/pytest-asyncio` | `v1.4.0` | -| Sphinx | `sphinx-doc/sphinx` | `v8.2.3` — what this project resolves. `v9.1.0` is cited only where a difference is called out | +| Sphinx | `sphinx-doc/sphinx` | `v8.2.3` — what this project resolves. `v9.0.0` is cited only for the bare-node group fallback change | | MyST-Parser | `executablebooks/MyST-Parser` | `v5.1.0` on Python ≥ 3.11; `v4.0.1` below | | Sybil | `simplistix/sybil` | `10.0.1` | | xdoctest | `Erotemic/xdoctest` | `v1.3.2` | From 19e2813a1d7389fc81b60e7b394bebb5b4fe5d1e Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sat, 8 Aug 2026 07:03:38 -0500 Subject: [PATCH 25/31] doctest_core(feat[core]): Add typed core why: Test the ADR boundaries against stock prompt semantics, typed markup extraction, shared groups, and extended execution. what: - Add immutable contracts, settings, models, and registry snapshots - Extract and project reStructuredText and MyST into group recipes - Run prompt and exec lanes with typed results and cleanup policy - Cover projection, runtime, registry, and Sphinx doctree behavior --- src/doctest_core/__init__.py | 131 +++++ src/doctest_core/contracts.py | 247 +++++++++ src/doctest_core/markup.py | 579 +++++++++++++++++++++ src/doctest_core/model.py | 401 ++++++++++++++ src/doctest_core/project.py | 313 +++++++++++ src/doctest_core/py.typed | 1 + src/doctest_core/registry.py | 313 +++++++++++ src/doctest_core/runner.py | 601 +++++++++++++++++++++ src/doctest_core/settings.py | 56 ++ tests/test_doctest_core_projection.py | 538 +++++++++++++++++++ tests/test_doctest_core_registry.py | 144 ++++++ tests/test_doctest_core_runner.py | 717 ++++++++++++++++++++++++++ tests/test_doctest_core_sphinx.py | 88 ++++ 13 files changed, 4129 insertions(+) create mode 100644 src/doctest_core/__init__.py create mode 100644 src/doctest_core/contracts.py create mode 100644 src/doctest_core/markup.py create mode 100644 src/doctest_core/model.py create mode 100644 src/doctest_core/project.py create mode 100644 src/doctest_core/py.typed create mode 100644 src/doctest_core/registry.py create mode 100644 src/doctest_core/runner.py create mode 100644 src/doctest_core/settings.py create mode 100644 tests/test_doctest_core_projection.py create mode 100644 tests/test_doctest_core_registry.py create mode 100644 tests/test_doctest_core_runner.py create mode 100644 tests/test_doctest_core_sphinx.py diff --git a/src/doctest_core/__init__.py b/src/doctest_core/__init__.py new file mode 100644 index 0000000..82b24bc --- /dev/null +++ b/src/doctest_core/__init__.py @@ -0,0 +1,131 @@ +"""Typed, host-neutral doctest planning and execution.""" + +from __future__ import annotations + +from .contracts import ( + CheckerFactory, + Contributor, + DocumentParser, + ExceptionPolicy, + ExecutionProfile, + ExecutionRuntime, + Provider, + Registrar, + Registration, + RuntimeOutcome, + RuntimeSettings, +) +from .markup import ( + DoctestDirective, + MockTabDirective, + MystDocumentParser, + RstDocumentParser, + TestcleanupDirective, + TestcodeDirective, + TestoutputDirective, + TestsetupDirective, + ensure_directives_registered, + extract_blocks, + parse_document, +) +from .model import ( + BlockKind, + BlockResult, + Counts, + Diagnostic, + Errored, + ExampleRecipe, + ExpectedOutput, + Failed, + Failure, + GroupPlan, + GroupResult, + ParsedBlock, + ParsedOutput, + ParseResult, + Passed, + Phase, + ProjectedBlock, + Skipped, + SkipReason, +) +from .project import project +from .registry import ( + RegistryClosedError, + RegistryCollisionError, + RegistryError, + RegistrySnapshot, + build_registry, +) +from .runner import ( + DefaultExceptionPolicy, + ExecExecutionProfile, + ExecRuntime, + PromptExecutionProfile, + PromptRuntime, + materialize, + reset_globs, + run_group, +) +from .settings import ParseSettings, ProjectionSettings, RunSettings + +__all__ = [ + "BlockKind", + "BlockResult", + "CheckerFactory", + "Contributor", + "Counts", + "DefaultExceptionPolicy", + "Diagnostic", + "DoctestDirective", + "DocumentParser", + "Errored", + "ExampleRecipe", + "ExceptionPolicy", + "ExecExecutionProfile", + "ExecRuntime", + "ExecutionProfile", + "ExecutionRuntime", + "ExpectedOutput", + "Failed", + "Failure", + "GroupPlan", + "GroupResult", + "MockTabDirective", + "MystDocumentParser", + "ParseResult", + "ParseSettings", + "ParsedBlock", + "ParsedOutput", + "Passed", + "Phase", + "ProjectedBlock", + "ProjectionSettings", + "PromptExecutionProfile", + "PromptRuntime", + "Provider", + "Registrar", + "Registration", + "RegistryClosedError", + "RegistryCollisionError", + "RegistryError", + "RegistrySnapshot", + "RstDocumentParser", + "RunSettings", + "RuntimeOutcome", + "RuntimeSettings", + "SkipReason", + "Skipped", + "TestcleanupDirective", + "TestcodeDirective", + "TestoutputDirective", + "TestsetupDirective", + "build_registry", + "ensure_directives_registered", + "extract_blocks", + "materialize", + "parse_document", + "project", + "reset_globs", + "run_group", +] diff --git a/src/doctest_core/contracts.py b/src/doctest_core/contracts.py new file mode 100644 index 0000000..9873d9e --- /dev/null +++ b/src/doctest_core/contracts.py @@ -0,0 +1,247 @@ +"""Public structural contracts for doctest-core extensions.""" + +from __future__ import annotations + +import contextlib +import dataclasses +import doctest +import pathlib +import typing as t + +from docutils import nodes + +from .model import BlockKind, Diagnostic, Failure +from .settings import ParseSettings + + +class Provider(t.NamedTuple): + """Identity attached to every contributed capability. + + Attributes + ---------- + name : str + Stable provider name. + version : str or None + Provider version when one is available. + + >>> Provider("example", "1").name + 'example' + """ + + name: str + version: str | None + + +T = t.TypeVar("T") + + +@dataclasses.dataclass(frozen=True, slots=True) +class Registration(t.Generic[T]): + """One immutable, attributed registry entry. + + Attributes + ---------- + name : str + Case-sensitive registration name. + value : T + Registered capability. + provider : Provider + Contributor that supplied the value. + """ + + name: str + value: T + provider: Provider + + +class RuntimeOutcome(t.NamedTuple): + """Outcome returned by an execution runtime. + + Attributes + ---------- + results : doctest.TestResults + Standard attempted and failed totals. + failures : tuple of Failure + Failures retained for host-native reporting. + skipped : int + Examples reached and skipped by the runtime. + """ + + results: doctest.TestResults + failures: tuple[Failure, ...] + skipped: int + + +class ExceptionPolicy(t.Protocol): + """Classify exceptions that must escape a runtime's doctest loop.""" + + def should_propagate(self, error: BaseException) -> bool: + """Return whether ``error`` belongs to the embedding host. + + >>> isinstance(KeyboardInterrupt(), BaseException) + True + """ + ... + + def is_abort(self, error: BaseException) -> bool: + """Return whether ``error`` must outrank every recorded outcome. + + >>> isinstance(KeyboardInterrupt(), BaseException) + True + """ + ... + + +class RuntimeSettings(t.NamedTuple): + """Resolved objects and policy used by one execution runtime. + + Attributes + ---------- + optionflags : int + Runner-level doctest option bitmask. + continue_on_failure : bool + Continue after an example mismatch. + checker : doctest.OutputChecker + Fresh checker used for comparison and explanation. + exception_policy : ExceptionPolicy + Host-neutral classifier for exceptions that must propagate. + """ + + optionflags: int + continue_on_failure: bool + checker: doctest.OutputChecker + exception_policy: ExceptionPolicy + + +class CheckerFactory(t.Protocol): + """Construct a fresh output checker for an execution runtime.""" + + def __call__(self) -> doctest.OutputChecker: + r"""Return a checker used for comparison and failure explanation. + + >>> doctest.OutputChecker().check_output("42\n", "42\n", 0) + True + """ + ... + + +class ExecutionRuntime(t.Protocol): + """Attempt-local executor for materialized stock doctests.""" + + def run(self, test: doctest.DocTest) -> RuntimeOutcome: + """Execute ``test`` without clearing its shared globals. + + >>> test = doctest.DocTest([], {}, "example", "example.rst", 0, "") + >>> test.name + 'example' + """ + ... + + +class ExecutionProfile(t.Protocol): + """Immutable factory for attempt-local execution runtimes.""" + + def open( + self, + settings: RuntimeSettings, + ) -> contextlib.AbstractContextManager[ExecutionRuntime]: + """Open a runtime whose resources live for one group attempt. + + >>> issubclass(contextlib.AbstractContextManager, object) + True + """ + ... + + +class DocumentParser(t.Protocol): + """Parse one markup language into a docutils document.""" + + suffixes: t.ClassVar[frozenset[str]] + + def parse( + self, + text: str, + path: pathlib.Path, + *, + settings: ParseSettings, + ) -> tuple[nodes.document, tuple[Diagnostic, ...]]: + """Parse ``text`` while retaining normalized diagnostics. + + >>> pathlib.Path("guide.rst").suffix + '.rst' + """ + ... + + +class Registrar(t.Protocol): + """Provider-bound mutation surface available during contribution.""" + + def add_block_kind( + self, + name: str, + kind: BlockKind, + *, + replace: bool = False, + ) -> None: + """Register a block kind. + + >>> BlockKind.__name__ + 'BlockKind' + """ + ... + + def add_document_parser( + self, + name: str, + parser: DocumentParser, + *, + replace: bool = False, + ) -> None: + """Register a document parser and its suffix claims. + + >>> ".rst" in frozenset({".rst"}) + True + """ + ... + + def add_execution_profile( + self, + name: str, + profile: ExecutionProfile, + *, + replace: bool = False, + ) -> None: + """Register an execution-profile factory. + + >>> "prompt".islower() + True + """ + ... + + def add_output_checker( + self, + name: str, + factory: CheckerFactory, + *, + replace: bool = False, + ) -> None: + """Register an output-checker factory. + + >>> callable(doctest.OutputChecker) + True + """ + ... + + +class Contributor(t.Protocol): + """Host-neutral source of attributed registry entries.""" + + provider: Provider + + def contribute(self, registrar: Registrar) -> None: + """Add capabilities through the provider-bound ``registrar``. + + >>> Provider("example", None).version is None + True + """ + ... diff --git a/src/doctest_core/markup.py b/src/doctest_core/markup.py new file mode 100644 index 0000000..910bc5d --- /dev/null +++ b/src/doctest_core/markup.py @@ -0,0 +1,579 @@ +"""Docutils and MyST front ends for doctest core.""" + +from __future__ import annotations + +import doctest +import io +import pathlib +import re +import textwrap +import typing as t +import warnings + +from docutils import nodes +from docutils.frontend import OptionParser +from docutils.parsers.rst import Directive, Parser, directives +from docutils.utils import new_document + +from .model import Diagnostic, ParsedBlock, ParsedOutput, ParseResult +from .settings import ParseSettings + +if t.TYPE_CHECKING: + from .contracts import DocumentParser + from .registry import RegistrySnapshot + + +_BLANKLINE_RE = re.compile(r"^\s*", re.MULTILINE) +_DOCTEST_OPTION_RE = re.compile(r"[ \t]*#\s*doctest:.+$", re.MULTILINE) +_TEST_KINDS = frozenset( + {"doctest", "testsetup", "testcleanup", "testcode", "testoutput"}, +) +_REQUIRED_DIRECTIVES = (*sorted(_TEST_KINDS), "tab") + + +class _TestDirective(Directive): + """Create nodes carrying the Sphinx doctest attribute vocabulary.""" + + has_content = True + required_arguments = 0 + optional_arguments = 1 + final_argument_whitespace = True + + def run(self) -> list[nodes.Node]: + """Return one node stamped with inert doctest metadata.""" + code = "\n".join(self.content) + test = code + trim = "no-trim-doctest-flags" not in self.options + if self.name == "doctest" and trim: + display = _BLANKLINE_RE.sub("", code) + display = _DOCTEST_OPTION_RE.sub("", display) + else: + display = code + + node_type: type[nodes.TextElement] = nodes.literal_block + hidden = "hide" in self.options + if self.name in {"testsetup", "testcleanup"} or hidden: + node_type = nodes.comment + + groups = ( + [item.strip() for item in self.arguments[0].split(",")] + if self.arguments + else ["default"] + ) + node = node_type( + display, + display, + testnodetype=self.name, + groups=groups, + hidden=hidden, + ) + source, line = self.state_machine.get_source_and_line(self.lineno) + node.source = source + node.line = line + node["testline"] = self.content_offset + 1 + if test != display: + node["test"] = test + if self.name == "doctest": + node["language"] = "pycon3" + + node["options"] = self._parse_options() + for key in ("skipif", "pyversion"): + if key in self.options: + node[key] = self.options[key] + if "trim-doctest-flags" in self.options: + node["trim_flags"] = True + elif "no-trim-doctest-flags" in self.options: + node["trim_flags"] = False + return [node] + + def _parse_options(self) -> dict[int, bool]: + """Parse Sphinx ``:options:`` into doctest's integer flag mapping.""" + parsed: dict[int, bool] = {} + value = self.options.get("options") + if not isinstance(value, str): + return parsed + for option in value.replace(",", " ").split(): + if len(option) < 2 or option[0] not in "+-": + self.state.document.reporter.warning( + f"missing '+' or '-' in '{option}' option", + line=self.lineno, + ) + continue + flag = doctest.OPTIONFLAGS_BY_NAME.get(option[1:]) + if flag is None: + self.state.document.reporter.warning( + f"'{option[1:]}' is not a valid doctest option", + line=self.lineno, + ) + continue + parsed[flag] = option[0] == "+" + return parsed + + +class TestsetupDirective(_TestDirective): + """Parse a Sphinx-compatible ``testsetup`` directive.""" + + option_spec: t.ClassVar = { + "hide": directives.flag, + "skipif": directives.unchanged_required, + } + + +class TestcleanupDirective(_TestDirective): + """Parse a Sphinx-compatible ``testcleanup`` directive.""" + + option_spec: t.ClassVar = { + "hide": directives.flag, + "skipif": directives.unchanged_required, + } + + +class DoctestDirective(_TestDirective): + """Parse a Sphinx-compatible ``doctest`` directive.""" + + option_spec: t.ClassVar = { + "hide": directives.flag, + "no-trim-doctest-flags": directives.flag, + "options": directives.unchanged, + "pyversion": directives.unchanged_required, + "skipif": directives.unchanged_required, + "trim-doctest-flags": directives.flag, + } + + +class TestcodeDirective(_TestDirective): + """Parse a Sphinx-compatible ``testcode`` directive.""" + + option_spec: t.ClassVar = { + "hide": directives.flag, + "no-trim-doctest-flags": directives.flag, + "pyversion": directives.unchanged_required, + "skipif": directives.unchanged_required, + "trim-doctest-flags": directives.flag, + } + + +class TestoutputDirective(_TestDirective): + """Parse a Sphinx-compatible ``testoutput`` directive.""" + + option_spec: t.ClassVar = { + "hide": directives.flag, + "no-trim-doctest-flags": directives.flag, + "options": directives.unchanged, + "pyversion": directives.unchanged_required, + "skipif": directives.unchanged_required, + "trim-doctest-flags": directives.flag, + } + + +class MockTabDirective(Directive): + """Parse tab content when sphinx-inline-tabs is not installed.""" + + has_content = True + + def run(self) -> list[nodes.Node]: + """Return a transparent container around nested content.""" + self.assert_has_content() + content = nodes.container("", is_div=True, classes=["tab-content"]) + self.state.nested_parse(self.content, self.content_offset, content) + return [content] + + +_DIRECTIVE_TYPES: t.Mapping[str, type[Directive]] = { + "doctest": DoctestDirective, + "testsetup": TestsetupDirective, + "testcleanup": TestcleanupDirective, + "testcode": TestcodeDirective, + "testoutput": TestoutputDirective, + "tab": MockTabDirective, +} + + +def ensure_directives_registered() -> None: + """Register missing standalone directives without replacing Sphinx's. + + >>> ensure_directives_registered() + >>> all(name in directives._directives for name in _REQUIRED_DIRECTIVES) + True + """ + registry = t.cast(dict[str, t.Any], directives.__dict__["_directives"]) + for name, directive in _DIRECTIVE_TYPES.items(): + if name not in registry: + directives.register_directive(name, directive) + + +def _settings(parser_type: type[Parser]) -> t.Any: + """Build quiet docutils settings while retaining reporter messages.""" + with warnings.catch_warnings(): + warnings.simplefilter("ignore", DeprecationWarning) + settings = OptionParser(components=(parser_type,)).get_default_values() + settings.report_level = 5 + settings.halt_level = 6 + settings.warning_stream = io.StringIO() + return settings + + +def _diagnostic_from_message(message: nodes.system_message) -> Diagnostic: + """Convert one docutils system message to a typed diagnostic.""" + level_number = int(message.get("level", 1)) + level: t.Literal["info", "warning", "error"] + if level_number >= 3: + level = "error" + elif level_number >= 2: + level = "warning" + else: + level = "info" + text = message.astext() + code = None + if "Unknown interpreted text role" in text or "No role entry for" in text: + code = "docutils.unknown-role" + elif "Unknown directive type" in text or "No directive entry for" in text: + code = "docutils.unknown-directive" + return Diagnostic( + level=level, + code=code, + message=text, + path=pathlib.Path(message.source or ""), + line=message.line, + ) + + +def _attach_diagnostics(document: nodes.document) -> list[Diagnostic]: + """Attach an observer and return its mutable capture list.""" + captured: list[Diagnostic] = [] + + def observe(message: nodes.system_message) -> None: + captured.append(_diagnostic_from_message(message)) + + document.reporter.attach_observer(observe) + return captured + + +class RstDocumentParser: + """Parse reStructuredText into a docutils document.""" + + suffixes: t.ClassVar = frozenset({".rst", ".txt"}) + + def parse( + self, + text: str, + path: pathlib.Path, + *, + settings: ParseSettings, + ) -> tuple[nodes.document, tuple[Diagnostic, ...]]: + """Parse text and return its doctree and diagnostics.""" + del settings + ensure_directives_registered() + parser = Parser() + document = new_document(str(path), settings=_settings(Parser)) + captured = _attach_diagnostics(document) + parser.parse(text, document) + return document, tuple(captured) + + +class MystDocumentParser: + """Parse MyST Markdown into a docutils document.""" + + suffixes: t.ClassVar = frozenset({".md"}) + + def parse( + self, + text: str, + path: pathlib.Path, + *, + settings: ParseSettings, + ) -> tuple[nodes.document, tuple[Diagnostic, ...]]: + """Parse text and return its doctree and diagnostics.""" + del settings + from myst_parser.config.main import MdParserConfig + from myst_parser.mdit_to_docutils.base import DocutilsRenderer + from myst_parser.parsers.docutils_ import Parser as MystParser + from myst_parser.parsers.mdit import create_md_parser + + ensure_directives_registered() + document = new_document(str(path), settings=_settings(MystParser)) + captured = _attach_diagnostics(document) + parser = create_md_parser( + MdParserConfig(commonmark_only=False), + DocutilsRenderer, + ) + parser.options["document"] = document + parser.render(text) + _stamp_myst_source_lines(document, text) + return document, tuple(captured) + + +def _normalize_body(value: str) -> str: + """Dedent a node body and preserve the executable trailing newline.""" + body = textwrap.dedent(value).strip("\n") + return f"{body}\n" if body else "" + + +def _groups(node: nodes.Element) -> tuple[str, ...]: + """Narrow a node's untyped group attribute.""" + value: object = node.get("groups", ()) + if isinstance(value, str): + return (value,) + if isinstance(value, (list, tuple)): + values = t.cast(list[object] | tuple[object, ...], value) + return tuple(str(item) for item in values) + return () + + +def _options(node: nodes.Element) -> t.Mapping[int, bool]: + """Narrow and copy a node's untyped option mapping.""" + value: object = node.get("options", {}) + if not isinstance(value, dict): + return {} + options = t.cast(dict[object, object], value) + return { + flag: bool(enabled) + for flag, enabled in options.items() + if isinstance(flag, int) + } + + +def _optional_text(node: nodes.Element, name: str) -> str | None: + """Validate an optional text attribute at the typed-model boundary. + + >>> node = nodes.literal_block("", "", skipif="enabled") + >>> _optional_text(node, "skipif") + 'enabled' + """ + value: object = node.get(name) + if value is None or isinstance(value, str): + return value + message = f"{name} node attribute must be str or None, got {type(value).__name__}" + raise TypeError(message) + + +def _node_kind(node: nodes.Node) -> str | None: + """Return the registered kind represented by a doctree node.""" + if isinstance(node, nodes.Element): + stamped = node.get("testnodetype") + if isinstance(stamped, str) and stamped: + return stamped + if isinstance(node, nodes.doctest_block): + return "doctest" + if isinstance(node, nodes.literal_block) and re.match( + doctest.DocTestParser._EXAMPLE_RE, # type: ignore[attr-defined] + node.astext(), + ): + return "doctest" + return None + + +def _stamp_myst_source_lines(doctree: nodes.document, text: str) -> None: + r"""Retain root-document body lines lost from MyST literal-block nodes. + + >>> tree = new_document("guide.md") + >>> node = nodes.literal_block(">>> 1 + 1\n2\n", ">>> 1 + 1\n2\n") + >>> node.source, node.line = "guide.md", 1 + >>> tree += node + >>> _stamp_myst_source_lines(tree, "```\n>>> 1 + 1\n2\n```\n") + >>> node["doctest_core_line"] + 2 + """ + lines = text.splitlines() + document_source = doctree.current_source or doctree.get("source") + for node in doctree.findall(nodes.literal_block): + if _node_kind(node) is None or node.line is None: + continue + if document_source and node.source != document_source: + continue + source = _normalize_body(str(node.get("test", node.astext()))) + if not source: + continue + first_line = source.splitlines()[0].strip() + for index in range(max(node.line - 1, 0), len(lines)): + if lines[index].strip() == first_line: + node["doctest_core_line"] = index + 1 + break + + +def _node_line(node: nodes.Element, source: str) -> int | None: + """Normalize parser-specific line conventions to the first body line.""" + if ":docstring of " in pathlib.Path(node.source or "").name: + return None + core_line = node.get("doctest_core_line") + if isinstance(core_line, int): + return core_line + suffix = pathlib.Path(node.source or "").suffix + if suffix == ".md" and isinstance(node, nodes.literal_block): + if node.line is None: + return None + local_testline = node.get("testline") + if isinstance(local_testline, int): + return node.line + local_testline + return node.line + 1 + testline = node.get("testline") + if isinstance(testline, int): + return testline + if node.line is None: + return None + if isinstance(node, nodes.doctest_block) and suffix != ".md": + return node.line - len(source.rstrip("\n").splitlines()) + 1 + return node.line + + +def extract_blocks( + doctree: nodes.document, + *, + settings: ParseSettings | None = None, + registry: RegistrySnapshot | None = None, +) -> ParseResult: + """Extract typed doctest records from an existing resolved doctree. + + >>> from docutils import nodes + >>> tree = nodes.document("", "") + >>> extract_blocks(tree).blocks + () + """ + if registry is None: + from .registry import build_registry + + registry = build_registry() + settings = settings or ParseSettings() + output_kinds = frozenset( + registration.value.pairs_with + for registration in registry.block_kinds.values() + if registration.value.pairs_with is not None + ) + blocks: list[ParsedBlock] = [] + outputs: list[ParsedOutput] = [] + block_ordinal = 0 + document_order = 0 + for node in doctree.findall(): + kind = _node_kind(node) + if kind is None or not isinstance(node, nodes.Element): + continue + source = _normalize_body(str(node.get("test", node.astext()))) + path = pathlib.Path(node.source or doctree.source or "") + line = _node_line(node, source) + groups = _groups(node) + options = _options(node) + skipif = _optional_text(node, "skipif") + pyversion = _optional_text(node, "pyversion") + if kind in output_kinds: + outputs.append( + ParsedOutput( + kind=kind, + text=source, + path=path, + line=line, + document_order=document_order, + groups=groups, + options=options, + skipif=skipif, + pyversion=pyversion, + ), + ) + elif kind in registry.block_kinds: + blocks.append( + ParsedBlock( + kind=kind, + source=source, + path=path, + line=line, + document_order=document_order, + block_ordinal=block_ordinal, + groups=groups, + options=options, + skipif=skipif, + pyversion=pyversion, + hidden=isinstance(node, nodes.comment) + or bool(node.get("hidden", False)), + ), + ) + block_ordinal += 1 + document_order += 1 + + diagnostics = tuple( + diagnostic + for node in doctree.findall(nodes.system_message) + if (diagnostic := _diagnostic_from_message(node)).code + not in settings.suppressed_diagnostics + ) + return ParseResult(tuple(blocks), tuple(outputs), diagnostics) + + +def _parser_for_path( + path: pathlib.Path, + registry: RegistrySnapshot, +) -> DocumentParser: + """Select one parser by its declared suffix.""" + matches = [ + registration.value + for registration in registry.document_parsers.values() + if path.suffix in registration.value.suffixes + ] + if len(matches) != 1: + message = f"expected one document parser for suffix {path.suffix!r}" + raise ValueError(message) + return matches[0] + + +def _merge_diagnostics( + parser_diagnostics: t.Iterable[Diagnostic], + tree_diagnostics: t.Iterable[Diagnostic], + settings: ParseSettings, +) -> tuple[Diagnostic, ...]: + """Merge parser channels and prefer tree copies with source provenance. + + >>> diagnostic = Diagnostic("error", "example", "bad", pathlib.Path("x"), 1) + >>> merged = _merge_diagnostics((diagnostic,), (diagnostic,), ParseSettings()) + >>> (len(merged), merged[0].line) + (1, 1) + """ + tree = [ + diagnostic + for diagnostic in tree_diagnostics + if diagnostic.code not in settings.suppressed_diagnostics + ] + unmatched_tree = [ + (diagnostic.level, diagnostic.code, diagnostic.message) for diagnostic in tree + ] + merged: list[Diagnostic] = [] + for diagnostic in parser_diagnostics: + if diagnostic.code in settings.suppressed_diagnostics: + continue + key = (diagnostic.level, diagnostic.code, diagnostic.message) + try: + matched_index = unmatched_tree.index(key) + except ValueError: + merged.append(diagnostic) + else: + unmatched_tree.pop(matched_index) + merged.extend(tree) + return tuple(merged) + + +def parse_document( + text: str, + path: pathlib.Path, + *, + settings: ParseSettings | None = None, + registry: RegistrySnapshot | None = None, +) -> ParseResult: + r"""Parse and extract a documentation page through the frozen registry. + + >>> parse_document('>>> 1 + 1\n2\n', pathlib.Path('x.rst')).blocks[0].kind + 'doctest' + """ + if registry is None: + from .registry import build_registry + + registry = build_registry() + settings = settings or ParseSettings() + parser = _parser_for_path(path, registry) + doctree, parser_diagnostics = parser.parse(text, path, settings=settings) + extracted = extract_blocks(doctree, settings=settings, registry=registry) + return ParseResult( + blocks=extracted.blocks, + outputs=extracted.outputs, + diagnostics=_merge_diagnostics( + parser_diagnostics, + extracted.diagnostics, + settings, + ), + ) diff --git a/src/doctest_core/model.py b/src/doctest_core/model.py new file mode 100644 index 0000000..23eb542 --- /dev/null +++ b/src/doctest_core/model.py @@ -0,0 +1,401 @@ +"""Host-neutral values carried through the doctest pipeline.""" + +from __future__ import annotations + +import doctest +import enum +import pathlib +import typing as t + + +class Phase(enum.IntEnum): + """Execution phase for a projected block. + + Setup and cleanup surround test blocks while retaining source order within + each phase. + + >>> Phase.SETUP < Phase.TEST < Phase.CLEANUP + True + """ + + SETUP = 0 + TEST = 1 + CLEANUP = 2 + + +class Diagnostic(t.NamedTuple): + """A parser diagnostic that can cross host boundaries. + + Attributes + ---------- + level : {"info", "warning", "error"} + Normalized severity. + code : str or None + Stable classifier when the parser provides one. + message : str + Human-readable explanation. + path : pathlib.Path + Source containing the diagnostic. + line : int or None + One-based source line when available. + """ + + level: t.Literal["info", "warning", "error"] + code: str | None + message: str + path: pathlib.Path + line: int | None + + +class ParsedBlock(t.NamedTuple): + """An inert runnable block extracted from markup. + + Attributes + ---------- + kind : str + Registered block-kind name. + source : str + Dedented author text. + path : pathlib.Path + File containing the block. + line : int or None + One-based source line when available. + document_order : int + Position in the shared block-and-output stream. + block_ordinal : int + Stable position among runnable blocks. + groups : tuple of str + Author-declared Sphinx group names. + options : mapping of int to bool + Doctest option overrides. + skipif : str or None + Unevaluated skip expression. + pyversion : str or None + Unevaluated PEP 440 version specifier. + hidden : bool + Whether the markup hides the block from rendered output. + """ + + kind: str + source: str + path: pathlib.Path + line: int | None + document_order: int + block_ordinal: int + groups: tuple[str, ...] + options: t.Mapping[int, bool] + skipif: str | None + pyversion: str | None + hidden: bool + + +class ParsedOutput(t.NamedTuple): + """An inert expected-output body. + + Attributes + ---------- + kind : str + Output-kind name referenced by a registered block kind. + text : str + Expected output text. + path : pathlib.Path + File containing the output. + line : int or None + One-based source line when available. + document_order : int + Position in the shared block-and-output stream. + groups : tuple of str + Author-declared Sphinx group names. + options : mapping of int to bool + Doctest option overrides. + skipif : str or None + Unevaluated skip expression. + pyversion : str or None + Unevaluated PEP 440 version specifier. + """ + + kind: str + text: str + path: pathlib.Path + line: int | None + document_order: int + groups: tuple[str, ...] + options: t.Mapping[int, bool] + skipif: str | None + pyversion: str | None + + +class ParseResult(t.NamedTuple): + """Complete typed output of parsing one document. + + Attributes + ---------- + blocks : tuple of ParsedBlock + Runnable blocks in source order. + outputs : tuple of ParsedOutput + Expected-output records in source order. + diagnostics : tuple of Diagnostic + Parser diagnostics retained as data. + """ + + blocks: tuple[ParsedBlock, ...] + outputs: tuple[ParsedOutput, ...] + diagnostics: tuple[Diagnostic, ...] + + +class BlockKind(t.NamedTuple): + """Projection policy registered for one markup block kind. + + Attributes + ---------- + phase : Phase + Phase in which the block executes. + profile_name : str + Registered execution-profile name. + pairs_with : str or None + Expected-output kind paired with the block, if any. + """ + + phase: Phase + profile_name: str + pairs_with: str | None + + +class ExpectedOutput(t.NamedTuple): + """Expected output paired with an executable block. + + Attributes + ---------- + text : str + Expected output text. + options : mapping of int to bool + Output-specific doctest option overrides. + skipif : str or None + Unevaluated skip expression. + pyversion : str or None + Unevaluated PEP 440 version specifier. + """ + + text: str + options: t.Mapping[int, bool] + skipif: str | None + pyversion: str | None + + +class ExampleRecipe(t.NamedTuple): + """Fields needed to rebuild one stock :class:`doctest.Example`. + + Attributes + ---------- + source : str + Python source ending in a newline. + want : str + Expected output ending in a newline when non-empty. + exc_msg : str or None + Expected exception detail. + lineno : int + Zero-based line relative to the block. + indent : int + Prompt indentation. + options : mapping of int to bool + Inline doctest option overrides. + """ + + source: str + want: str + exc_msg: str | None + lineno: int + indent: int + options: t.Mapping[int, bool] + + +class ProjectedBlock(t.NamedTuple): + """Immutable recipe for one runnable source block. + + Attributes + ---------- + phase : Phase + Execution phase. + name : str + Unique, machine-independent doctest name. + block_ordinal : int + Stable position among runnable source blocks. + examples : tuple of ExampleRecipe + Recipes materialized into stock examples per attempt. + docstring : str + Source used by failure renderers. + filename : str + Source filename shown by doctest and host adapters. + lineno : int or None + One-based block line when known. + options : mapping of int to bool + Block-level doctest option overrides. + profile_name : str + Registered execution-profile name. + skipif : str or None + Unevaluated skip expression. + pyversion : str or None + Unevaluated PEP 440 version specifier. + expected : ExpectedOutput or None + Paired expected output for an executable-code block. + """ + + phase: Phase + name: str + block_ordinal: int + examples: tuple[ExampleRecipe, ...] + docstring: str + filename: str + lineno: int | None + options: t.Mapping[int, bool] + profile_name: str + skipif: str | None + pyversion: str | None + expected: ExpectedOutput | None + + +class GroupPlan(t.NamedTuple): + """Structurally immutable execution plan for one Sphinx group. + + Attributes + ---------- + group : str + Author-facing group name. + blocks : tuple of ProjectedBlock + Block recipes in execution order. + seed : mapping of str to Any + Initial names copied into each attempt's live mapping. + """ + + group: str + blocks: tuple[ProjectedBlock, ...] + seed: t.Mapping[str, t.Any] + + +Failure: t.TypeAlias = doctest.DocTestFailure | doctest.UnexpectedException + + +class Counts(t.NamedTuple): + """Failure, attempt, and skip counts for one block. + + Attributes + ---------- + failed : int + Number of mismatches, including failures hidden from detailed reports. + attempted : int + Number of examples doctest attempted. + skipped : int + Number of examples skipped by inline or block policy. + """ + + failed: int + attempted: int + skipped: int + + +class SkipReason(t.NamedTuple): + """Structured explanation for a skipped block. + + Attributes + ---------- + kind : {"skipif", "inline-flag", "pyversion"} + Policy that skipped the block. + detail : str + Gate expression, option name, specifier, or profile explanation. + """ + + kind: t.Literal["skipif", "inline-flag", "pyversion"] + detail: str + + +class Passed(t.NamedTuple): + """Successful block result. + + Attributes + ---------- + block : ProjectedBlock + Block that ran. + counts : Counts + Failure, attempt, and skip totals. + """ + + block: ProjectedBlock + counts: Counts + + +class Failed(t.NamedTuple): + """Doctest-comparison failures from one block. + + Attributes + ---------- + block : ProjectedBlock + Block that ran. + counts : Counts + Failure, attempt, and skip totals. + failures : tuple of Failure + Failures retained in example order. + checker : doctest.OutputChecker + Exact checker instance that compared and must explain the failures. + """ + + block: ProjectedBlock + counts: Counts + failures: tuple[Failure, ...] + checker: doctest.OutputChecker + + +class Skipped(t.NamedTuple): + """Block skipped by a run-time policy. + + Attributes + ---------- + block : ProjectedBlock + Block that did not run. + counts : Counts + Failure, attempt, and skip totals. + reason : SkipReason + Structured skip explanation. + """ + + block: ProjectedBlock + counts: Counts + reason: SkipReason + + +class Errored(t.NamedTuple): + """Infrastructure or gate error from one block. + + Attributes + ---------- + block : ProjectedBlock + Block whose execution failed outside doctest comparison. + error : BaseException + Original exception retained for the host adapter. + """ + + block: ProjectedBlock + error: BaseException + + +BlockResult: t.TypeAlias = Passed | Failed | Skipped | Errored + + +class GroupResult(t.NamedTuple): + """Result of one group attempt. + + Attributes + ---------- + group : str + Author-facing group name. + blocks : tuple of BlockResult + Results in execution order. + primary : BaseException or None + Body exception a host should re-raise. + secondary : tuple of BaseException + Additional exceptions, such as cleanup errors after body failure. + """ + + group: str + blocks: tuple[BlockResult, ...] + primary: BaseException | None + secondary: tuple[BaseException, ...] diff --git a/src/doctest_core/project.py b/src/doctest_core/project.py new file mode 100644 index 0000000..dd6200d --- /dev/null +++ b/src/doctest_core/project.py @@ -0,0 +1,313 @@ +"""Pure projection from parsed records to immutable group plans.""" + +from __future__ import annotations + +import doctest +import pathlib +import types +import typing as t + +from .model import ( + ExampleRecipe, + ExpectedOutput, + GroupPlan, + ParsedBlock, + ParsedOutput, + ParseResult, + Phase, + ProjectedBlock, +) +from .settings import ProjectionSettings + +if t.TYPE_CHECKING: + from .registry import RegistrySnapshot + + +class _GroupKey(t.NamedTuple): + """Collision-free identity for one projected group. + + Attributes + ---------- + author_name : str or None + Declared group name, or ``None`` for a generated block group. + block_ordinal : int or None + Runnable-block ordinal for a generated group, or ``None`` for a + declared group. + """ + + author_name: str | None + block_ordinal: int | None + + +def _read_only(values: t.Mapping[t.Any, t.Any]) -> t.Mapping[t.Any, t.Any]: + """Copy a mapping behind a read-only view.""" + return types.MappingProxyType(dict(values)) + + +def _groups_for_block( + block: ParsedBlock, + settings: ProjectionSettings, +) -> tuple[_GroupKey, ...]: + """Resolve an ordinary block's non-wildcard group names.""" + if block.groups: + return tuple(_GroupKey(group, None) for group in block.groups) + if settings.ungrouped == "default": + return (_GroupKey("default", None),) + return (_GroupKey(None, block.block_ordinal),) + + +def _test_groups( + parsed: ParseResult, + settings: ProjectionSettings, + registry: RegistrySnapshot, +) -> tuple[_GroupKey, ...]: + """Return executable group names in first declaration order.""" + groups: list[_GroupKey] = [] + has_wildcard = False + for block in parsed.blocks: + registration = registry.block_kinds.get(block.kind) + if registration is None or registration.value.phase is not Phase.TEST: + continue + for group in _groups_for_block(block, settings): + if group.author_name == "*": + has_wildcard = True + elif group not in groups: + groups.append(group) + if has_wildcard and not groups: + groups.append(_GroupKey("default", None)) + return tuple(groups) + + +def _group_labels(groups: tuple[_GroupKey, ...]) -> dict[_GroupKey, str]: + """Derive concise unique display labels without changing group identity.""" + reserved = {group.author_name for group in groups if group.author_name is not None} + labels: dict[_GroupKey, str] = {} + used: set[str] = set() + for group in groups: + if group.author_name is not None: + label = group.author_name + else: + base = f"block-{group.block_ordinal}" + label = base + suffix = 1 + while label in reserved or label in used: + marker = "anonymous" if suffix == 1 else f"anonymous-{suffix}" + label = f"{base}[{marker}]" + suffix += 1 + labels[group] = label + used.add(label) + return labels + + +def _destinations( + block: ParsedBlock, + *, + settings: ProjectionSettings, + groups: tuple[_GroupKey, ...], +) -> tuple[_GroupKey, ...]: + """Expand one block's declared groups against document groups.""" + declared = _groups_for_block(block, settings) + if any(group.author_name == "*" for group in declared): + return groups + return tuple(group for group in declared if group in groups) + + +def _output_matches(output: ParsedOutput, group: _GroupKey) -> bool: + """Return whether an expected-output record belongs to ``group``.""" + return "*" in output.groups or ( + group.author_name is not None and group.author_name in output.groups + ) + + +def _paired_output( + block: ParsedBlock, + output_kind: str, + group: _GroupKey, + parsed: ParseResult, + settings: ProjectionSettings, + groups: tuple[_GroupKey, ...], + registry: RegistrySnapshot, +) -> ParsedOutput | None: + """Find the latest output before the next test block in this group.""" + later_blocks = [ + candidate.document_order + for candidate in parsed.blocks + if candidate.document_order > block.document_order + and candidate.kind in registry.block_kinds + and registry.block_kinds[candidate.kind].value.phase is Phase.TEST + and group + in _destinations( + candidate, + settings=settings, + groups=groups, + ) + ] + boundary = min(later_blocks, default=2**63 - 1) + matches = [ + output + for output in parsed.outputs + if output.kind == output_kind + and block.document_order < output.document_order < boundary + and _output_matches(output, group) + ] + return matches[-1] if matches else None + + +def _prompt_recipes(block: ParsedBlock) -> tuple[ExampleRecipe, ...]: + """Project through the unmodified standard-library parser.""" + test = doctest.DocTestParser().get_doctest( + block.source, + {}, + "", + str(block.path), + 0, + ) + return tuple( + ExampleRecipe( + source=example.source, + want=example.want, + exc_msg=example.exc_msg, + lineno=example.lineno, + indent=example.indent, + options=_read_only(example.options), + ) + for example in test.examples + ) + + +def _exec_recipe(block: ParsedBlock) -> tuple[ExampleRecipe, ...]: + """Represent one prompt-free body as a stock example recipe.""" + return ( + ExampleRecipe( + source=block.source, + want="", + exc_msg=None, + lineno=0, + indent=0, + options=_read_only({}), + ), + ) + + +def _project_block( + block: ParsedBlock, + *, + group: _GroupKey, + group_label: str, + document_name: str, + parsed: ParseResult, + settings: ProjectionSettings, + groups: tuple[_GroupKey, ...], + registry: RegistrySnapshot, +) -> ProjectedBlock: + """Create a fresh group-qualified recipe for one parsed block.""" + kind = registry.block_kinds[block.kind].value + output = ( + _paired_output( + block, + kind.pairs_with, + group, + parsed, + settings, + groups, + registry, + ) + if kind.pairs_with + else None + ) + expected = ( + ExpectedOutput( + text=output.text, + options=_read_only(output.options), + skipif=output.skipif, + pyversion=output.pyversion, + ) + if output is not None + else None + ) + examples = ( + _prompt_recipes(block) if kind.profile_name == "prompt" else _exec_recipe(block) + ) + stem = pathlib.PurePath(document_name).stem + return ProjectedBlock( + phase=kind.phase, + name=f"{stem}::{group_label}[{block.block_ordinal}]", + block_ordinal=block.block_ordinal, + examples=examples, + docstring=block.source, + filename=str(block.path), + lineno=None if block.line is None else max(block.line - 1, 0), + options=_read_only(block.options), + profile_name=kind.profile_name, + skipif=block.skipif, + pyversion=block.pyversion, + expected=expected, + ) + + +def project( + parsed: ParseResult, + *, + document_name: str, + settings: ProjectionSettings | None = None, + registry: RegistrySnapshot | None = None, + seed: t.Mapping[str, t.Any] | None = None, +) -> tuple[GroupPlan, ...]: + """Project inert records into one immutable plan per shared-state group. + + Grouping and pairing are pure: no user code, filesystem access, docutils, + or host lifecycle object crosses this boundary. + + >>> from .model import ParseResult + >>> project(ParseResult((), (), ()), document_name="empty.rst") + () + """ + if registry is None: + from .registry import build_registry + + registry = build_registry() + settings = settings or ProjectionSettings() + groups = _test_groups(parsed, settings, registry) + labels = _group_labels(groups) + by_group: dict[_GroupKey, list[tuple[int, ProjectedBlock]]] = { + group: [] for group in groups + } + for block in parsed.blocks: + if block.kind not in registry.block_kinds: + continue + for group in _destinations(block, settings=settings, groups=groups): + projected = _project_block( + block, + group=group, + group_label=labels[group], + document_name=document_name, + parsed=parsed, + settings=settings, + groups=groups, + registry=registry, + ) + if projected.phase is Phase.TEST and not projected.examples: + continue + by_group[group].append( + ( + block.document_order, + projected, + ), + ) + + frozen_seed = _read_only(seed or {}) + return tuple( + GroupPlan( + group=labels[group], + blocks=tuple( + block + for _, block in sorted( + by_group[group], + key=lambda entry: (entry[1].phase, entry[0]), + ) + ), + seed=frozen_seed, + ) + for group in groups + if any(block.phase is Phase.TEST for _, block in by_group[group]) + ) diff --git a/src/doctest_core/py.typed b/src/doctest_core/py.typed new file mode 100644 index 0000000..8b13789 --- /dev/null +++ b/src/doctest_core/py.typed @@ -0,0 +1 @@ + diff --git a/src/doctest_core/registry.py b/src/doctest_core/registry.py new file mode 100644 index 0000000..6ec014b --- /dev/null +++ b/src/doctest_core/registry.py @@ -0,0 +1,313 @@ +"""Deterministic construction of immutable doctest-core registries.""" + +from __future__ import annotations + +import doctest +import re +import types +import typing as t + +from .contracts import ( + CheckerFactory, + Contributor, + DocumentParser, + ExecutionProfile, + Provider, + Registrar, + Registration, +) +from .model import BlockKind, Phase + + +class RegistryError(ValueError): + """Base error for invalid registry construction.""" + + +class RegistryClosedError(RegistryError): + """Raised when a retained registrar is used after registry freeze.""" + + +class RegistryCollisionError(RegistryError): + """Raised when a registration would implicitly replace another.""" + + +class RegistrySnapshot(t.NamedTuple): + """Read-only capability set consumed by pipeline stages. + + Attributes + ---------- + block_kinds : mapping of str to Registration[BlockKind] + Projection policies in declaration order. + document_parsers : mapping of str to Registration[DocumentParser] + Markup parsers in declaration order. + execution_profiles : mapping of str to Registration[ExecutionProfile] + Runtime factories in declaration order. + output_checkers : mapping of str to Registration[CheckerFactory] + Checker factories in declaration order. + """ + + block_kinds: t.Mapping[str, Registration[BlockKind]] + document_parsers: t.Mapping[str, Registration[DocumentParser]] + execution_profiles: t.Mapping[str, Registration[ExecutionProfile]] + output_checkers: t.Mapping[str, Registration[CheckerFactory]] + + +_NAME_PATTERN = re.compile(r"[a-z][a-z0-9_.-]*\Z", flags=re.ASCII) +_BUILTIN_PROVIDER = Provider(name="builtin", version=None) +U = t.TypeVar("U") + + +def _read_only( + entries: dict[str, Registration[U]], +) -> t.Mapping[str, Registration[U]]: + return types.MappingProxyType(dict(entries)) + + +class _RegistryBuilder: + def __init__(self) -> None: + self.block_kinds: dict[str, Registration[BlockKind]] = {} + self.document_parsers: dict[str, Registration[DocumentParser]] = {} + self.execution_profiles: dict[str, Registration[ExecutionProfile]] = {} + self.output_checkers: dict[str, Registration[CheckerFactory]] = {} + self.closed = False + + def registrar(self, provider: Provider) -> _BoundRegistrar: + self._require_open() + return _BoundRegistrar(self, provider) + + def close(self) -> None: + self.closed = True + + def freeze(self) -> RegistrySnapshot: + self._require_open() + self._validate_references() + self.close() + return RegistrySnapshot( + block_kinds=_read_only(self.block_kinds), + document_parsers=_read_only(self.document_parsers), + execution_profiles=_read_only(self.execution_profiles), + output_checkers=_read_only(self.output_checkers), + ) + + def _validate_references(self) -> None: + """Reject block policies that cannot resolve unambiguously.""" + for name, registration in self.block_kinds.items(): + kind = registration.value + if kind.profile_name not in self.execution_profiles: + message = ( + f"block kind {name!r} from provider " + f"{registration.provider.name!r} references missing execution " + f"profile {kind.profile_name!r}" + ) + raise RegistryError(message) + if kind.pairs_with is None: + continue + self._validate_name(kind.pairs_with) + output_collision = self.block_kinds.get(kind.pairs_with) + if output_collision is not None: + message = ( + f"block kind {name!r} from provider " + f"{registration.provider.name!r} pairs with {kind.pairs_with!r}, " + "which is also a runnable block kind from provider " + f"{output_collision.provider.name!r}" + ) + raise RegistryCollisionError(message) + + def add( + self, + category: str, + entries: dict[str, Registration[U]], + name: str, + value: U, + provider: Provider, + *, + replace: bool, + ) -> None: + self._require_open() + self._validate_name(name) + incumbent = entries.get(name) + if incumbent is not None and not replace: + msg = ( + f"{category} {name!r} from provider " + f"{incumbent.provider.name!r} already exists; provider " + f"{provider.name!r} must pass replace=True" + ) + raise RegistryCollisionError(msg) + entries[name] = Registration(name, value, provider) + + def add_document_parser( + self, + name: str, + parser: DocumentParser, + provider: Provider, + *, + replace: bool, + ) -> None: + self._require_open() + self._validate_name(name) + for incumbent_name, incumbent in self.document_parsers.items(): + if incumbent_name == name: + continue + overlap = parser.suffixes & incumbent.value.suffixes + if overlap: + suffixes = ", ".join(sorted(overlap)) + msg = ( + f"document parser {name!r} from provider " + f"{provider.name!r} overlaps {incumbent_name!r} from " + f"provider {incumbent.provider.name!r} for {suffixes}" + ) + raise RegistryCollisionError(msg) + self.add( + "document parser", + self.document_parsers, + name, + parser, + provider, + replace=replace, + ) + + def _require_open(self) -> None: + if self.closed: + msg = "registry registration is closed" + raise RegistryClosedError(msg) + + @staticmethod + def _validate_name(name: str) -> None: + if _NAME_PATTERN.fullmatch(name) is None: + msg = f"invalid registry name {name!r}; expected [a-z][a-z0-9_.-]*" + raise RegistryError(msg) + + +class _BoundRegistrar: + def __init__(self, builder: _RegistryBuilder, provider: Provider) -> None: + self._builder = builder + self._provider = provider + + def add_block_kind( + self, + name: str, + kind: BlockKind, + *, + replace: bool = False, + ) -> None: + self._builder.add( + "block kind", + self._builder.block_kinds, + name, + kind, + self._provider, + replace=replace, + ) + + def add_document_parser( + self, + name: str, + parser: DocumentParser, + *, + replace: bool = False, + ) -> None: + self._builder.add_document_parser( + name, + parser, + self._provider, + replace=replace, + ) + + def add_execution_profile( + self, + name: str, + profile: ExecutionProfile, + *, + replace: bool = False, + ) -> None: + self._builder.add( + "execution profile", + self._builder.execution_profiles, + name, + profile, + self._provider, + replace=replace, + ) + + def add_output_checker( + self, + name: str, + factory: CheckerFactory, + *, + replace: bool = False, + ) -> None: + self._builder.add( + "output checker", + self._builder.output_checkers, + name, + factory, + self._provider, + replace=replace, + ) + + +def _register_builtins(registrar: Registrar) -> None: + # Import implementations only while constructing a registry. This keeps the + # foundational contracts independent of parsing and execution modules. + from .markup import MystDocumentParser, RstDocumentParser + from .runner import ExecExecutionProfile, PromptExecutionProfile + + registrar.add_block_kind( + "doctest", + BlockKind(Phase.TEST, "prompt", None), + ) + registrar.add_block_kind( + "testsetup", + BlockKind(Phase.SETUP, "exec", None), + ) + registrar.add_block_kind( + "testcleanup", + BlockKind(Phase.CLEANUP, "exec", None), + ) + registrar.add_block_kind( + "testcode", + BlockKind(Phase.TEST, "exec", "testoutput"), + ) + registrar.add_document_parser("rst", RstDocumentParser()) + registrar.add_document_parser("myst", MystDocumentParser()) + registrar.add_execution_profile("prompt", PromptExecutionProfile()) + registrar.add_execution_profile("exec", ExecExecutionProfile()) + registrar.add_output_checker("stdlib", doctest.OutputChecker) + + +def build_registry( + contributors: t.Iterable[Contributor] = (), +) -> RegistrySnapshot: + """Build and freeze a deterministic capability snapshot. + + Built-ins retain their declaration order and contributors are applied once + in the order supplied by the host. + + Parameters + ---------- + contributors : iterable of Contributor + Explicit host-discovered contributions. + + Returns + ------- + RegistrySnapshot + Immutable registry mappings and attributed records. + + Raises + ------ + RegistryCollisionError + If a contribution replaces a capability without explicit permission. + + Examples + -------- + >>> tuple(build_registry().output_checkers) + ('stdlib',) + """ + builder = _RegistryBuilder() + try: + _register_builtins(builder.registrar(_BUILTIN_PROVIDER)) + for contributor in contributors: + contributor.contribute(builder.registrar(contributor.provider)) + return builder.freeze() + finally: + builder.close() diff --git a/src/doctest_core/runner.py b/src/doctest_core/runner.py new file mode 100644 index 0000000..a7bc4a8 --- /dev/null +++ b/src/doctest_core/runner.py @@ -0,0 +1,601 @@ +"""Fresh materialization and host-neutral group execution.""" + +from __future__ import annotations +import __future__ + +import contextlib +import doctest +import io +import sys +import traceback +import types +import typing as t + +from packaging.specifiers import SpecifierSet +from packaging.version import Version + +from .contracts import ( + ExceptionPolicy, + ExecutionRuntime, + RuntimeOutcome, + RuntimeSettings, +) +from .model import ( + BlockResult, + Counts, + Errored, + Failed, + Failure, + GroupPlan, + GroupResult, + Passed, + Phase, + ProjectedBlock, + Skipped, + SkipReason, +) +from .settings import RunSettings + +if t.TYPE_CHECKING: + from doctest import _Out + + from .registry import RegistrySnapshot + + +class DefaultExceptionPolicy: + """Preserve the standard-library doctest exception boundary.""" + + def should_propagate(self, error: BaseException) -> bool: + """Return whether ``error`` must escape doctest handling. + + >>> DefaultExceptionPolicy().should_propagate(KeyboardInterrupt()) + True + >>> DefaultExceptionPolicy().should_propagate(ValueError()) + False + """ + return isinstance(error, KeyboardInterrupt) + + def is_abort(self, error: BaseException) -> bool: + """Return whether ``error`` must outrank block and cleanup results. + + >>> DefaultExceptionPolicy().is_abort(SystemExit()) + False + >>> DefaultExceptionPolicy().is_abort(ValueError()) + False + """ + return isinstance(error, KeyboardInterrupt) + + +def _results(failed: int, attempted: int, skipped: int) -> doctest.TestResults: + """Construct ``TestResults`` across CPython's supported shapes.""" + try: + constructor = t.cast(t.Any, doctest.TestResults) + return t.cast( + doctest.TestResults, + constructor(failed, attempted, skipped=skipped), + ) + except TypeError: + return doctest.TestResults(failed, attempted) + + +def _effective_flags(defaults: int, options: t.Mapping[int, bool]) -> int: + """Apply per-example boolean overrides to an option bitmask.""" + flags = defaults + for flag, enabled in options.items(): + if enabled: + flags |= flag + else: + flags &= ~flag + return flags + + +def _compile_flags(globs: t.Mapping[str, t.Any]) -> int: + """Return future-feature compiler flags already active in ``globs``. + + >>> _compile_flags({}) + 0 + """ + flags = 0 + for name in __future__.all_feature_names: + feature: t.Any = getattr(__future__, name) + compiler_flag: int = feature.compiler_flag + if globs.get(name) is feature: + flags |= compiler_flag + return flags + + +def _captured_output(stream: io.StringIO) -> str: + r"""Return captured stdout with doctest's implied trailing newline. + + >>> stream = io.StringIO("partial") + >>> _captured_output(stream) + 'partial\n' + """ + output = stream.getvalue() + if output and not output.endswith("\n"): + return f"{output}\n" + return output + + +class _CollectingRunner(doctest.DocTestRunner): + """Stock prompt runner with pytest-neutral failure collection.""" + + def __init__(self, settings: RuntimeSettings) -> None: + super().__init__( + checker=settings.checker, + optionflags=settings.optionflags, + ) + self.original_optionflags = settings.optionflags + self.continue_on_failure = settings.continue_on_failure + self.exception_policy = settings.exception_policy + self.recorded_failures: list[Failure] = [] + + def report_failure( + self, + out: _Out, + test: doctest.DocTest, + example: doctest.Example, + got: str, + ) -> None: + """Retain a comparison failure for the embedding host.""" + del out + self.recorded_failures.append(doctest.DocTestFailure(test, example, got)) + if not self.continue_on_failure: + self.optionflags |= doctest.FAIL_FAST + + def report_unexpected_exception( + self, + out: _Out, + test: doctest.DocTest, + example: doctest.Example, + exc_info: tuple[ + type[BaseException], + BaseException, + types.TracebackType, + ], + ) -> None: + """Retain Python failures while propagating host-owned exceptions.""" + del out + if self.exception_policy.should_propagate(exc_info[1]): + raise exc_info[1] + self.recorded_failures.append( + doctest.UnexpectedException(test, example, exc_info), + ) + if not self.continue_on_failure: + self.optionflags |= doctest.FAIL_FAST + + +class PromptRuntime: + """Execute prompt-form examples on CPython's untouched example loop.""" + + def __init__(self, settings: RuntimeSettings) -> None: + self.runner = _CollectingRunner(settings) + + def run(self, test: doctest.DocTest) -> RuntimeOutcome: + """Run a stock doctest without clearing its shared globals.""" + self.runner.recorded_failures.clear() + results = self.runner.run( + test, + out=lambda _: None, + clear_globs=False, + ) + failures = tuple(self.runner.recorded_failures) + skipped = getattr(results, "skipped", None) + if skipped is None: + skipped = _prompt_skipped(test, failures, self.runner) + return RuntimeOutcome(results, failures, skipped) + + +def _prompt_skipped( + test: doctest.DocTest, + failures: tuple[Failure, ...], + runner: _CollectingRunner, +) -> int: + """Reconstruct reached skips on CPython versions that do not report them.""" + stop_index: int | None = None + if failures: + first_failure = test.examples.index(failures[0].example) + if not runner.continue_on_failure: + stop_index = first_failure + else: + for index in range(first_failure, len(test.examples)): + example = test.examples[index] + flags = _effective_flags( + runner.original_optionflags, + example.options, + ) + if flags & doctest.SKIP: + continue + if flags & doctest.FAIL_FAST: + stop_index = index + break + reached = test.examples if stop_index is None else test.examples[: stop_index + 1] + return sum( + bool( + _effective_flags(runner.original_optionflags, example.options) + & doctest.SKIP + ) + for example in reached + ) + + +class ExecRuntime: + """Execute prompt-free Sphinx blocks with doctest comparison semantics.""" + + def __init__(self, settings: RuntimeSettings) -> None: + self.settings = settings + + def run(self, test: doctest.DocTest) -> RuntimeOutcome: + """Run examples in ``exec`` mode against the test's live mapping.""" + failures: list[Failure] = [] + attempted = 0 + skipped = 0 + for index, example in enumerate(test.examples): + attempted += 1 + flags = _effective_flags(self.settings.optionflags, example.options) + if flags & doctest.SKIP: + skipped += 1 + continue + got_stream = io.StringIO() + exc_info: ( + tuple[ + type[BaseException], + BaseException, + types.TracebackType | None, + ] + | None + ) = None + try: + code = compile( + example.source, + f"", + "exec", + _compile_flags(test.globs), + dont_inherit=True, + ) + with contextlib.redirect_stdout(got_stream): + # Doctests execute author-provided Python by definition. + exec(code, test.globs) # noqa: S102 + except BaseException as error: + if self.settings.exception_policy.should_propagate(error): + raise + traceback_head = error.__traceback__ + exc_info = ( + type(error), + error, + None if traceback_head is None else traceback_head.tb_next, + ) + + got = _captured_output(got_stream) + failure = self._compare(test, example, got, exc_info, flags) + if failure is not None: + failures.append(failure) + if not self.settings.continue_on_failure or flags & doctest.FAIL_FAST: + break + return RuntimeOutcome( + _results(len(failures), attempted, skipped), + tuple(failures), + skipped, + ) + + def _compare( + self, + test: doctest.DocTest, + example: doctest.Example, + got: str, + exc_info: tuple[ + type[BaseException], + BaseException, + types.TracebackType | None, + ] + | None, + flags: int, + ) -> Failure | None: + """Return a stock failure object when one example does not match.""" + if exc_info is not None: + if example.exc_msg is None: + return doctest.UnexpectedException( + test, + example, + t.cast(t.Any, exc_info), + ) + formatted = traceback.format_exception_only(exc_info[0], exc_info[1]) + if issubclass(exc_info[0], SyntaxError): + prefixes = ( + f"{exc_info[0].__qualname__}:", + f"{exc_info[0].__module__}.{exc_info[0].__qualname__}:", + ) + message_index = next( + index + for index, line in enumerate(formatted) + if line.startswith(prefixes) + ) + formatted = formatted[message_index:] + exc_msg = "".join(formatted) + if self.settings.checker.check_output(example.exc_msg, exc_msg, flags): + return None + if flags & doctest.IGNORE_EXCEPTION_DETAIL: + expected = _strip_exception_details(example.exc_msg) + actual = _strip_exception_details(exc_msg) + if self.settings.checker.check_output(expected, actual, flags): + return None + traceback_text = "".join(traceback.format_exception(*exc_info)) + return doctest.DocTestFailure(test, example, got + traceback_text) + if example.exc_msg is not None: + return doctest.DocTestFailure(test, example, got) + if self.settings.checker.check_output(example.want, got, flags): + return None + return doctest.DocTestFailure(test, example, got) + + +def _strip_exception_details(message: str) -> str: + r"""Retain only the exception name for detail-insensitive comparison. + + >>> _strip_exception_details("package.Error: detail\n") + 'Error' + """ + line = message.split("\n", 1)[0] + name = line.split(":", 1)[0] + return name.rsplit(".", 1)[-1] + + +class PromptExecutionProfile: + """Factory for the vanilla prompt runtime.""" + + def open( + self, + settings: RuntimeSettings, + ) -> contextlib.AbstractContextManager[ExecutionRuntime]: + """Return an attempt-local prompt runtime.""" + return contextlib.nullcontext(PromptRuntime(settings)) + + +class ExecExecutionProfile: + """Factory for prompt-free ``testcode`` and phase blocks.""" + + def open( + self, + settings: RuntimeSettings, + ) -> contextlib.AbstractContextManager[ExecutionRuntime]: + """Return an attempt-local exec runtime.""" + return contextlib.nullcontext(ExecRuntime(settings)) + + +def _expected_enabled( + block: ProjectedBlock, + globs: dict[str, t.Any], +) -> bool: + """Evaluate the paired output's gates against the live group mapping.""" + expected = block.expected + if expected is None: + return False + if expected.skipif is not None and bool(eval(expected.skipif, globs)): + return False + return expected.pyversion is None or _version_allowed(expected.pyversion) + + +def _exception_message(want: str) -> str | None: + r"""Extract doctest's expected exception tail from paired output. + + >>> _exception_message( + ... 'Traceback (most recent call last):\n...\nValueError: bad\n' + ... ) + 'ValueError: bad\n' + >>> _exception_message('ordinary output\n') is None + True + """ + match = doctest.DocTestParser._EXCEPTION_RE.match(want) # type: ignore[attr-defined] + return match.group("msg") if match is not None else None + + +def materialize( + block: ProjectedBlock, + globs: dict[str, t.Any], + *, + expected_enabled: bool = True, +) -> doctest.DocTest: + r"""Build fresh stock ``Example`` and ``DocTest`` objects for an attempt. + + >>> import doctest + >>> type(doctest.Example("pass\n", "")) is doctest.Example + True + """ + examples: list[doctest.Example] = [] + for index, recipe in enumerate(block.examples): + options = dict(block.options) + want = recipe.want + exc_msg = recipe.exc_msg + if block.expected is not None: + if expected_enabled: + options.update(block.expected.options) + options[doctest.DONT_ACCEPT_BLANKLINE] = True + want = block.expected.text + exc_msg = _exception_message(want) + else: + want = "" + exc_msg = None + options.update(recipe.options) + examples.append( + doctest.Example( + source=recipe.source, + want=want, + exc_msg=exc_msg, + lineno=recipe.lineno, + indent=recipe.indent, + options=options, + ), + ) + if block.expected is not None and index == 0: + break + test = doctest.DocTest( + examples, + globs, + block.name, + block.filename, + block.lineno, + block.docstring, + ) + test.globs = globs + return test + + +def reset_globs( + plan: GroupPlan, + globs: dict[str, t.Any], + *, + extraglobs: t.Mapping[str, t.Any] | None = None, +) -> None: + """Clear and reseed one canonical group mapping in place. + + >>> mapping = {"old": True} + >>> reset_globs(GroupPlan("default", (), {"seed": 1}), mapping) + >>> mapping + {'seed': 1, '__name__': '__main__'} + """ + globs.clear() + globs.update(plan.seed) + if extraglobs is not None: + globs.update(extraglobs) + globs.setdefault("__name__", "__main__") + + +def _version_allowed(specifier: str) -> bool: + """Return whether the current interpreter satisfies a PEP 440 specifier.""" + version = Version(".".join(str(part) for part in sys.version_info[:3])) + return version in SpecifierSet(specifier) + + +def _block_gate( + block: ProjectedBlock, + globs: dict[str, t.Any], +) -> SkipReason | None: + """Evaluate one block gate at the execution boundary.""" + if block.skipif is not None and bool(eval(block.skipif, globs)): + return SkipReason("skipif", block.skipif) + if block.pyversion is not None and not _version_allowed(block.pyversion): + return SkipReason("pyversion", block.pyversion) + return None + + +def _run_block( + block: ProjectedBlock, + globs: dict[str, t.Any], + runtime: ExecutionRuntime, + checker: doctest.OutputChecker, + settings: RunSettings, +) -> BlockResult: + """Gate, materialize, and run one projected block.""" + try: + gate = _block_gate(block, globs) + if gate is not None: + return Skipped(block, Counts(0, 0, 0), gate) + expected_enabled = _expected_enabled(block, globs) + test = materialize(block, globs, expected_enabled=expected_enabled) + outcome = runtime.run(test) + # The exception policy decides which host and process outcomes propagate. + except BaseException as error: # noqa: BLE001 + return Errored(block, error) + counts = Counts( + outcome.results.failed, + outcome.results.attempted, + outcome.skipped, + ) + if outcome.failures: + return Failed(block, counts, outcome.failures, checker) + if test.examples and outcome.skipped == len(test.examples): + return Skipped( + block, + counts, + SkipReason("inline-flag", "SKIP"), + ) + return Passed(block, counts) + + +def run_group( + plan: GroupPlan, + globs: dict[str, t.Any], + *, + settings: RunSettings | None = None, + registry: RegistrySnapshot | None = None, + exception_policy: ExceptionPolicy | None = None, +) -> GroupResult: + """Run one group attempt with setup/test/cleanup phase semantics.""" + if registry is None: + from .registry import build_registry + + registry = build_registry() + settings = settings or RunSettings() + exception_policy = exception_policy or DefaultExceptionPolicy() + checker_registration = registry.output_checkers[settings.checker_name] + results: list[BlockResult] = [] + primary: BaseException | None = None + secondary: list[BaseException] = [] + body_failed = False + stop_after_failure = not settings.continue_on_failure or bool( + settings.optionflags & doctest.FAIL_FAST + ) + + with contextlib.ExitStack() as stack: + runtimes: dict[str, ExecutionRuntime] = {} + checkers: dict[str, doctest.OutputChecker] = {} + for profile_name in dict.fromkeys(block.profile_name for block in plan.blocks): + profile = registry.execution_profiles[profile_name].value + checker = checker_registration.value() + runtime_settings = RuntimeSettings( + optionflags=settings.optionflags, + continue_on_failure=settings.continue_on_failure, + checker=checker, + exception_policy=exception_policy, + ) + checkers[profile_name] = checker + runtimes[profile_name] = stack.enter_context(profile.open(runtime_settings)) + + setup_failed = False + for phase in (Phase.SETUP, Phase.TEST): + if phase is Phase.TEST and setup_failed: + break + for block in (item for item in plan.blocks if item.phase is phase): + result = _run_block( + block, + globs, + runtimes[block.profile_name], + checkers[block.profile_name], + settings, + ) + results.append(result) + if isinstance(result, Errored): + primary = result.error + body_failed = True + setup_failed = phase is Phase.SETUP + break + if isinstance(result, Failed): + body_failed = True + setup_failed = phase is Phase.SETUP + if setup_failed or stop_after_failure: + break + if primary is not None or setup_failed: + break + + for block in (item for item in plan.blocks if item.phase is Phase.CLEANUP): + result = _run_block( + block, + globs, + runtimes[block.profile_name], + checkers[block.profile_name], + settings, + ) + results.append(result) + if isinstance(result, Errored): + if exception_policy.is_abort(result.error): + if primary is None or not exception_policy.is_abort(primary): + if primary is not None: + secondary.append(primary) + primary = result.error + else: + secondary.append(result.error) + elif primary is None and not body_failed: + primary = result.error + else: + secondary.append(result.error) + + return GroupResult(plan.group, tuple(results), primary, tuple(secondary)) diff --git a/src/doctest_core/settings.py b/src/doctest_core/settings.py new file mode 100644 index 0000000..7ddb6a0 --- /dev/null +++ b/src/doctest_core/settings.py @@ -0,0 +1,56 @@ +"""Immutable settings resolved before doctest-core pipeline stages.""" + +from __future__ import annotations + +import typing as t + + +class ParseSettings(t.NamedTuple): + """Settings for markup parsing and extraction. + + Attributes + ---------- + suppressed_diagnostics : frozenset of str + Diagnostic codes omitted from the returned parse result. + + >>> ParseSettings().suppressed_diagnostics + frozenset({'docutils.unknown-role'}) + """ + + suppressed_diagnostics: frozenset[str] = frozenset({"docutils.unknown-role"}) + + +class ProjectionSettings(t.NamedTuple): + """Settings for pure block-to-group projection. + + Attributes + ---------- + ungrouped : {"default", "block"} + Put unlabelled blocks in the shared ``default`` group or isolate them. + + >>> ProjectionSettings().ungrouped + 'default' + """ + + ungrouped: t.Literal["default", "block"] = "default" + + +class RunSettings(t.NamedTuple): + """Serializable policy for one doctest run. + + Attributes + ---------- + optionflags : int + Runner-level doctest option bitmask. + continue_on_failure : bool + Retain later failures from the same block after a mismatch. + checker_name : str + Output-checker registration selected for the run. + + >>> (RunSettings().continue_on_failure, RunSettings().checker_name) + (True, 'stdlib') + """ + + optionflags: int = 0 + continue_on_failure: bool = True + checker_name: str = "stdlib" diff --git a/tests/test_doctest_core_projection.py b/tests/test_doctest_core_projection.py new file mode 100644 index 0000000..770645b --- /dev/null +++ b/tests/test_doctest_core_projection.py @@ -0,0 +1,538 @@ +"""Tests for doctree extraction and pure group projection.""" + +from __future__ import annotations + +import doctest +import pathlib +import typing as t + +import pytest +from docutils import nodes +from docutils.utils import new_document + +from doctest_core import ( + BlockKind, + ParseSettings, + Phase, + ProjectionSettings, + Provider, + build_registry, + extract_blocks, + parse_document, + project, +) +from doctest_core.markup import _stamp_myst_source_lines + + +@pytest.fixture(params=["rst", "myst"]) +def grouped_document(request: pytest.FixtureRequest) -> tuple[pathlib.Path, str]: + """Return equivalent reStructuredText and MyST documents.""" + if request.param == "rst": + return ( + pathlib.Path("guide.rst"), + """ +.. testsetup:: alpha, beta + + value = 40 + +.. doctest:: alpha, beta + :options: +ELLIPSIS + :skipif: False + :pyversion: >=3.10 + + >>> value + 2 + 42 + +.. testcode:: alpha + + print(value + 3) + +.. testoutput:: alpha + :options: +NORMALIZE_WHITESPACE + + 43 + +.. testcleanup:: alpha, beta + + del value +""", + ) + return ( + pathlib.Path("guide.md"), + """ +```{testsetup} alpha, beta +value = 40 +``` + +```{doctest} alpha, beta +:options: +ELLIPSIS +:skipif: False +:pyversion: ">=3.10" + +>>> value + 2 +42 +``` + +```{testcode} alpha +print(value + 3) +``` + +```{testoutput} alpha +:options: +NORMALIZE_WHITESPACE + +43 +``` + +```{testcleanup} alpha, beta +del value +``` +""", + ) + + +def test_extracts_sphinx_vocabulary_as_typed_data( + grouped_document: tuple[pathlib.Path, str], +) -> None: + """Both front ends preserve groups, gates, options, and source order.""" + path, source = grouped_document + + result = parse_document( + source, + path, + settings=ParseSettings(), + registry=build_registry(), + ) + + assert [block.kind for block in result.blocks] == [ + "testsetup", + "doctest", + "testcode", + "testcleanup", + ] + assert [block.document_order for block in result.blocks] == [0, 1, 2, 4] + assert [block.block_ordinal for block in result.blocks] == [0, 1, 2, 3] + assert result.blocks[1].groups == ("alpha", "beta") + assert result.blocks[1].options == {doctest.ELLIPSIS: True} + assert result.blocks[1].skipif == "False" + assert result.blocks[1].pyversion == ">=3.10" + assert result.outputs[0].document_order == 3 + assert result.outputs[0].options == {doctest.NORMALIZE_WHITESPACE: True} + assert not [item for item in result.diagnostics if item.level == "error"] + + +def test_projection_groups_without_merging_blocks( + grouped_document: tuple[pathlib.Path, str], +) -> None: + """One plan owns shared state while every source block keeps a recipe.""" + path, source = grouped_document + parsed = parse_document(source, path, registry=build_registry()) + + plans = project( + parsed, + document_name=path.name, + settings=ProjectionSettings(), + registry=build_registry(), + ) + + assert [plan.group for plan in plans] == ["alpha", "beta"] + assert [block.phase for block in plans[0].blocks] == [ + Phase.SETUP, + Phase.TEST, + Phase.TEST, + Phase.CLEANUP, + ] + assert [block.phase for block in plans[1].blocks] == [ + Phase.SETUP, + Phase.TEST, + Phase.CLEANUP, + ] + assert plans[0].blocks[1] is not plans[1].blocks[1] + assert plans[0].blocks[1].name == "guide::alpha[1]" + assert plans[1].blocks[1].name == "guide::beta[1]" + assert plans[0].blocks[2].expected is not None + assert plans[0].blocks[2].expected.text == "43\n" + + +def test_prompt_recipe_is_exactly_stdlib_normalized() -> None: + """Projection preserves every field emitted by ``DocTestParser``.""" + source = """ +>>> value = 1 +>>> value + 1 +2 +>>> int('bad') +Traceback (most recent call last): +ValueError: invalid literal... +""" + parsed = parse_document(source, pathlib.Path("guide.rst")) + + plan = project(parsed, document_name="guide")[0] + recipes = plan.blocks[0].examples + expected = ( + doctest.DocTestParser() + .get_doctest( + parsed.blocks[0].source, + {}, + "guide", + "guide.rst", + 0, + ) + .examples + ) + + assert [tuple(recipe) for recipe in recipes] == [ + ( + example.source, + example.want, + example.exc_msg, + example.lineno, + example.indent, + example.options, + ) + for example in expected + ] + + +def test_ungrouped_policy_is_explicit() -> None: + """The projection setting chooses sharing without changing block identity.""" + parsed = parse_document( + ">>> one = 1\n\nSome prose.\n\n>>> one + 1\n2\n", + pathlib.Path("guide.rst"), + ) + + shared = project( + parsed, + document_name="guide", + settings=ProjectionSettings(ungrouped="default"), + ) + isolated = project( + parsed, + document_name="guide", + settings=ProjectionSettings(ungrouped="block"), + ) + + assert [(plan.group, len(plan.blocks)) for plan in shared] == [("default", 2)] + assert [(plan.group, len(plan.blocks)) for plan in isolated] == [ + ("block-0", 1), + ("block-1", 1), + ] + + +def test_anonymous_group_does_not_collide_with_named_group() -> None: + """A generated block group cannot alias an author-declared group.""" + parsed = parse_document( + """>>> anonymous = True + +.. doctest:: block-0 + + >>> named = True +""", + pathlib.Path("guide.rst"), + ) + + plans = project( + parsed, + document_name="guide", + settings=ProjectionSettings(ungrouped="block"), + ) + + assert len(plans) == 2 + assert len({plan.group for plan in plans}) == 2 + assert [[block.block_ordinal for block in plan.blocks] for plan in plans] == [ + [0], + [1], + ] + + +def test_parse_deduplicates_reporter_and_doctree_diagnostics() -> None: + """One parser problem produces one typed diagnostic with its best line.""" + parsed = parse_document( + ".. unknown-directive::\n", + pathlib.Path("guide.rst"), + settings=ParseSettings(suppressed_diagnostics=frozenset()), + ) + + errors = [ + diagnostic for diagnostic in parsed.diagnostics if diagnostic.level == "error" + ] + + assert len(errors) == 1 + assert errors[0].code == "docutils.unknown-directive" + assert errors[0].line == 1 + + +def test_default_unknown_role_suppression_includes_lookup_companion() -> None: + """Suppressing an unknown role removes both docutils messages.""" + parsed = parse_document( + ":missing-role:`value`\n", + pathlib.Path("guide.rst"), + ) + + assert not [ + diagnostic + for diagnostic in parsed.diagnostics + if "role" in diagnostic.message.lower() + ] + + +def test_registered_block_kind_survives_generic_node_extraction() -> None: + """Extraction preserves stamps while projection resolves their policy.""" + tree = new_document("guide.rst") + node = nodes.literal_block( + ">>> 6 * 7\n42\n", + ">>> 6 * 7\n42\n", + testnodetype="example", + groups=["shared"], + ) + node.source = "guide.rst" + node.line = 1 + tree += node + + class Contributor: + provider = Provider("example", "1") + + def contribute(self, registrar: t.Any) -> None: + """Register the node stamp's projection policy.""" + registrar.add_block_kind( + "example", + BlockKind(Phase.TEST, "prompt", None), + ) + + registry = build_registry([Contributor()]) + parsed = extract_blocks(tree, registry=registry) + plans = project(parsed, document_name="guide.rst", registry=registry) + + assert parsed.blocks[0].kind == "example" + assert plans[0].blocks[0].examples[0].source == "6 * 7\n" + + +def test_unregistered_stamp_does_not_rename_runnable_blocks() -> None: + """Foreign node metadata cannot consume a runnable identity ordinal.""" + tree = new_document("guide.rst") + tree += nodes.literal_block( + "foreign", + "foreign", + testnodetype="foreign", + ) + tree += nodes.doctest_block(">>> 6 * 7\n42\n", ">>> 6 * 7\n42\n") + + parsed = extract_blocks(tree) + + assert [(block.kind, block.block_ordinal) for block in parsed.blocks] == [ + ("doctest", 0), + ] + + +def test_extraction_rejects_malformed_typed_text_stamps() -> None: + """Dynamic node metadata cannot violate the public parsed-record types.""" + tree = new_document("guide.rst") + node = nodes.literal_block( + ">>> 6 * 7\n42\n", + ">>> 6 * 7\n42\n", + testnodetype="doctest", + groups=["shared"], + skipif=123, + ) + tree += node + + with pytest.raises(TypeError, match="skipif node attribute"): + extract_blocks(tree) + + +def test_registered_block_kind_pairs_with_custom_output_stamp() -> None: + """A block kind can name an output stamp without parser changes.""" + tree = new_document("guide.rst") + code = nodes.literal_block( + 'print("answer")', + 'print("answer")', + testnodetype="example", + groups=["shared"], + ) + code.source = "guide.rst" + code.line = 1 + output = nodes.literal_block( + "answer", + "answer", + testnodetype="expected", + groups=["shared"], + ) + output.source = "guide.rst" + output.line = 3 + tree += code + tree += output + + class Contributor: + provider = Provider("example", "1") + + def contribute(self, registrar: t.Any) -> None: + """Register the custom executable and output relationship.""" + registrar.add_block_kind( + "example", + BlockKind(Phase.TEST, "exec", "expected"), + ) + + registry = build_registry([Contributor()]) + parsed = extract_blocks(tree, registry=registry) + plans = project(parsed, document_name="guide.rst", registry=registry) + + assert parsed.outputs[0].kind == "expected" + assert plans[0].blocks[0].expected is not None + assert plans[0].blocks[0].expected.text == "answer\n" + + +@pytest.mark.parametrize("body", ["", "value = 42"]) +def test_prompt_free_doctest_does_not_project_a_group(body: str) -> None: + """A doctest directive without examples cannot become a host item.""" + parsed = parse_document( + f".. doctest::\n\n {body}\n", + pathlib.Path("guide.rst"), + ) + + assert project(parsed, document_name="guide.rst") == () + + +@pytest.mark.parametrize( + ("path", "source"), + [ + ( + pathlib.Path("guide.rst"), + """ +.. testcode:: + :trim-doctest-flags: + + print("answer") + +.. testoutput:: + :no-trim-doctest-flags: + + answer +""", + ), + ( + pathlib.Path("guide.md"), + """ +```{testcode} +:trim-doctest-flags: +print("answer") +``` + +```{testoutput} +:no-trim-doctest-flags: +answer +``` +""", + ), + ], +) +def test_testcode_output_accept_sphinx_trim_options( + path: pathlib.Path, + source: str, +) -> None: + """Standalone parsers accept Sphinx's full trim-option vocabulary.""" + parsed = parse_document(source, path) + + assert [block.kind for block in parsed.blocks] == ["testcode"] + assert [output.kind for output in parsed.outputs] == ["testoutput"] + assert not [item for item in parsed.diagnostics if item.level == "error"] + + +def test_myst_directive_line_is_document_absolute() -> None: + """MyST-local content offsets do not replace the fence's document line.""" + parsed = parse_document( + """Heading +======= +```{doctest} +>>> 1 + 1 +3 +``` +""", + pathlib.Path("guide.md"), + ) + + assert parsed.blocks[0].line == 4 + plan = project(parsed, document_name="guide.md")[0] + assert plan.blocks[0].lineno == 3 + + with_options = parse_document( + """Heading +======= +```{doctest} +:options: +ELLIPSIS + +>>> 1 + 1 +3 +``` +""", + pathlib.Path("guide.md"), + ) + + assert with_options.blocks[0].line == 6 + + +@pytest.mark.parametrize( + ("source", "expected_line"), + [ + ("Text\n\n >>> 1 + 1\n 2\n", 3), + ("Text\n\n```\n>>> 1 + 1\n2\n```\n", 4), + ], +) +def test_myst_bare_prompt_line_distinguishes_indent_and_fence( + source: str, + expected_line: int, +) -> None: + """Standalone MyST stamps the first executable source line.""" + parsed = parse_document(source, pathlib.Path("guide.md")) + + assert parsed.blocks[0].line == expected_line + + +def test_myst_line_stamper_does_not_attribute_included_source_to_root() -> None: + """Root text cannot supply an absolute line for an included node.""" + tree = new_document("guide.md") + node = nodes.literal_block( + ">>> 1 + 1\n2\n", + ">>> 1 + 1\n2\n", + testnodetype="doctest", + ) + node.source = "included.md" + node.line = 1 + tree += node + + _stamp_myst_source_lines(tree, "Text\n\n>>> 1 + 1\n2\n") + parsed = extract_blocks(tree) + + assert parsed.blocks[0].path == pathlib.Path("included.md") + assert parsed.blocks[0].line == 2 + + +def test_output_pairing_is_group_local_and_latest_wins() -> None: + """Other groups do not break pairing and later output replaces earlier.""" + source = """ +.. testcode:: alpha + + print("alpha") + +.. testcode:: beta + + print("beta") + +.. testoutput:: alpha + + stale + +.. testoutput:: alpha + + alpha + +.. testoutput:: beta + + beta +""" + parsed = parse_document(source, pathlib.Path("guide.rst")) + + plans = project(parsed, document_name="guide") + + assert [plan.group for plan in plans] == ["alpha", "beta"] + assert plans[0].blocks[0].expected is not None + assert plans[0].blocks[0].expected.text == "alpha\n" + assert plans[1].blocks[0].expected is not None + assert plans[1].blocks[0].expected.text == "beta\n" diff --git a/tests/test_doctest_core_registry.py b/tests/test_doctest_core_registry.py new file mode 100644 index 0000000..18ff7b0 --- /dev/null +++ b/tests/test_doctest_core_registry.py @@ -0,0 +1,144 @@ +"""Tests for the doctest core registry.""" + +from __future__ import annotations + +import doctest +import typing as t + +import pytest + +from doctest_core import ( + BlockKind, + Phase, + Provider, + RegistryClosedError, + RegistryCollisionError, + RegistryError, + build_registry, +) + + +class RecordingContributor: + """Register one checker and retain the registrar for the freeze test.""" + + provider = Provider(name="tests", version="1") + + def __init__(self, *, replace: bool = False) -> None: + self.registrar: t.Any = None + self.replace = replace + + def contribute(self, registrar: t.Any) -> None: + """Register a checker factory. + + >>> contributor = RecordingContributor() + >>> contributor.provider.name + 'tests' + """ + self.registrar = registrar + registrar.add_output_checker( + "stdlib", + doctest.OutputChecker, + replace=self.replace, + ) + + +def test_registry_snapshot_is_ordered_and_immutable() -> None: + """Built-ins freeze in declaration order behind read-only mappings.""" + snapshot = build_registry() + + assert tuple(snapshot.block_kinds) == ( + "doctest", + "testsetup", + "testcleanup", + "testcode", + ) + assert tuple(snapshot.document_parsers) == ("rst", "myst") + assert tuple(snapshot.execution_profiles) == ("prompt", "exec") + assert tuple(snapshot.output_checkers) == ("stdlib",) + + with pytest.raises(TypeError): + snapshot.output_checkers["other"] = snapshot.output_checkers["stdlib"] # type: ignore[index] + + +def test_registry_rejects_implicit_collision() -> None: + """A contributor cannot silently replace another provider's capability.""" + with pytest.raises(RegistryCollisionError, match=r"stdlib.*builtin.*tests"): + build_registry([RecordingContributor()]) + + +def test_registry_explicit_replacement_preserves_position() -> None: + """An explicit replacement retains the incumbent's precedence.""" + contributor = RecordingContributor(replace=True) + + snapshot = build_registry([contributor]) + + assert tuple(snapshot.output_checkers) == ("stdlib",) + assert snapshot.output_checkers["stdlib"].provider == contributor.provider + + +def test_registry_retained_registrar_closes_after_freeze() -> None: + """A retained registrar cannot mutate a frozen snapshot.""" + contributor = RecordingContributor(replace=True) + build_registry([contributor]) + + with pytest.raises(RegistryClosedError): + contributor.registrar.add_output_checker("late", doctest.OutputChecker) + + +def test_registry_rejects_missing_execution_profile_reference() -> None: + """A frozen block kind cannot defer a missing-profile KeyError to runtime.""" + + class Contributor: + provider = Provider("broken", "1") + + def contribute(self, registrar: t.Any) -> None: + """Register a block kind with no executable profile.""" + registrar.add_block_kind( + "example", + BlockKind(Phase.TEST, "missing", None), + ) + + with pytest.raises(RegistryError, match=r"example.*broken.*missing"): + build_registry([Contributor()]) + + +def test_registry_rejects_runnable_output_kind_collision() -> None: + """One node stamp cannot be both executable and expected output.""" + + class Contributor: + provider = Provider("ambiguous", "1") + + def contribute(self, registrar: t.Any) -> None: + """Register contradictory executable and output roles.""" + registrar.add_block_kind( + "example", + BlockKind(Phase.TEST, "exec", "expected"), + ) + registrar.add_block_kind( + "expected", + BlockKind(Phase.TEST, "exec", None), + ) + + with pytest.raises( + RegistryCollisionError, + match=r"example.*ambiguous.*expected.*ambiguous", + ): + build_registry([Contributor()]) + + +@pytest.mark.parametrize("output_kind", ["", "Expected", "bad name"]) +def test_registry_rejects_invalid_output_kind_reference(output_kind: str) -> None: + """Cross-references obey the same name grammar as registrations.""" + + class Contributor: + provider = Provider("broken", "1") + + def contribute(self, registrar: t.Any) -> None: + """Register a block kind with a malformed output reference.""" + registrar.add_block_kind( + "example", + BlockKind(Phase.TEST, "exec", output_kind), + ) + + with pytest.raises(RegistryError, match=r"invalid registry name"): + build_registry([Contributor()]) diff --git a/tests/test_doctest_core_runner.py b/tests/test_doctest_core_runner.py new file mode 100644 index 0000000..9ffe5a4 --- /dev/null +++ b/tests/test_doctest_core_runner.py @@ -0,0 +1,717 @@ +"""Tests for fresh materialization and group execution.""" + +from __future__ import annotations + +import doctest +import pathlib +import sys +import traceback + +from doctest_core import ( + Counts, + ExampleRecipe, + Failed, + GroupPlan, + Passed, + Phase, + ProjectedBlock, + RunSettings, + build_registry, + materialize, + parse_document, + project, + reset_globs, + run_group, +) + + +def test_materializes_fresh_stock_objects_against_one_mapping() -> None: + """Plans retain recipes while attempts receive ordinary fresh objects.""" + parsed = parse_document(">>> 1 + 1\n2\n", pathlib.Path("guide.rst")) + block = project(parsed, document_name="guide")[0].blocks[0] + globs: dict[str, object] = {} + + first = materialize(block, globs) + second = materialize(block, globs) + + assert type(first) is doctest.DocTest + assert type(first.examples[0]) is doctest.Example + assert first is not second + assert first.examples[0] is not second.examples[0] + assert first.globs is globs + assert second.globs is globs + assert first.examples[0].source == "1 + 1\n" + assert first.examples[0].want == "2\n" + + +def test_group_runner_shares_state_and_always_runs_cleanup() -> None: + """Separate block tests share one mapping owned by their group attempt.""" + source = """ +.. testsetup:: example + + value = 40 + +.. doctest:: example + + >>> value + 2 + 42 + +.. doctest:: example + + >>> value + 3 + 99 + +.. testcleanup:: example + + cleaned = True +""" + plan = project( + parse_document(source, pathlib.Path("guide.rst")), + document_name="guide", + )[0] + globs: dict[str, object] = {"residue": "old"} + identity = id(globs) + reset_globs(plan, globs) + + result = run_group( + plan, + globs, + settings=RunSettings(continue_on_failure=False), + registry=build_registry(), + ) + + assert id(globs) == identity + assert "residue" not in globs + assert globs["cleaned"] is True + assert isinstance(result.blocks[0], Passed) + assert isinstance(result.blocks[1], Passed) + assert isinstance(result.blocks[2], Failed) + failure = result.blocks[2].failures[0] + assert failure.test.name == "guide::example[2]" + assert failure.test.globs is globs + assert result.primary is None + + +def test_exec_profile_pairs_testcode_output() -> None: + """Prompt-free Sphinx code uses its paired output and shared globals.""" + source = """ +.. testsetup:: example + + value = 41 + +.. testcode:: example + + print(value + 1) + +.. testoutput:: example + + 42 +""" + plan = project( + parse_document(source, pathlib.Path("guide.rst")), + document_name="guide", + )[0] + globs: dict[str, object] = {} + reset_globs(plan, globs) + + result = run_group(plan, globs) + + assert [type(block) for block in result.blocks] == [Passed, Passed] + test_result = result.blocks[-1] + assert isinstance(test_result, Passed) + assert test_result.counts == Counts(failed=0, attempted=1, skipped=0) + + +def test_reset_globs_reseeds_each_attempt() -> None: + """A second attempt cannot observe mutations left by its predecessor.""" + parsed = parse_document(">>> token\n'fresh'\n", pathlib.Path("guide.rst")) + plan = project(parsed, document_name="guide", seed={"token": "fresh"})[0] + globs: dict[str, object] = {} + + reset_globs(plan, globs) + globs["token"] = "mutated" + reset_globs(plan, globs, extraglobs={"fixture": 42}) + + assert globs == {"token": "fresh", "fixture": 42, "__name__": "__main__"} + + +def test_gate_error_is_recorded_and_cleanup_still_runs() -> None: + """An author-controlled gate cannot escape the group cleanup boundary.""" + source = """ +.. doctest:: shared + :skipif: 1 / 0 + + >>> value = 42 + +.. testcleanup:: shared + + cleaned = True +""" + plan = project( + parse_document(source, pathlib.Path("guide.rst")), + document_name="guide", + )[0] + globs: dict[str, object] = {} + reset_globs(plan, globs) + + result = run_group(plan, globs) + + assert isinstance(result.primary, ZeroDivisionError) + assert globs["cleaned"] is True + assert [type(block).__name__ for block in result.blocks] == [ + "Errored", + "Passed", + ] + + +def test_exec_profile_does_not_inherit_core_future_flags() -> None: + """Exec bodies inherit document state, not this module's future imports.""" + source = """ +.. testcode:: shared + + def identity(value: int) -> int: + return value + print(identity.__annotations__) + +.. testoutput:: shared + + {'value': , 'return': } +""" + plan = project( + parse_document(source, pathlib.Path("guide.rst")), + document_name="guide", + )[0] + globs: dict[str, object] = {} + reset_globs(plan, globs) + + result = run_group(plan, globs) + + assert [type(block).__name__ for block in result.blocks] == ["Passed"] + + +def test_exec_profile_restores_stdout_and_records_unexpected_exception() -> None: + """The extended lane restores process state and emits stock failures.""" + source = """ +.. testcode:: shared + + print("before") + raise ValueError("boom") +""" + plan = project( + parse_document(source, pathlib.Path("guide.rst")), + document_name="guide", + )[0] + globs: dict[str, object] = {} + reset_globs(plan, globs) + stdout = sys.stdout + + result = run_group(plan, globs) + + assert sys.stdout is stdout + block = result.blocks[0] + assert isinstance(block, Failed) + assert isinstance(block.failures[0], doctest.UnexpectedException) + assert block.counts == Counts(failed=1, attempted=1, skipped=0) + rendered = "".join(traceback.format_exception(*block.failures[0].exc_info)) + assert "src/doctest_core/runner.py" not in rendered + assert "" in rendered + + +def test_default_exception_policy_retains_system_exit() -> None: + """The host-neutral default matches CPython's unexpected-exception rule.""" + plan = project( + parse_document(">>> raise SystemExit(7)\n", pathlib.Path("guide.rst")), + document_name="guide", + )[0] + globs: dict[str, object] = {} + reset_globs(plan, globs) + + result = run_group(plan, globs) + + block = result.blocks[0] + assert isinstance(block, Failed) + assert isinstance(block.failures[0], doctest.UnexpectedException) + assert result.primary is None + + +def test_prompt_profile_uses_stock_fail_fast_and_skip_accounting() -> None: + """Prompt execution retains CPython's option merge and attempt counts.""" + source = """ +.. doctest:: shared + + >>> 1 + 1 # doctest: +SKIP + 99 + >>> 2 + 2 + 5 + >>> 3 + 3 + 6 +""" + plan = project( + parse_document(source, pathlib.Path("guide.rst")), + document_name="guide", + )[0] + globs: dict[str, object] = {} + reset_globs(plan, globs) + + result = run_group( + plan, + globs, + settings=RunSettings(continue_on_failure=False), + ) + + block = result.blocks[0] + assert isinstance(block, Failed) + stock_has_skip_count = hasattr(doctest.TestResults(0, 0), "skipped") + expected_attempts = 2 if stock_has_skip_count else 1 + assert block.counts == Counts(failed=1, attempted=expected_attempts, skipped=1) + assert len(block.failures) == 1 + + +def test_prompt_skip_count_excludes_unreached_examples() -> None: + """Old CPython fallback counts only skips reached before fail-fast.""" + source = """ +.. doctest:: shared + + >>> 1 + 1 + 3 + >>> 2 + 2 # doctest: +SKIP + 4 +""" + plan = project( + parse_document(source, pathlib.Path("guide.rst")), + document_name="guide", + )[0] + globs: dict[str, object] = {} + reset_globs(plan, globs) + + result = run_group( + plan, + globs, + settings=RunSettings( + optionflags=doctest.FAIL_FAST, + continue_on_failure=True, + ), + ) + + block = result.blocks[0] + assert isinstance(block, Failed) + assert block.counts == Counts(failed=1, attempted=1, skipped=0) + + +def test_report_only_first_retains_total_failure_count() -> None: + """Report suppression does not reduce typed or summary failure totals.""" + source = """ +.. doctest:: shared + + >>> 1 + 1 + 3 + >>> 2 + 2 + 5 +""" + plan = project( + parse_document(source, pathlib.Path("guide.rst")), + document_name="guide", + )[0] + globs: dict[str, object] = {} + reset_globs(plan, globs) + + result = run_group( + plan, + globs, + settings=RunSettings( + optionflags=doctest.REPORT_ONLY_FIRST_FAILURE, + continue_on_failure=True, + ), + ) + + block = result.blocks[0] + assert isinstance(block, Failed) + assert block.counts == Counts(failed=2, attempted=2, skipped=0) + assert len(block.failures) == 1 + + +def test_report_only_hidden_fail_fast_bounds_old_skip_fallback() -> None: + """A quiet stopping failure does not make later skips look examined.""" + source = """ +.. doctest:: shared + + >>> 1 + 1 + 3 + >>> 2 + 2 # doctest: +FAIL_FAST + 5 + >>> 3 + 3 # doctest: +SKIP + 6 +""" + plan = project( + parse_document(source, pathlib.Path("guide.rst")), + document_name="guide", + )[0] + globs: dict[str, object] = {} + reset_globs(plan, globs) + + result = run_group( + plan, + globs, + settings=RunSettings( + optionflags=doctest.REPORT_ONLY_FIRST_FAILURE, + continue_on_failure=True, + ), + ) + + block = result.blocks[0] + assert isinstance(block, Failed) + assert block.counts == Counts(failed=2, attempted=2, skipped=0) + assert len(block.failures) == 1 + + +def test_inline_report_only_preserves_cpython_sequencing() -> None: + """Prompt reporting retains CPython's previous-example option timing.""" + source = """ +.. doctest:: shared + + >>> 1 + 1 # doctest: +REPORT_ONLY_FIRST_FAILURE + 3 + >>> 2 + 2 + 5 + >>> 3 + 3 + 7 +""" + plan = project( + parse_document(source, pathlib.Path("guide.rst")), + document_name="guide", + )[0] + globs: dict[str, object] = {} + reset_globs(plan, globs) + + result = run_group(plan, globs) + + block = result.blocks[0] + assert isinstance(block, Failed) + assert block.counts == Counts(failed=3, attempted=3, skipped=0) + assert [failure.example.source for failure in block.failures] == [ + "1 + 1 # doctest: +REPORT_ONLY_FIRST_FAILURE\n", + "3 + 3\n", + ] + + +def test_exec_profile_honors_explicit_fail_fast_option() -> None: + """The extended lane keeps runner flags distinct from host continuation.""" + source = """ +.. testcode:: shared + + print(1) + +.. testoutput:: shared + + 2 + +.. testcode:: shared + + print(3) + +.. testoutput:: shared + + 4 +""" + plan = project( + parse_document(source, pathlib.Path("guide.rst")), + document_name="guide", + )[0] + globs: dict[str, object] = {} + reset_globs(plan, globs) + + result = run_group( + plan, + globs, + settings=RunSettings( + optionflags=doctest.FAIL_FAST, + continue_on_failure=True, + ), + ) + + assert len(result.blocks) == 1 + assert isinstance(result.blocks[0], Failed) + + +def test_default_run_settings_continue_across_failed_blocks() -> None: + """The host-neutral default follows doctest and Sphinx continuation.""" + source = """ +.. doctest:: shared + + >>> 1 + 1 + 3 + +.. doctest:: shared + + >>> 2 + 2 + 5 +""" + plan = project( + parse_document(source, pathlib.Path("guide.rst")), + document_name="guide", + )[0] + globs: dict[str, object] = {} + reset_globs(plan, globs) + + result = run_group(plan, globs) + + assert [type(block) for block in result.blocks] == [Failed, Failed] + + +def test_host_stop_policy_cannot_be_disabled_inline() -> None: + """An example cannot override the host's debugger-style stop policy.""" + source = """ +.. doctest:: shared + + >>> state = [] + >>> 1 + 1 # doctest: -FAIL_FAST + 3 + >>> state.append("ran") +""" + plan = project( + parse_document(source, pathlib.Path("guide.rst")), + document_name="guide", + )[0] + globs: dict[str, object] = {} + reset_globs(plan, globs) + + result = run_group( + plan, + globs, + settings=RunSettings(continue_on_failure=False), + ) + + assert isinstance(result.blocks[0], Failed) + assert result.blocks[0].counts == Counts(failed=1, attempted=2, skipped=0) + assert globs["state"] == [] + + +def test_cleanup_abort_is_not_demoted_after_test_failure() -> None: + """A process abort from cleanup remains stronger than block failures.""" + source = """ +.. testcode:: shared + + print(1) + +.. testoutput:: shared + + 2 + +.. testcleanup:: shared + + raise KeyboardInterrupt() +""" + plan = project( + parse_document(source, pathlib.Path("guide.rst")), + document_name="guide", + )[0] + globs: dict[str, object] = {} + reset_globs(plan, globs) + + result = run_group(plan, globs) + + assert isinstance(result.blocks[0], Failed) + assert isinstance(result.primary, KeyboardInterrupt) + + +def test_exec_profile_accepts_expected_exception() -> None: + """A Sphinx testoutput traceback supplies the stock expected exception.""" + source = """ +.. testcode:: shared + + raise ValueError("boom") + +.. testoutput:: shared + + Traceback (most recent call last): + ... + ValueError: boom +""" + plan = project( + parse_document(source, pathlib.Path("guide.rst")), + document_name="guide", + )[0] + globs: dict[str, object] = {} + reset_globs(plan, globs) + + result = run_group(plan, globs) + + assert [type(block) for block in result.blocks] == [Passed] + + +def test_exec_profile_ignores_stdout_before_expected_exception() -> None: + """Expected exceptions compare the exception tail, as CPython does.""" + source = """ +.. testcode:: shared + + print("before") + raise ValueError("boom") + +.. testoutput:: shared + + Traceback (most recent call last): + ... + ValueError: boom +""" + plan = project( + parse_document(source, pathlib.Path("guide.rst")), + document_name="guide", + )[0] + globs: dict[str, object] = {} + reset_globs(plan, globs) + + result = run_group(plan, globs) + + assert [type(block) for block in result.blocks] == [Passed] + + +def test_exec_profile_honors_ignore_exception_detail() -> None: + """Expected exceptions retain CPython's detail-insensitive fallback.""" + source = """ +.. testcode:: shared + + raise ValueError("actual detail") + +.. testoutput:: shared + :options: +IGNORE_EXCEPTION_DETAIL + + Traceback (most recent call last): + ... + ValueError: expected detail +""" + plan = project( + parse_document(source, pathlib.Path("guide.rst")), + document_name="guide", + )[0] + globs: dict[str, object] = {} + reset_globs(plan, globs) + + result = run_group(plan, globs) + + assert [type(block) for block in result.blocks] == [Passed] + + +def test_expected_exception_mismatch_hides_runtime_frame() -> None: + """Failure traceback ownership starts at the author's compiled block.""" + source = """ +.. testcode:: shared + + raise ValueError("actual") + +.. testoutput:: shared + + Traceback (most recent call last): + ... + TypeError: expected +""" + plan = project( + parse_document(source, pathlib.Path("guide.rst")), + document_name="guide", + )[0] + globs: dict[str, object] = {} + reset_globs(plan, globs) + + result = run_group(plan, globs) + + block = result.blocks[0] + assert isinstance(block, Failed) + failure = block.failures[0] + assert isinstance(failure, doctest.DocTestFailure) + assert "src/doctest_core/runner.py" not in failure.got + assert "" in failure.got + + +def test_exec_profile_normalizes_missing_stdout_newline() -> None: + """Extended output keeps doctest's unrepresentable-newline convention.""" + source = """ +.. testcode:: shared + + import sys + sys.stdout.write("answer") + +.. testoutput:: shared + + answer +""" + plan = project( + parse_document(source, pathlib.Path("guide.rst")), + document_name="guide", + )[0] + globs: dict[str, object] = {} + reset_globs(plan, globs) + + result = run_group(plan, globs) + + assert [type(block) for block in result.blocks] == [Passed] + + +def test_exec_profile_normalizes_syntax_error_details() -> None: + """Syntax errors compare from the exception line across Python versions.""" + source = """ +.. testcode:: shared + + compile("if:", "bad.py", "exec") + +.. testoutput:: shared + + Traceback (most recent call last): + ... + SyntaxError: invalid syntax +""" + plan = project( + parse_document(source, pathlib.Path("guide.rst")), + document_name="guide", + )[0] + globs: dict[str, object] = {} + reset_globs(plan, globs) + + result = run_group(plan, globs) + + assert [type(block) for block in result.blocks] == [Passed] + + +def test_exec_profile_honors_inline_fail_fast() -> None: + """The current example's effective flags control extended execution.""" + block = ProjectedBlock( + phase=Phase.TEST, + name="guide::shared[0]", + block_ordinal=0, + examples=( + ExampleRecipe( + source='print("first")\n', + want="wrong\n", + exc_msg=None, + lineno=0, + indent=0, + options={doctest.FAIL_FAST: True}, + ), + ExampleRecipe( + source='print("second")\n', + want="wrong\n", + exc_msg=None, + lineno=1, + indent=0, + options={}, + ), + ), + docstring="", + filename="guide.rst", + lineno=0, + options={}, + profile_name="exec", + skipif=None, + pyversion=None, + expected=None, + ) + plan = GroupPlan("shared", (block,), {}) + globs: dict[str, object] = {} + reset_globs(plan, globs) + + result = run_group( + plan, + globs, + settings=RunSettings(continue_on_failure=True), + ) + + assert isinstance(result.blocks[0], Failed) + assert result.blocks[0].counts == Counts(failed=1, attempted=1, skipped=0) diff --git a/tests/test_doctest_core_sphinx.py b/tests/test_doctest_core_sphinx.py new file mode 100644 index 0000000..a07525d --- /dev/null +++ b/tests/test_doctest_core_sphinx.py @@ -0,0 +1,88 @@ +"""Acceptance tests for consuming Sphinx-resolved doctrees.""" + +from __future__ import annotations + +import pathlib +import typing as t + +from docutils import nodes +from docutils.utils import new_document + +from doctest_core import extract_blocks + +if t.TYPE_CHECKING: + from sphinx.testing.util import SphinxTestApp + + from .conftest import MakeAppParams + + +def test_extracts_hidden_blocks_from_sphinx_resolved_doctree( + make_app: t.Callable[..., SphinxTestApp], + make_app_params: MakeAppParams, + tmp_path: pathlib.Path, +) -> None: + """Resolved includes retain Sphinx comment nodes and source ownership.""" + included_path = tmp_path / "examples.rst" + included_path.write_text( + """.. testsetup:: shared + + value = 40 + +.. doctest:: shared + + >>> value + 2 + 42 + +.. testcleanup:: shared + + del value +""", + encoding="utf8", + ) + args, kwargs = make_app_params( + index="""Resolved page +============= + +.. include:: examples.rst +""", + confoverrides={"extensions": ["sphinx.ext.doctest"]}, + ) + app = make_app(*args, **kwargs) + app.build() + doctree = app.env.get_and_resolve_doctree("index", app.builder) + + sphinx_hidden_kinds = [ + str(node["testnodetype"]) + for node in doctree.findall(nodes.comment) + if node.get("testnodetype") in {"testsetup", "testcleanup"} + ] + assert sphinx_hidden_kinds == ["testsetup", "testcleanup"] + + result = extract_blocks(doctree) + + assert [block.kind for block in result.blocks] == [ + "testsetup", + "doctest", + "testcleanup", + ] + assert {block.path for block in result.blocks} == {included_path} + assert [block.hidden for block in result.blocks] == [True, False, True] + assert all(block.line is not None for block in result.blocks) + + +def test_sphinx_docstring_source_has_unknown_file_line() -> None: + """Docstring-relative node lines are not fabricated as file locations.""" + tree = new_document("module.rst") + node = nodes.literal_block( + ">>> 6 * 7\n42\n", + ">>> 6 * 7\n42\n", + testnodetype="doctest", + groups=["shared"], + ) + node.source = "/tmp/:docstring of package.module" + node.line = 7 + tree += node + + result = extract_blocks(tree) + + assert result.blocks[0].line is None From 7b4d0a57dd63672d2a181f0da72bcef6dee41dfd Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sat, 8 Aug 2026 07:28:41 -0500 Subject: [PATCH 26/31] doctest_docutils(refactor): Use typed core why: Keep the released direct API and command shape while exercising the same extraction, projection, and execution path as other hosts. what: - Adapt the finder and direct runner to typed block and group results - Preserve stock failure and summary accounting across Python versions - Add direct facade ordering, location, and reporting regressions --- src/doctest_docutils.py | 489 +++++++++++++++++---------------- tests/test_doctest_docutils.py | 127 +++++++++ 2 files changed, 375 insertions(+), 241 deletions(-) diff --git a/src/doctest_docutils.py b/src/doctest_docutils.py index f8f2cde..50bea8b 100644 --- a/src/doctest_docutils.py +++ b/src/doctest_docutils.py @@ -3,27 +3,30 @@ from __future__ import annotations import doctest -import linecache import logging import os import pathlib -import pprint import re import sys +import types import typing as t import docutils -from docutils import nodes -from docutils.parsers.rst import Directive, directives +from docutils.parsers.rst import directives from packaging.specifiers import InvalidSpecifier, SpecifierSet from packaging.version import Version -from docutils_compat import findall +import doctest_core +from doctest_core.markup import ( + DoctestDirective as _CoreDoctestDirective, + MockTabDirective as _CoreMockTabDirective, + TestcleanupDirective as _CoreTestcleanupDirective, + TestsetupDirective as _CoreTestsetupDirective, + _TestDirective as _CoreTestDirective, +) if t.TYPE_CHECKING: - import types - - from docutils.nodes import Node, TextElement + from docutils.nodes import Node logger = logging.getLogger(__name__) @@ -53,13 +56,10 @@ def is_allowed_version(version: str, spec: str) -> bool: return Version(version) in SpecifierSet(spec) -class TestDirective(Directive): - """Base class for doctest-related directives.""" +class TestDirective(_CoreTestDirective): + """Compatibility base for doctest-related directives.""" - has_content = True - required_arguments = 0 - optional_arguments = 1 - final_argument_whitespace = True + __test__ = False def get_source_info(self) -> tuple[str, int]: """Get source and line number.""" @@ -69,115 +69,21 @@ def set_source_info(self, node: Node) -> None: """Set source and line number to the node.""" node.source, node.line = self.get_source_info() - def run(self) -> list[Node]: - """Run docutils test directive.""" - # use ordinary docutils nodes for test code: they get special attributes - # so that our builder recognizes them, and the other builders are happy. - code = "\n".join(self.content) - test = None - - logger.debug(f"directive run: self.name {self.name}") - if self.name == "doctest": - if "" in code: - # convert s to ordinary blank lines for presentation - test = code - code = blankline_re.sub("", code) - if ( - doctestopt_re.search(code) - and "no-trim-doctest-flags" not in self.options - ): - if not test: - test = code - code = doctestopt_re.sub("", code) - nodetype: type[TextElement] = nodes.literal_block - if self.name in {"testsetup", "testcleanup"} or "hide" in self.options: - nodetype = nodes.comment - if self.arguments: - groups = [x.strip() for x in self.arguments[0].split(",")] - else: - groups = ["default"] - node = nodetype(code, code, testnodetype=self.name, groups=groups) - self.set_source_info(node) - if test is not None: - # only save if it differs from code - node["test"] = test - if self.name == "doctest": - node["language"] = "pycon3" - node["options"] = {} - if self.name in ("doctest") and "options" in self.options: - # parse doctest-like output comparison flags - option_strings = self.options["options"].replace(",", " ").split() - for option in option_strings: - prefix, option_name = option[0], option[1:] - if prefix not in "+-": - self.state.document.reporter.warning( - f"missing '+' or '-' in '{option}' option.", - line=self.lineno, - ) - continue - if option_name not in doctest.OPTIONFLAGS_BY_NAME: - self.state.document.reporter.warning( - f"'{option_name}' is not a valid option.", - line=self.lineno, - ) - continue - flag = doctest.OPTIONFLAGS_BY_NAME[option[1:]] - node["options"][flag] = option[0] == "+" - if self.name == "doctest" and "pyversion" in self.options: - try: - spec = self.options["pyversion"] - python_version = ".".join([str(v) for v in sys.version_info[:3]]) - if not is_allowed_version(spec, python_version): - flag = doctest.OPTIONFLAGS_BY_NAME["SKIP"] - node["options"][flag] = True # Skip the test - except InvalidSpecifier: - self.state.document.reporter.warning( - f"'{spec}' is not a valid pyversion option", - line=self.lineno, - ) - if "skipif" in self.options: - node["skipif"] = self.options["skipif"] - if "trim-doctest-flags" in self.options: - node["trim_flags"] = True - elif "no-trim-doctest-flags" in self.options: - node["trim_flags"] = False - return [node] - - -class TestsetupDirective(TestDirective): - """Test setup directive.""" - - option_spec: t.ClassVar = {"skipif": directives.unchanged_required} - -class TestcleanupDirective(TestDirective): - """Test cleanup directive.""" +class TestsetupDirective(_CoreTestsetupDirective, TestDirective): + """Compatibility name for the core ``testsetup`` directive.""" - option_spec: t.ClassVar = {"skipif": directives.unchanged_required} +class TestcleanupDirective(_CoreTestcleanupDirective, TestDirective): + """Compatibility name for the core ``testcleanup`` directive.""" -class DoctestDirective(TestDirective): - """Doctest directive.""" - option_spec: t.ClassVar = { - "no-trim-doctest-flags": directives.flag, - "options": directives.unchanged, - "pyversion": directives.unchanged_required, - "skipif": directives.unchanged_required, - "trim-doctest-flags": directives.flag, - } +class DoctestDirective(_CoreDoctestDirective, TestDirective): + """Compatibility name for the core ``doctest`` directive.""" -class MockTabDirective(TestDirective): - """Mock tab directive.""" - - def run(self) -> list[Node]: - """Parse a mock-tabs directive.""" - self.assert_has_content() - - content = nodes.container("", is_div=True, classes=["tab-content"]) - self.state.nested_parse(self.content, self.content_offset, content) - return [content] +class MockTabDirective(_CoreMockTabDirective, TestDirective): + """Compatibility name for the core mock tab directive.""" def setup() -> dict[str, t.Any]: @@ -188,32 +94,19 @@ def setup() -> dict[str, t.Any]: # Third party mock directive: sphinx-inline-tabs @ 2022.01.02.beta11 directives.register_directive("tab", MockTabDirective) + doctest_core.ensure_directives_registered() return {"version": docutils.__version__, "parallel_read_safe": True} -# For backward compatibility, a global instance of a DocTestRunner -# class, updated by testmod. -master = None +# For backward compatibility, a global runner updated by ``testdocutils``. +master: doctest.DocTestRunner | None = None parser = doctest.DocTestParser() -_DIRECTIVES_READY = False -_REQUIRED_DIRECTIVES = ("doctest", "testsetup", "testcleanup", "tab") - - -def _directive_registry() -> dict[str, t.Any]: - """Return docutils directive registry with typing info.""" - return t.cast(dict[str, t.Any], directives.__dict__["_directives"]) def _ensure_directives_registered() -> None: - """Register doctest-related directives once per interpreter.""" - global _DIRECTIVES_READY - registry = _directive_registry() - missing = any(name not in registry for name in _REQUIRED_DIRECTIVES) - if _DIRECTIVES_READY and not missing: - return - setup() - _DIRECTIVES_READY = True + """Register missing core directives without replacing another owner.""" + doctest_core.ensure_directives_registered() class DocTestFinderNameDoesNotExist(ValueError): @@ -271,12 +164,6 @@ def find( if name is None: raise DocTestFinderNameDoesNotExist(string=string) - # No access to a loader, so assume it's a normal - # filesystem path - source_lines = linecache.getlines(name) or None - if not source_lines: - source_lines = None - # Initialize globals, and merge in extraglobs. globs = {} if globs is None else globs.copy() if extraglobs is not None: @@ -288,12 +175,7 @@ def find( source_path: pathlib.Path | None = ( pathlib.Path(name) if name is not None else None ) - self._find(tests, string, name, source_lines, globs, {}, source_path) - # Sort the tests by alpha order of names, for consistency in - # verbose-mode output. This was a feature of doctest in Pythons - # <= 2.3 that got lost by accident in 2.4. It was repaired in - # 2.4.4 and 2.5. - tests.sort() + self._find(tests, string, name, None, globs, {}, source_path) return tests def _find( @@ -308,98 +190,60 @@ def _find( ) -> None: """Find tests for the given string, and add them to `tests`.""" if self._verbose: - logger.info(f"Finding tests in {name}") + logger.info("finding tests in %s", name) # If we've already processed this string, then ignore it. if id(string) in seen: return seen[id(string)] = 1 - - # Find a test for this string, and add it to the list of tests. - logger.debug( - "_find({})".format( - pprint.pformat( - { - "tests": tests, - "string": string, - "name": name, - "source_lines": source_lines, - "globs": globs, - "seen": seen, - }, - ), - ), - ) - ext = pathlib.Path(name).suffix - logger.debug(f"parse, ext: {ext}") - if ext == ".md": - import myst_parser.parsers.docutils_ - from myst_parser.config.main import MdParserConfig - from myst_parser.mdit_to_docutils.base import ( - DocutilsRenderer, - make_document, - ) - from myst_parser.parsers.mdit import create_md_parser - - DocutilsParser = myst_parser.parsers.docutils_.Parser - config: MdParserConfig = MdParserConfig(commonmark_only=False) - md_parser = create_md_parser(config, DocutilsRenderer) - - doc = make_document( - source_path=str(source_path), - parser_cls=DocutilsParser, - ) - md_parser.options["document"] = doc - md_parser.render(string) - else: - import docutils.utils - from docutils.frontend import OptionParser - from docutils.parsers.rst import Parser - - parser = Parser() - settings = OptionParser(components=(Parser,)).get_default_values() - - doc = docutils.utils.new_document( - source_path=str(source_path), - settings=settings, - ) - parser.parse(string, doc) - - def condition(node: Node) -> bool: - return ( - ( - isinstance(node, (nodes.literal_block, nodes.comment)) - and "testnodetype" in node - ) - or ( - isinstance(node, nodes.literal_block) - and re.match( - doctest.DocTestParser._EXAMPLE_RE, # type:ignore - node.astext(), - ) - is not None - ) - or isinstance(node, nodes.doctest_block) - ) - - for idx, node in enumerate(findall(doc)(condition)): - logger.debug(f"() node: {node.astext()}") - assert isinstance(node, nodes.Element) - test_name = node.get("groups") - if isinstance(test_name, list): - test_name = test_name[0] - if test_name is None or test_name == "default": - test_name = f"{name}[{idx}]" - logger.debug(f"() node: {test_name}") + del source_lines + parse_path = source_path or pathlib.Path(name) + if parse_path.suffix not in {".md", ".rst", ".txt"}: + parse_path = parse_path.with_suffix(".rst") + parsed = doctest_core.parse_document(string, parse_path) + for block in parsed.blocks: + test_name = self._compatibility_name(block, name) test = self._get_test( - string=node.astext(), + string=block.source, name=test_name, - filename=name, + filename=str(block.path), globs=globs, - source_lines=[str(node.line)], + source_lines=[ + str(0 if block.line is None else max(block.line - 1, 0)), + ], ) - if test is not None: - tests.append(test) + self._apply_block_options(test, block) + tests.append(test) + + @staticmethod + def _compatibility_name(block: doctest_core.ParsedBlock, name: str) -> str: + """Reproduce the legacy first-group and anonymous naming scheme.""" + group = block.groups[0] if block.groups else None + if group is None or group == "default": + return f"{name}[{block.block_ordinal}]" + return group + + @staticmethod + def _apply_block_options( + test: doctest.DocTest, + block: doctest_core.ParsedBlock, + ) -> None: + """Merge directive policy into each stock example's inline options.""" + block_options = dict(block.options) + if block.pyversion is not None: + version = ".".join(str(value) for value in sys.version_info[:3]) + try: + if not is_allowed_version(version, block.pyversion): + block_options[doctest.SKIP] = True + except InvalidSpecifier: + logger.warning( + "invalid pyversion option", + extra={"doctest_source_file": test.filename}, + ) + for example in test.examples: + options = block_options.copy() + options.update(example.options) + example.options = options def _get_test( self, @@ -416,9 +260,137 @@ def _get_test( return self._parser.get_doctest(string, globs, name, filename, lineno) +def _direct_plan( + plan: doctest_core.GroupPlan, + *, + filename: str, + parser: doctest.DocTestParser, +) -> doctest_core.GroupPlan: + """Adapt a typed plan to the compatibility facade's names and parser.""" + blocks: list[doctest_core.ProjectedBlock] = [] + for block in plan.blocks: + block_name = ( + plan.group + if plan.group != "default" + else f"{filename}[{block.block_ordinal}]" + ) + examples = block.examples + if block.profile_name == "prompt": + parsed_test = parser.get_doctest( + block.docstring, + {}, + block_name, + block.filename, + 0, + ) + examples = tuple( + doctest_core.ExampleRecipe( + source=example.source, + want=example.want, + exc_msg=example.exc_msg, + lineno=example.lineno, + indent=example.indent, + options=types.MappingProxyType(dict(example.options)), + ) + for example in parsed_test.examples + ) + blocks.append(block._replace(name=block_name, examples=examples)) + return plan._replace(blocks=tuple(blocks)) + + +def _report_failure( + runner: doctest.DocTestRunner, + failure: doctest.DocTestFailure | doctest.UnexpectedException, +) -> None: + """Render a core failure through the stock direct runner hooks.""" + if isinstance(failure, doctest.DocTestFailure): + runner.report_failure( + sys.stdout.write, + failure.test, + failure.example, + failure.got, + ) + return + runner.report_unexpected_exception( + sys.stdout.write, + failure.test, + failure.example, + failure.exc_info, + ) + + +def _record_statistics( + runner: doctest.DocTestRunner, + *, + name: str, + failures: int, + attempted: int, + skipped: int, +) -> None: + """Populate CPython's version-specific summary bookkeeping.""" + runner.failures += failures + runner.tries += attempted + stats = getattr(runner, "_stats", None) + if isinstance(stats, dict): + typed_stats = t.cast(dict[str, tuple[int, int, int]], stats) + old_failures, old_attempted, old_skipped = typed_stats.get( + name, + (0, 0, 0), + ) + typed_stats[name] = ( + old_failures + failures, + old_attempted + attempted, + old_skipped + skipped, + ) + runner.skips += skipped # type: ignore[attr-defined] + return + name_to_counts = t.cast( + dict[str, tuple[int, int]], + runner.__dict__["_name2ft"], + ) + old_failures, old_attempted = name_to_counts.get(name, (0, 0)) + name_to_counts[name] = ( + old_failures + failures, + old_attempted + attempted, + ) + + +def _consume_result( + runner: doctest.DocTestRunner, + result: doctest_core.GroupResult, +) -> None: + """Project one core group result onto the direct doctest runner.""" + for block_result in result.blocks: + failures = 0 + attempted = 0 + skipped = 0 + if isinstance(block_result, doctest_core.Failed): + failures = block_result.counts.failed + attempted = block_result.counts.attempted + skipped = block_result.counts.skipped + for failure in block_result.failures: + _report_failure(runner, failure) + elif block_result.block.phase is doctest_core.Phase.TEST and isinstance( + block_result, (doctest_core.Passed, doctest_core.Skipped) + ): + attempted = block_result.counts.attempted + skipped = block_result.counts.skipped + _record_statistics( + runner, + name=block_result.block.name, + failures=failures, + attempted=attempted, + skipped=skipped, + ) + if result.primary is not None: + raise result.primary + + class TestDocutilsPackageRelativeError(Exception): """Raise when doctest_docutils is called for package not relative to module.""" + __test__ = False + def __init__(self) -> None: super().__init__( "Package may only be specified for module-relative paths.", @@ -451,7 +423,7 @@ def testdocutils( # Keep the absolute file paths. This is needed for Include directies to work. # The absolute path will be applied to source_path when creating the docutils doc. _ensure_directives_registered() - text, _ = doctest._load_testfile( # type: ignore + text, source_filename = doctest._load_testfile( # type: ignore filename, package, module_relative, @@ -469,9 +441,6 @@ def testdocutils( if "__name__" not in globs: globs["__name__"] = "__main__" - # Find, parse, and run all tests in the given module. - finder = DocutilsDocTestFinder() - runner: doctest.DebugRunner | doctest.DocTestRunner if raise_on_error: @@ -479,8 +448,38 @@ def testdocutils( else: runner = doctest.DocTestRunner(verbose=verbose, optionflags=optionflags) - for test in finder.find(text, filename, globs=globs, extraglobs=extraglobs): - runner.run(test) + source_path = pathlib.Path(source_filename) + if source_path.suffix not in {".md", ".rst", ".txt"}: + source_path = source_path.with_suffix(".rst") + registry = doctest_core.build_registry() + parsed = doctest_core.parse_document( + text, + source_path, + registry=registry, + ) + plans = doctest_core.project( + parsed, + document_name=name, + registry=registry, + seed=globs, + ) + settings = doctest_core.RunSettings( + optionflags=optionflags, + continue_on_failure=( + not raise_on_error and not bool(optionflags & doctest.FAIL_FAST) + ), + ) + for plan in plans: + direct_plan = _direct_plan(plan, filename=filename, parser=parser) + live_globs: dict[str, t.Any] = {} + doctest_core.reset_globs(direct_plan, live_globs) + result = doctest_core.run_group( + direct_plan, + live_globs, + settings=settings, + registry=registry, + ) + _consume_result(runner, result) if report: runner.summarize() @@ -490,16 +489,24 @@ def testdocutils( else: master.merge(runner) + if hasattr(runner, "skips"): + constructor = t.cast(t.Any, doctest.TestResults) + return t.cast( + doctest.TestResults, + constructor( + runner.failures, + runner.tries, + skipped=runner.skips, + ), + ) return doctest.TestResults(runner.failures, runner.tries) -def _test() -> int: - """Execute doctest module via CLI. +testdocutils.__test__ = False # type: ignore[attr-defined] - Port changes from standard library at 3.10: - - Sets up logging.basicLogging(level=logging.DEBUG) w/ args.verbose - """ +def _test() -> int: + """Execute doctest module via CLI.""" import argparse p = argparse.ArgumentParser(description="doctest runner") @@ -508,7 +515,7 @@ def _test() -> int: "--verbose", action="store_true", default=False, - help="logger.debug very verbose output for all tests", + help="list tested groups in the final summary", ) p.add_argument( "--log-level", diff --git a/tests/test_doctest_docutils.py b/tests/test_doctest_docutils.py index 42807e1..9b34ee8 100644 --- a/tests/test_doctest_docutils.py +++ b/tests/test_doctest_docutils.py @@ -336,3 +336,130 @@ def test_docutils_package_relative_error_message() -> None: exc = doctest_docutils.TestDocutilsPackageRelativeError() assert str(exc) == "Package may only be specified for module-relative paths." + + +def test_finder_keeps_stock_per_block_results() -> None: + """The compatibility finder returns independent stock doctest objects.""" + source = """.. testsetup:: shared + + value = 40 + +.. doctest:: shared + + >>> value + 2 + 42 + +.. testcleanup:: shared + + del value +""" + + tests = doctest_docutils.DocutilsDocTestFinder().find(source, "guide.rst") + + assert len(tests) == 3 + assert all(type(test) is doctest.DocTest for test in tests) + assert len({id(test.globs) for test in tests}) == 3 + + +def test_finder_preserves_source_order_past_single_digit_ordinals() -> None: + """Compatibility results retain source order instead of sorting names.""" + source = "\n".join(f">>> {ordinal}\n{ordinal}\n" for ordinal in range(12)) + + tests = doctest_docutils.DocutilsDocTestFinder().find(source, "guide.rst") + + assert [test.name for test in tests] == [ + f"guide.rst[{ordinal}]" for ordinal in range(12) + ] + + +def test_finder_preserves_zero_based_line_and_include_source( + tmp_path: pathlib.Path, +) -> None: + """Stock finder results report the block's physical source location.""" + included = tmp_path / "included.rst" + included.write_text(">>> 1 + 1\n3\n", encoding="utf-8") + root = tmp_path / "guide.rst" + root.write_text(".. include:: included.rst\n", encoding="utf-8") + + tests = doctest_docutils.DocutilsDocTestFinder().find( + root.read_text(encoding="utf-8"), + str(root), + ) + + assert len(tests) == 1 + assert tests[0].filename == str(included) + assert tests[0].lineno == 0 + + +def test_testdocutils_owns_group_lifecycle(tmp_path: pathlib.Path) -> None: + """The direct runner shares setup state and cleans up after failure.""" + source_path = tmp_path / "guide.rst" + source_path.write_text( + """.. testsetup:: shared + + value = 40 + +.. doctest:: shared + + >>> value + 2 + 99 + +.. testcleanup:: shared + + cleaned.append(value) +""", + encoding="utf-8", + ) + cleaned: list[int] = [] + + result = doctest_docutils.testdocutils( + str(source_path), + module_relative=False, + globs={"cleaned": cleaned}, + report=False, + ) + + assert result.failed == 1 + assert result.attempted == 1 + assert cleaned == [40] + + +def test_testdocutils_retains_report_suppressed_failure_total( + tmp_path: pathlib.Path, +) -> None: + """The direct summary count is independent of detailed failure reports.""" + source_path = tmp_path / "guide.rst" + source_path.write_text( + ">>> 1 + 1\n3\n>>> 2 + 2\n5\n", + encoding="utf-8", + ) + + result = doctest_docutils.testdocutils( + str(source_path), + module_relative=False, + report=False, + optionflags=doctest.REPORT_ONLY_FIRST_FAILURE, + ) + + assert result.failed == 2 + assert result.attempted == 2 + + +def test_testdocutils_returns_modern_skip_statistics( + tmp_path: pathlib.Path, +) -> None: + """The direct facade retains CPython's version-shaped skip total.""" + source_path = tmp_path / "guide.rst" + source_path.write_text( + ">>> 1 + 1 # doctest: +SKIP\n99\n>>> 2 + 2\n4\n", + encoding="utf-8", + ) + + result = doctest_docutils.testdocutils( + str(source_path), + module_relative=False, + report=False, + ) + + if hasattr(result, "skipped"): + assert t.cast(t.Any, result).skipped == 1 From 6116393926566f7e6bc57697a761df22d3d44f8f Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sat, 8 Aug 2026 08:52:03 -0500 Subject: [PATCH 27/31] pytest_doctest_docutils(refactor): Compose core why: Preserve pytest fixtures, checker semantics, and reporting without blocking the built-in doctest plugin or splitting shared group state. what: - Add one item per group with fresh attempt-local doctest materialization - Quarantine private pytest compatibility and filter duplicate collectors - Exercise fixtures, reruns, xdist, async fixtures, and contributions - Package the typed core, adapters, tests, and corrected plugin entry point --- pyproject.toml | 21 +- src/_pytest_doctest_compat.py | 291 +++++++++ src/pytest_doctest_docutils.py | 701 +++++++++++---------- tests/regressions/test_autouse_fixtures.py | 2 +- tests/test_doctest_core_host_boundaries.py | 130 ++++ tests/test_doctest_core_pytest.py | 424 +++++++++++++ tests/test_doctest_options.py | 12 +- tests/test_plugin_suppression.py | 148 +---- tests/test_pytest_doctest_docutils.py | 17 +- uv.lock | 59 +- 10 files changed, 1337 insertions(+), 468 deletions(-) create mode 100644 src/_pytest_doctest_compat.py create mode 100644 tests/test_doctest_core_host_boundaries.py create mode 100644 tests/test_doctest_core_pytest.py diff --git a/pyproject.toml b/pyproject.toml index 36b38c1..d17c496 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -35,9 +35,10 @@ readme = 'README.md' keywords = [] homepage = "https://gp-libs.git-pull.com" dependencies = [ - "myst_parser", - "docutils>=0.20", - "pytest>=8.3.3" + "docutils>=0.20.1", + "myst-parser>=2.0.0", + "packaging", + "pytest>=8.3.3", ] [project.urls] @@ -56,6 +57,8 @@ dev = [ # Testing "gp-libs", "pytest", + "pytest-asyncio", + "pytest-xdist", "pytest-rerunfailures", "pytest-mock", "pytest-watcher", @@ -79,6 +82,8 @@ docs = [ testing = [ "gp-libs", "pytest", + "pytest-asyncio", + "pytest-xdist", "pytest-rerunfailures", "pytest-mock", "pytest-watcher", @@ -96,7 +101,7 @@ lint = [ ] [project.entry-points.pytest11] -sphinx = "pytest_doctest_docutils" +pytest_doctest_docutils = "pytest_doctest_docutils" [build-system] requires = ["hatchling"] @@ -128,11 +133,17 @@ sphinx-ux-autodoc-layout = false sphinx-ux-badges = false [tool.hatch.build.targets.sdist] -include = ["src/*.py"] +include = [ + "src/*.py", + "src/doctest_core/**", + "tests/**", +] [tool.hatch.build.targets.wheel] packages = [ + "src/_pytest_doctest_compat.py", "src/docutils_compat.py", + "src/doctest_core", "src/doctest_docutils.py", "src/gp_libs.py", "src/linkify_issues.py", diff --git a/src/_pytest_doctest_compat.py b/src/_pytest_doctest_compat.py new file mode 100644 index 0000000..f0728ee --- /dev/null +++ b/src/_pytest_doctest_compat.py @@ -0,0 +1,291 @@ +"""Quarantine pytest's private doctest APIs. + +The public functions in this module are the only boundary at which the +``pytest_doctest_docutils`` adapter should depend on ``_pytest.doctest``. +""" + +from __future__ import annotations + +import collections.abc +import doctest +import traceback +import typing as t + +import pytest +from _pytest import doctest as pytest_doctest +from _pytest._code import ExceptionInfo +from _pytest._code.code import ReprFileLocation, TerminalRepr + +DoctestTextfile = pytest_doctest.DoctestTextfile +MultipleDoctestFailures = pytest_doctest.MultipleDoctestFailures + + +class _OptionflagsContext: + """Present the option interface expected by pytest 7 through 9. + + Attributes + ---------- + config : pytest.Config + Configuration exposed for pytest 7's collector-like call shape. + """ + + def __init__(self, config: pytest.Config) -> None: + """Store the pytest configuration. + + Parameters + ---------- + config : pytest.Config + Configuration whose doctest flags are requested. + + Examples + -------- + The compatibility object retains the exact config object. + + >>> marker = object() + >>> context = _OptionflagsContext(t.cast(pytest.Config, marker)) + >>> context.config is marker + True + """ + self.config = config + + def getini(self, name: str) -> object: + """Delegate ini access for pytest 8 and newer. + + Parameters + ---------- + name : str + Ini option name. + + Returns + ------- + object + Parsed ini value. + + Examples + -------- + >>> class Config: + ... def getini(self, name: str) -> object: + ... return name + >>> context = _OptionflagsContext(t.cast(pytest.Config, Config())) + >>> context.getini("doctest_optionflags") + 'doctest_optionflags' + """ + return self.config.getini(name) + + +def get_checker() -> doctest.OutputChecker: + """Return pytest's extended doctest output checker. + + Returns + ------- + doctest.OutputChecker + Checker supporting pytest's ``ALLOW_*`` and ``NUMBER`` flags. + + Examples + -------- + >>> isinstance(get_checker(), doctest.OutputChecker) + True + """ + return pytest_doctest._get_checker() + + +def get_continue_on_failure(config: pytest.Config) -> bool: + """Return pytest's resolved continue-on-failure policy. + + Parameters + ---------- + config : pytest.Config + Active pytest configuration. + + Returns + ------- + bool + False when pdb requires stopping at the first failure. + + Examples + -------- + The result is always a concrete boolean for a configured session. + + >>> callable(get_continue_on_failure) + True + """ + return pytest_doctest._get_continue_on_failure(config) + + +def get_optionflags(config: pytest.Config) -> int: + """Return doctest option flags across pytest 7 through 9. + + Pytest 7 expects a collector-like object with ``.config``; pytest 8 and + newer expect ``Config`` directly. The compatibility context supports both. + + Parameters + ---------- + config : pytest.Config + Active pytest configuration. + + Returns + ------- + int + Bitwise combination of configured doctest flags. + + Examples + -------- + >>> callable(get_optionflags) + True + """ + context = _OptionflagsContext(config) + return pytest_doctest.get_optionflags(context) # type: ignore[arg-type] + + +def make_multiple_failures( + failures: collections.abc.Sequence[ + doctest.DocTestFailure | doctest.UnexpectedException + ], +) -> BaseException: + """Build pytest's aggregate doctest failure across its narrow annotation. + + Pytest's runner stores both ordinary and unexpected doctest failures, while + the private exception constructor is annotated for ordinary failures only. + + Parameters + ---------- + failures : sequence of doctest failures + Failures retained in source order. + + Returns + ------- + BaseException + Pytest's aggregate failure carrying the original sequence. + + Examples + -------- + >>> callable(make_multiple_failures) + True + """ + constructor = t.cast(t.Any, MultipleDoctestFailures) + return t.cast(BaseException, constructor(failures)) + + +def repr_failure_with_checkers( + item: pytest.DoctestItem, + excinfo: ExceptionInfo[BaseException], + checkers: t.Mapping[int, doctest.OutputChecker], +) -> str | TerminalRepr | None: + """Render doctest failures with their comparison-time checkers. + + Parameters + ---------- + item : pytest.DoctestItem + Item whose configuration selects pytest's report style. + excinfo : pytest.ExceptionInfo + Failure raised by the item. + checkers : mapping of int to doctest.OutputChecker + Checker instances indexed by ``id(failure)``. + + Returns + ------- + str, pytest.TerminalRepr, or None + Pytest's doctest representation, or ``None`` for non-doctest errors. + + Examples + -------- + >>> callable(repr_failure_with_checkers) + True + """ + failures: ( + collections.abc.Sequence[doctest.DocTestFailure | doctest.UnexpectedException] + | None + ) = None + if isinstance( + excinfo.value, + (doctest.DocTestFailure, doctest.UnexpectedException), + ): + failures = [excinfo.value] + elif isinstance(excinfo.value, MultipleDoctestFailures): + failures = t.cast( + collections.abc.Sequence[ + doctest.DocTestFailure | doctest.UnexpectedException + ], + excinfo.value.failures, + ) + if failures is None: + return None + + reprlocation_lines: list[tuple[t.Any, list[str]]] = [] + report_choice = pytest_doctest._get_report_choice( + item.config.getoption("doctestreport"), + ) + for failure in failures: + example = failure.example + test = failure.test + lineno = None if test.lineno is None else test.lineno + example.lineno + 1 + reprlocation = ReprFileLocation( + t.cast(str, test.filename), + lineno, # type: ignore[arg-type] + type(failure).__name__, + ) + if lineno is not None: + assert test.docstring is not None + assert test.lineno is not None + lines = [ + f"{index + test.lineno + 1:03d} {line}" + for index, line in enumerate(test.docstring.splitlines(False)) + ] + lines = lines[max(example.lineno - 9, 0) : example.lineno + 1] + else: + lines = [ + "EXAMPLE LOCATION UNKNOWN, not showing all tests of that example", + ] + indent = ">>>" + for line in example.source.splitlines(): + lines.append(f"??? {indent} {line}") + indent = "..." + + if isinstance(failure, doctest.DocTestFailure): + checker = checkers[id(failure)] + lines.extend( + checker.output_difference( + example, + failure.got, + report_choice, + ).split("\n"), + ) + else: + inner_excinfo = ExceptionInfo.from_exc_info( + failure.exc_info, + ) + lines.append(f"UNEXPECTED EXCEPTION: {inner_excinfo.value!r}") + lines.extend( + line.strip("\n") + for line in traceback.format_exception(*failure.exc_info) + ) + reprlocation_lines.append((reprlocation, lines)) + return pytest_doctest.ReprFailDoctest(reprlocation_lines) + + +def disable_output_capturing_for_darwin(item: pytest.DoctestItem) -> None: + """Apply pytest's Darwin doctest capture workaround to an item. + + Parameters + ---------- + item : pytest.DoctestItem + Item about to execute doctest examples. + + Examples + -------- + >>> callable(disable_output_capturing_for_darwin) + True + """ + item._disable_output_capturing_for_darwin() + + +__all__ = [ + "DoctestTextfile", + "MultipleDoctestFailures", + "disable_output_capturing_for_darwin", + "get_checker", + "get_continue_on_failure", + "get_optionflags", + "make_multiple_failures", + "repr_failure_with_checkers", +] diff --git a/src/pytest_doctest_docutils.py b/src/pytest_doctest_docutils.py index 13c2db0..34cf1ad 100644 --- a/src/pytest_doctest_docutils.py +++ b/src/pytest_doctest_docutils.py @@ -1,121 +1,188 @@ -"""pytest plugin for doctest w/ reStructuredText and markdown. - -.. seealso:: - - - http://www.sphinx-doc.org/en/stable/ext/doctest.html - - https://github.com/sphinx-doc/sphinx/blob/master/sphinx/ext/doctest.py - - This is a derivative of my PR https://github.com/thisch/pytest-sphinx/pull/38 to - pytest-sphinx (BSD 3-clause), 2022-09-03. -""" +"""Pytest host adapter for the typed doctest core.""" from __future__ import annotations import bdb +import collections.abc import doctest -import io -import logging -import sys +import pathlib +import traceback import typing as t +import weakref -import _pytest import pytest -from _pytest import outcomes -from _pytest.outcomes import OutcomeException -from doctest_docutils import DocutilsDocTestFinder, _ensure_directives_registered +import _pytest_doctest_compat as compat +from doctest_core import ( + Contributor, + Errored, + Failed, + GroupPlan, + GroupResult, + Phase, + ProjectionSettings, + Provider, + Registrar, + RegistrySnapshot, + RunSettings, + Skipped, + build_registry, + parse_document, + project, + reset_globs, + run_group, +) +from doctest_core.markup import ensure_directives_registered if t.TYPE_CHECKING: - import pathlib - import types - from collections.abc import Iterable - from doctest import _Out - + from _pytest._code import ExceptionInfo + from _pytest._code.code import TerminalRepr + from _pytest.config import PytestPluginManager from _pytest.config.argparsing import Parser - from _pytest.doctest import DoctestItem -logger = logging.getLogger(__name__) +PYTEST_VERSION = tuple(int(part) for part in pytest.__version__.split(".")[:2]) +_REGISTRY_KEY: pytest.StashKey[RegistrySnapshot] = pytest.StashKey() +_FROZEN_PLUGIN_MANAGERS: weakref.WeakSet[t.Any] = weakref.WeakSet() + + +class DoctestCoreHooks: + """Hooks published by the doctest-core pytest adapter.""" + + @pytest.hookspec + def pytest_doctest_core_contributors( + self, + ) -> Contributor | collections.abc.Iterable[Contributor] | None: + """Return host-neutral contributors before collection.""" + + +class _PytestContributor: + """Use pytest's checker for the core's default checker registration.""" + + provider = Provider(name="pytest", version=pytest.__version__) + + def contribute(self, registrar: Registrar) -> None: + """Replace the stdlib checker with pytest's compatible extension.""" + registrar.add_output_checker( + "stdlib", + compat.get_checker, + replace=True, + ) + + +class _PytestExceptionPolicy: + """Classify pytest outcomes and process aborts for the core runtime.""" + + def should_propagate(self, error: BaseException) -> bool: + """Return whether pytest, rather than doctest, owns ``error``.""" + outcome_types = ( + pytest.skip.Exception, + pytest.xfail.Exception, + pytest.fail.Exception, + pytest.exit.Exception, + ) + return isinstance( + error, + (*outcome_types, KeyboardInterrupt, SystemExit, bdb.BdbQuit), + ) + + def is_abort(self, error: BaseException) -> bool: + """Return whether ``error`` must abort despite prior block outcomes. + + >>> _PytestExceptionPolicy().is_abort(KeyboardInterrupt()) + True + >>> _PytestExceptionPolicy().is_abort(ValueError()) + False + """ + return isinstance( + error, + (pytest.exit.Exception, KeyboardInterrupt, SystemExit, bdb.BdbQuit), + ) + + +def pytest_addhooks(pluginmanager: PytestPluginManager) -> None: + """Publish the contributor hook before pytest loads initial conftests.""" + pluginmanager.add_hookspecs(DoctestCoreHooks) + -# Parse pytest version for version-specific features -PYTEST_VERSION = tuple(int(x) for x in pytest.__version__.split(".")[:2]) +def pytest_plugin_registered( + plugin: object, + manager: PytestPluginManager, +) -> None: + """Reject contributor hooks registered after the host snapshot freezes. -# Lazy definition of runner class -RUNNER_CLASS = None + >>> callable(pytest_plugin_registered) + True + """ + if manager not in _FROZEN_PLUGIN_MANAGERS: + return + contributor_hook = getattr(plugin, "pytest_doctest_core_contributors", None) + if callable(contributor_hook): + plugin_name = manager.get_name(plugin) or type(plugin).__name__ + message = ( + f"pytest plugin {plugin_name!r} registered a doctest-core contributor " + "after the contribution phase closed" + ) + raise pytest.UsageError(message) def pytest_addoption(parser: Parser) -> None: - """Add options to py.test for doctest_docutils.""" + """Add doctest-docutils host options.""" group = parser.getgroup("collect") group.addoption( "--doctest-docutils-modules", action="store_true", default=False, - help="run doctest-doctests in .py modules (pass-through to pytest-doctest)", + help="run doctests in Python modules through pytest's doctest plugin", dest="doctestmodules", ) group.addoption( "--no-doctest-docutils-modules", action="store_false", - help="disable doctest-doctests in .py modules (pass-through to pytest-doctest)", + help="disable doctests in Python modules", dest="doctestmodules", ) + parser.addini( + "doctest_docutils_ungrouped", + "sharing policy for bare documentation blocks: block or default", + default="block", + ) -def pytest_configure(config: pytest.Config) -> None: - """Disable pytest.doctest to prevent running tests twice. +def _flatten_contributors(results: t.Iterable[object]) -> list[Contributor]: + """Flatten pluggy's per-implementation return values in hook order.""" + contributors: list[Contributor] = [] + for result in results: + if result is None: + continue + if hasattr(result, "contribute") and hasattr(result, "provider"): + contributors.append(t.cast(Contributor, result)) + continue + if isinstance(result, collections.abc.Iterable): + contributors.extend(t.cast(collections.abc.Iterable[Contributor], result)) + return contributors - Todo: Find a way to make these plugins cooperate without collecting twice. - """ - # Register HIDE eagerly, before collection parses any docstring. The .py - # path delegates to pytest's own DoctestModule (which never calls our - # _get_flag_lookup), so registering it here is what lets a docstring carry - # ``# doctest: +HIDE`` without raising ``invalid option`` at parse time. - _get_hide_flag() - if config.pluginmanager.has_plugin("doctest"): - config.pluginmanager.set_blocked("doctest") - - -def _unblock_doctest(config: pytest.Config) -> bool: - """Unblock doctest plugin (pytest 8.1+ only). - - Re-enables the built-in doctest plugin after it was blocked by - pytest_configure. Uses the public unblock() API introduced in pytest 8.1.0. - - Parameters - ---------- - config : pytest.Config - The pytest configuration object - - Returns - ------- - bool - True if unblocked successfully, False if API not available - """ - pm = config.pluginmanager - if PYTEST_VERSION >= (8, 1) and hasattr(pm, "unblock"): - return pm.unblock("doctest") - return False - -def pytest_unconfigure() -> None: - """Unconfigure hook for pytest-doctest-docutils.""" - global RUNNER_CLASS - - RUNNER_CLASS = None +@pytest.hookimpl(trylast=True) +def pytest_configure(config: pytest.Config) -> None: + """Freeze host contributions without unregistering pytest's doctest plugin.""" + doctest.register_optionflag("HIDE") + raw_hook = t.cast(t.Any, config.hook).pytest_doctest_core_contributors() + contributors = [_PytestContributor(), *_flatten_contributors(raw_hook)] + config.stash[_REGISTRY_KEY] = build_registry(contributors) + _FROZEN_PLUGIN_MANAGERS.add(config.pluginmanager) + value = config.getini("doctest_docutils_ungrouped") + if value not in {"block", "default"}: + message = "doctest_docutils_ungrouped must be 'block' or 'default'" + raise pytest.UsageError(message) def pytest_ignore_collect(collection_path: pathlib.Path) -> bool | None: - """Skip Sphinx ``_build/`` output during collection. + """Skip generated Sphinx ``_build`` trees. - pytest's default ``norecursedirs`` excludes ``build`` but not ``_build``, - so Sphinx output (which mirrors sources, broken relative includes and all) - would otherwise be collected and abort the session. - - >>> import pathlib - >>> pytest_ignore_collect(pathlib.Path("docs/_build/html/history.md")) + >>> pytest_ignore_collect(pathlib.Path("docs/_build/html/page.md")) True - >>> pytest_ignore_collect(pathlib.Path("docs/history.md")) is None + >>> pytest_ignore_collect(pathlib.Path("docs/page.md")) is None True """ if "_build" in collection_path.parts: @@ -123,276 +190,258 @@ def pytest_ignore_collect(collection_path: pathlib.Path) -> bool | None: return None -def pytest_collect_file( - file_path: pathlib.Path, - parent: pytest.Collector, -) -> DocTestDocutilsFile | _pytest.doctest.DoctestModule | None: - """Test collector for pytest-doctest-docutils.""" - config = parent.config - if file_path.suffix == ".py": - if config.option.doctestmodules and not any( - # if not any( - ( - _pytest.doctest._is_setup_py(file_path), - _pytest.doctest._is_main_py(file_path), - ), - ): - mod: DocTestDocutilsFile | _pytest.doctest.DoctestModule = ( - _pytest.doctest.DoctestModule.from_parent(parent, path=file_path) - ) - return mod - elif _is_doctest(config, file_path, parent): - return DocTestDocutilsFile.from_parent(parent, path=file_path) - return None - - def _is_doctest( config: pytest.Config, path: pathlib.Path, parent: pytest.Collector, ) -> bool: - if path.suffix in {".rst", ".md"} and parent.session.isinitpath(path): + """Return whether this adapter claims a documentation path.""" + registry = config.stash.get(_REGISTRY_KEY, None) + supported_suffixes = ( + { + suffix + for registration in registry.document_parsers.values() + for suffix in registration.value.suffixes + } + if registry is not None + else {".rst", ".md"} + ) + if path.suffix not in supported_suffixes: + return False + if parent.session.isinitpath(path): return True - globs = config.getoption("doctestglob") or ["*.rst", "*.md"] - return any(path.match(path_pattern=glob) for glob in globs) - + patterns = config.getoption("doctestglob", default=None) or ["*.rst", "*.md"] + return any(path.match(pattern) for pattern in patterns) -def _init_runner_class() -> type[doctest.DocTestRunner]: - import doctest - - class PytestDoctestRunner(doctest.DebugRunner): - """Runner to collect failures. - - Note that the out variable in this case is a list instead of a - stdout-like object. - """ - - def __init__( - self, - checker: doctest.OutputChecker | None = None, - verbose: bool | None = None, - optionflags: int = 0, - continue_on_failure: bool = True, - ) -> None: - super().__init__(checker=checker, verbose=verbose, optionflags=optionflags) - self.continue_on_failure = continue_on_failure - - def report_failure( - self, - out: _Out, - test: doctest.DocTest, - example: doctest.Example, - got: str, - ) -> None: - failure = doctest.DocTestFailure(test, example, got) - if self.continue_on_failure: - assert isinstance(out, list) - out.append(failure) - else: - raise failure - - def report_unexpected_exception( - self, - out: _Out, - test: doctest.DocTest, - example: doctest.Example, - exc_info: tuple[ - type[BaseException], - BaseException, - types.TracebackType, - ], - ) -> None: - if isinstance(exc_info[1], OutcomeException): - raise exc_info[1] - if isinstance(exc_info[1], bdb.BdbQuit): - outcomes.exit("Quitting debugger") - failure = doctest.UnexpectedException(test, example, exc_info) - if self.continue_on_failure: - assert isinstance(out, list) - out.append(failure) - else: - raise failure - - return PytestDoctestRunner - - -def _get_allow_unicode_flag() -> int: - """Register and return the ALLOW_UNICODE flag.""" - import doctest - - return doctest.register_optionflag("ALLOW_UNICODE") - - -def _get_allow_bytes_flag() -> int: - """Register and return the ALLOW_BYTES flag.""" - import doctest - - return doctest.register_optionflag("ALLOW_BYTES") - - -def _get_number_flag() -> int: - """Register and return the NUMBER flag.""" - import doctest - - return doctest.register_optionflag("NUMBER") - - -def _get_hide_flag() -> int: - """Register and return the HIDE flag. - - ``HIDE`` is a no-op for execution: the output checker never consults it. - It marks a doctest example that documentation tooling should drop from the - rendered output while still running it as a test. Registering it here means - ``# doctest: +HIDE`` parses instead of raising ``ValueError: invalid - option`` at collection time. - """ - import doctest - - return doctest.register_optionflag("HIDE") +@pytest.hookimpl(hookwrapper=True, tryfirst=True, specname="pytest_collect_file") +def pytest_collect_file_filter( + file_path: pathlib.Path, + parent: pytest.Collector, +) -> t.Generator[None, object, None]: + """Remove pytest's duplicate textfile collector before it parses the file.""" + outcome = yield + if not _is_doctest(parent.config, file_path, parent): + return + hook_result = t.cast(t.Any, outcome).get_result() + filtered = [ + collector + for collector in hook_result + if not isinstance(collector, compat.DoctestTextfile) + ] + t.cast(t.Any, outcome).force_result(filtered) -def _get_flag_lookup() -> dict[str, int]: - import doctest - return { - "DONT_ACCEPT_TRUE_FOR_1": doctest.DONT_ACCEPT_TRUE_FOR_1, - "DONT_ACCEPT_BLANKLINE": doctest.DONT_ACCEPT_BLANKLINE, - "NORMALIZE_WHITESPACE": doctest.NORMALIZE_WHITESPACE, - "ELLIPSIS": doctest.ELLIPSIS, - "IGNORE_EXCEPTION_DETAIL": doctest.IGNORE_EXCEPTION_DETAIL, - "COMPARISON_FLAGS": doctest.COMPARISON_FLAGS, - "ALLOW_UNICODE": _get_allow_unicode_flag(), - "ALLOW_BYTES": _get_allow_bytes_flag(), - "NUMBER": _get_number_flag(), - "HIDE": _get_hide_flag(), - } +def pytest_collect_file( + file_path: pathlib.Path, + parent: pytest.Collector, +) -> DocTestDocutilsFile | pytest.Collector | None: + """Collect documentation here and delegate Python modules to pytest.""" + config = parent.config + if file_path.suffix == ".py": + if config.option.doctestmodules and not config.pluginmanager.has_plugin( + "doctest", + ): + message = ( + f"{file_path}: --doctest-docutils-modules requires pytest's " + "built-in doctest plugin" + ) + raise pytest.UsageError(message) + return None + if _is_doctest(config, file_path, parent): + return DocTestDocutilsFile.from_parent(parent, path=file_path) + return None def get_optionflags(config: pytest.Config) -> int: - """Fetch optionflags from pytest configuration. - - Extracted from pytest.doctest 8.0 (license: MIT). - """ - optionflags = config.getini("doctest_optionflags") - # It takes this rocket surgery to satisfy mypy - optionflags_str = ( - [str(i) for i in optionflags] - if isinstance(optionflags, list) - and all( - isinstance( - item, - str, - ) - for item in optionflags + """Return pytest's resolved doctest option flags.""" + return compat.get_optionflags(config) + + +class DocutilsItem(pytest.DoctestItem): + """One pytest item owning one shared-state doctest group.""" + + @classmethod + def from_parent( # type: ignore[override] + cls, + parent: pytest.Collector, + *, + name: str, + runner: doctest.DocTestRunner, + dtest: doctest.DocTest, + plan: GroupPlan, + registry: RegistrySnapshot, + run_settings: RunSettings, + ) -> DocutilsItem: + """Construct through pytest's cooperative item factory.""" + item = super(pytest.DoctestItem, cls).from_parent( + parent=parent, + name=name, + runner=runner, + dtest=dtest, + plan=plan, + registry=registry, + run_settings=run_settings, ) - else [] - ) - - flag_lookup_table = _get_flag_lookup() - flag_acc = 0 - for flag in optionflags_str: - flag_acc |= flag_lookup_table[flag] - return flag_acc - - -def _get_runner( - checker: doctest.OutputChecker | None = None, - verbose: bool | None = None, - optionflags: int = 0, - continue_on_failure: bool = True, -) -> doctest.DocTestRunner: - # We need this in order to do a lazy import on doctest - global RUNNER_CLASS - if RUNNER_CLASS is None: - RUNNER_CLASS = _init_runner_class() - # Type ignored because the continue_on_failure argument is only defined on - # PytestDoctestRunner, which is lazily defined so can't be used as a type. - return RUNNER_CLASS( # type: ignore - checker=checker, - verbose=verbose, - optionflags=optionflags, - continue_on_failure=continue_on_failure, - ) + return item - -class DocutilsDocTestRunner(doctest.DocTestRunner): - """DocTestRunner for doctest_docutils.""" - - def summarize( # type: ignore + def __init__( self, - out: _Out, - verbose: bool | None = None, - ) -> tuple[int, int]: - """Summarize the test runs.""" - string_io = io.StringIO() - old_stdout = sys.stdout - sys.stdout = string_io - try: - res = super().summarize(verbose) - finally: - sys.stdout = old_stdout - out(string_io.getvalue()) - return res # type:ignore[return-value,unused-ignore] - - def _DocTestRunner__patched_linecache_getlines( + *, + plan: GroupPlan, + registry: RegistrySnapshot, + run_settings: RunSettings, + **kwargs: t.Any, + ) -> None: + super().__init__(**kwargs) + self.plan = plan + self.registry = registry + self.run_settings = run_settings + self.group_result: GroupResult | None = None + self._failure_checkers: dict[int, doctest.OutputChecker] = {} + + def setup(self) -> None: + """Reset attempt state, then let pytest inject fixtures in place.""" + reset_globs(self.plan, self.dtest.globs) + super().setup() + + def runtest(self) -> None: + """Run all block doctests in phase order against the carrier mapping.""" + compat.disable_output_capturing_for_darwin(self) + result = run_group( + self.plan, + self.dtest.globs, + settings=self.run_settings, + registry=self.registry, + exception_policy=_PytestExceptionPolicy(), + ) + self.group_result = result + self._failure_checkers = { + id(failure): block.checker + for block in result.blocks + if isinstance(block, Failed) + for failure in block.failures + } + if result.secondary: + details = "\n\n".join( + "".join( + traceback.format_exception( + type(error), + error, + error.__traceback__, + ), + ) + for error in result.secondary + ) + self.add_report_section("call", "doctest cleanup", details) + if result.primary is not None: + if isinstance(result.primary, bdb.BdbQuit): + pytest.exit("Quitting debugger") + cleanup_outcome = any( + isinstance(block, Errored) + and block.block.phase is Phase.CLEANUP + and block.error is result.primary + for block in result.blocks + ) + if cleanup_outcome and isinstance( + result.primary, + (pytest.skip.Exception, pytest.xfail.Exception), + ): + message = ( + "doctest cleanup raised " + f"{type(result.primary).__name__}: {result.primary}" + ) + raise RuntimeError(message) from result.primary + raise result.primary + + failures = [ + failure + for block in result.blocks + if isinstance(block, Failed) + for failure in block.failures + ] + if failures: + raise compat.make_multiple_failures(failures) + + test_results = [ + block + for block in result.blocks + if block.block.phase is Phase.TEST and not isinstance(block, Errored) + ] + if test_results and all(isinstance(block, Skipped) for block in test_results): + pytest.skip("all examples were skipped") + + def repr_failure( # type: ignore[override] self, - filename: str, - module_globals: t.Any = None, - ) -> t.Any: - # this is overridden from DocTestRunner adding the try-except below - m = self._DocTestRunner__LINECACHE_FILENAME_RE.match(filename) # type: ignore - if m and m.group("name") == self.test.name: - try: - example = self.test.examples[int(m.group("examplenum"))] - # because we compile multiple doctest blocks with the same name - # (viz. the group name) this might, for outer stack frames in a - # traceback, get the wrong test which might not have enough examples - except IndexError: - pass - else: - return example.source.splitlines(True) - return self.save_linecache_getlines(filename, module_globals) # type: ignore + excinfo: ExceptionInfo[BaseException], + ) -> str | TerminalRepr: + """Use comparison-time checkers for contributed output semantics.""" + rendered = compat.repr_failure_with_checkers( + self, + excinfo, + self._failure_checkers, + ) + if rendered is not None: + return rendered + return super().repr_failure(excinfo) class DocTestDocutilsFile(pytest.Module): - """Pytest module for doctest_docutils.""" - - obj = None # Fix pytest-asyncio issue. #46, pytest-asyncio#872 + """Documentation module projecting one item per doctest group.""" - def collect(self) -> Iterable[DoctestItem]: - """Collect tests for pytest module.""" - _ensure_directives_registered() + obj = None + def collect(self) -> collections.abc.Iterable[DocutilsItem]: + """Parse once, project pure plans, and build synthetic carriers.""" + if not self.config.pluginmanager.has_plugin("doctest"): + message = ( + f"{self.path}: documentation collection requires pytest's " + "built-in doctest plugin" + ) + raise pytest.UsageError(message) + ensure_directives_registered() encoding = self.config.getini("doctest_encoding") - text = self.path.read_text(encoding) - - # Uses internal doctest module parsing mechanism. - finder = DocutilsDocTestFinder() + text = self.path.read_text(encoding=encoding) + registry = self.config.stash[_REGISTRY_KEY] + parsed = parse_document(text, self.path, registry=registry) - # While doctests in .rst/.md files don't support fixtures directly, - # we still need to pick up autouse fixtures. - # Backported from pytest commit 9cd14b4ff (2024-02-06). - # https://github.com/pytest-dev/pytest/commit/9cd14b4ff - self.session._fixturemanager.parsefactories(self) - - optionflags = get_optionflags(self.config) - - runner = _get_runner( - verbose=False, - optionflags=optionflags, - checker=_pytest.doctest._get_checker(), - continue_on_failure=_pytest.doctest._get_continue_on_failure(self.config), + ungrouped = t.cast( + t.Literal["block", "default"], + self.config.getini("doctest_docutils_ungrouped"), ) - from _pytest.doctest import DoctestItem - - for test in finder.find( - text, - str(self.path), - ): - if test.examples: # skip empty doctests - yield DoctestItem.from_parent( - self, # type: ignore - name=test.name, - runner=runner, - dtest=test, - ) + plans = project( + parsed, + document_name=self.path.name, + settings=ProjectionSettings(ungrouped=ungrouped), + registry=registry, + ) + optionflags = get_optionflags(self.config) + continue_on_failure = compat.get_continue_on_failure(self.config) + for plan in plans: + globs: dict[str, t.Any] = {} + carrier = doctest.DocTest( + [], + globs, + plan.group, + str(self.path), + 0, + "", + ) + carrier.globs = globs + runner = doctest.DocTestRunner( + checker=compat.get_checker(), + optionflags=optionflags, + ) + yield DocutilsItem.from_parent( + self, + name=plan.group, + runner=runner, + dtest=carrier, + plan=plan, + registry=registry, + run_settings=RunSettings( + optionflags=optionflags, + continue_on_failure=continue_on_failure, + checker_name="stdlib", + ), + ) diff --git a/tests/regressions/test_autouse_fixtures.py b/tests/regressions/test_autouse_fixtures.py index 1058619..1e0b876 100644 --- a/tests/regressions/test_autouse_fixtures.py +++ b/tests/regressions/test_autouse_fixtures.py @@ -99,7 +99,7 @@ def test_autouse_fixtures_with_doctest_files( pytest=textwrap.dedent( """ [pytest] -addopts=-p no:doctest -vv +addopts=-vv """.strip(), ), ) diff --git a/tests/test_doctest_core_host_boundaries.py b/tests/test_doctest_core_host_boundaries.py new file mode 100644 index 0000000..9da3f3d --- /dev/null +++ b/tests/test_doctest_core_host_boundaries.py @@ -0,0 +1,130 @@ +"""Host-boundary acceptance tests for grouped doctest execution.""" + +from __future__ import annotations + +import textwrap + +import _pytest.pytester +import pytest + + +def test_rerun_reseeds_group_globs( + pytester: _pytest.pytester.Pytester, +) -> None: + """A rerun cannot pass by observing mutations from its first attempt.""" + pytester.plugins = ["pytest_doctest_docutils"] + pytester.makefile( + ".rst", + guide=textwrap.dedent( + """ + .. doctest:: shared + + >>> attempt = globals().get("attempt", 0) + 1 + >>> attempt + 2 + """, + ), + ) + + result = pytester.runpytest("guide.rst", "--reruns", "1", "-q") + + result.assert_outcomes(failed=1) + assert result.parseoutcomes()["rerun"] == 1 + + +@pytest.mark.parametrize("distribution", ["load", "worksteal"]) +def test_xdist_runs_stateful_groups_without_affinity( + pytester: _pytest.pytester.Pytester, + distribution: str, +) -> None: + """Each xdist scheduler may move groups without splitting group state.""" + pytester.plugins = ["pytest_doctest_docutils"] + pytester.makeconftest( + textwrap.dedent( + """ + import pytest + + @pytest.fixture(autouse=True) + def inject_worker(doctest_namespace, worker_id): + doctest_namespace["worker_id"] = worker_id + """, + ), + ) + pytester.makefile( + ".rst", + guide=textwrap.dedent( + """ + .. doctest:: first + + >>> state = [worker_id, "first"] + + .. doctest:: first + + >>> state[0].startswith("gw") + True + >>> state[1] + 'first' + + .. doctest:: second + + >>> state = [worker_id, "second"] + + .. doctest:: second + + >>> state[0].startswith("gw") + True + >>> state[1] + 'second' + """, + ), + ) + + result = pytester.runpytest( + "guide.rst", + "-n", + "2", + "--dist", + distribution, + "-q", + ) + + result.assert_outcomes(passed=2) + assert result.ret is pytest.ExitCode.OK + + +def test_pytest_asyncio_fixture_composes_with_document_item( + pytester: _pytest.pytester.Pytester, +) -> None: + """An async autouse fixture can populate the doctest namespace.""" + pytest.importorskip("pytest_asyncio", minversion="1.0") + pytester.plugins = ["pytest_doctest_docutils", "pytest_asyncio.plugin"] + pytester.makeini("[pytest]\nasyncio_mode = auto\n") + pytester.makeconftest( + textwrap.dedent( + """ + import asyncio + + import pytest_asyncio + + @pytest_asyncio.fixture(autouse=True) + async def inject_value(doctest_namespace): + await asyncio.sleep(0) + doctest_namespace["value"] = 42 + """, + ), + ) + pytester.makefile( + ".rst", + guide=textwrap.dedent( + """ + .. doctest:: + + >>> value + 42 + """, + ), + ) + + result = pytester.runpytest("guide.rst", "-q") + + result.assert_outcomes(passed=1) diff --git a/tests/test_doctest_core_pytest.py b/tests/test_doctest_core_pytest.py new file mode 100644 index 0000000..3f63d5d --- /dev/null +++ b/tests/test_doctest_core_pytest.py @@ -0,0 +1,424 @@ +"""End-to-end tests for the typed core's pytest host adapter.""" + +from __future__ import annotations + +import textwrap + +import _pytest.pytester +import pytest + + +def test_pytest_composes_with_builtin_and_collects_one_group( + pytester: _pytest.pytester.Pytester, +) -> None: + """The adapter keeps pytest doctest active and owns one grouped item.""" + pytester.plugins = ["pytest_doctest_docutils"] + pytester.makeconftest( + textwrap.dedent( + """ + import pytest + + def pytest_sessionstart(session): + assert session.config.pluginmanager.has_plugin("doctest") + + @pytest.fixture(autouse=True) + def inject(doctest_namespace): + doctest_namespace["fixture_value"] = 40 + """, + ), + ) + pytester.makefile( + ".rst", + guide=textwrap.dedent( + """ + .. doctest:: shared + + >>> value = fixture_value + + .. doctest:: shared + + >>> value + 2 + 42 + """, + ), + ) + pytester.makepyfile( + test_module=textwrap.dedent( + ''' + def answer(): + """Return the answer. + + >>> answer() + 42 + """ + return 42 + ''', + ), + ) + + result = pytester.runpytest( + "guide.rst", + "test_module.py", + "--doctest-docutils-modules", + "-q", + ) + + result.assert_outcomes(passed=2) + + +def test_pytest_outcome_escapes_and_cleanup_runs( + pytester: _pytest.pytester.Pytester, +) -> None: + """A host skip remains a skip and cannot bypass group cleanup.""" + pytester.plugins = ["pytest_doctest_docutils"] + marker = pytester.path / "cleaned" + pytester.makeconftest( + textwrap.dedent( + f""" + import pathlib + import pytest + + @pytest.fixture(autouse=True) + def inject(doctest_namespace): + doctest_namespace.update( + pytest=pytest, + marker=pathlib.Path({str(marker)!r}), + ) + """, + ), + ) + pytester.makefile( + ".rst", + guide=textwrap.dedent( + """ + .. doctest:: shared + + >>> pytest.skip("not available") + + .. testcleanup:: shared + + marker.write_text("yes") + """, + ), + ) + + result = pytester.runpytest("guide.rst", "-q", "-rs") + + result.assert_outcomes(skipped=1) + result.stdout.fnmatch_lines(["*SKIPPED*not available*"]) + assert marker.read_text(encoding="utf-8") == "yes" + + +def test_cleanup_outcome_is_a_failure( + pytester: _pytest.pytester.Pytester, +) -> None: + """A cleanup skip cannot relabel a completed test as skipped.""" + pytester.plugins = ["pytest_doctest_docutils"] + pytester.makeconftest( + textwrap.dedent( + """ + import pytest + + @pytest.fixture(autouse=True) + def inject(doctest_namespace): + doctest_namespace["pytest"] = pytest + """, + ), + ) + pytester.makefile( + ".rst", + guide=textwrap.dedent( + """ + .. doctest:: shared + + >>> 6 * 7 + 42 + + .. testcleanup:: shared + + pytest.skip("cleanup refused") + """, + ), + ) + + result = pytester.runpytest("guide.rst", "-q") + + result.assert_outcomes(failed=1) + result.stdout.fnmatch_lines(["*cleanup*cleanup refused*"]) + + +def test_cleanup_outcome_is_reported_beside_primary_failure( + pytester: _pytest.pytester.Pytester, +) -> None: + """A secondary cleanup outcome remains visible beside the primary failure.""" + pytester.plugins = ["pytest_doctest_docutils"] + pytester.makeconftest( + textwrap.dedent( + """ + import pytest + + @pytest.fixture(autouse=True) + def inject(doctest_namespace): + doctest_namespace["pytest"] = pytest + """, + ), + ) + pytester.makefile( + ".rst", + guide=textwrap.dedent( + """ + .. doctest:: shared + + >>> 1 + 1 + 3 + + .. testcleanup:: shared + + pytest.skip("cleanup refused after failure") + """, + ), + ) + + result = pytester.runpytest("guide.rst", "-q") + + result.assert_outcomes(failed=1) + result.stdout.fnmatch_lines( + ["*doctest cleanup*", "*cleanup refused after failure*"], + ) + + +def test_pytest_direct_path_has_no_builtin_duplicate( + pytester: _pytest.pytester.Pytester, +) -> None: + """The collector wrapper removes pytest's textfile collector before parse.""" + pytester.plugins = ["pytest_doctest_docutils"] + pytester.makefile( + ".rst", + guide=">>> 6 * 7\n42\n", + ) + + result = pytester.runpytest("guide.rst", "--collect-only", "-q") + + result.assert_outcomes(errors=0) + result.stdout.fnmatch_lines(["*1 test collected*"]) + + +def test_anonymous_item_does_not_collide_with_named_group( + pytester: _pytest.pytester.Pytester, +) -> None: + """A bare block and a same-named author group get separate items and state.""" + pytester.plugins = ["pytest_doctest_docutils"] + pytester.makefile( + ".rst", + guide=textwrap.dedent( + """ + >>> marker = "anonymous" + + .. doctest:: block-0 + + >>> "marker" in globals() + False + """, + ), + ) + + result = pytester.runpytest("guide.rst", "-q") + + result.assert_outcomes(passed=2) + + +@pytest.mark.parametrize("body", ["", "value = 42"]) +def test_pytest_does_not_collect_prompt_free_doctest( + pytester: _pytest.pytester.Pytester, + body: str, +) -> None: + """An empty stock DocTest does not become a passing carrier item.""" + pytester.plugins = ["pytest_doctest_docutils"] + pytester.makefile(".rst", guide=f".. doctest::\n\n {body}\n") + + result = pytester.runpytest("guide.rst", "--collect-only", "-q") + + result.assert_outcomes(errors=0) + result.stdout.fnmatch_lines(["*no tests collected*"]) + + +def test_custom_doctest_glob_remains_owned_by_pytest( + pytester: _pytest.pytester.Pytester, +) -> None: + """Unsupported parser suffixes remain the built-in plugin's concern.""" + pytester.plugins = ["pytest_doctest_docutils"] + path = pytester.path / "guide.foo" + path.write_text(">>> 6 * 7\n42\n", encoding="utf-8") + + result = pytester.runpytest(str(path), "--doctest-glob=*.foo", "-q") + + result.assert_outcomes(passed=1) + + +def test_pytest_preserves_per_block_failure_locations( + pytester: _pytest.pytester.Pytester, +) -> None: + """One group item retains each failed block's source line.""" + pytester.plugins = ["pytest_doctest_docutils"] + pytester.makefile( + ".rst", + guide=""".. doctest:: shared + + >>> 1 + 1 + 3 + +.. doctest:: shared + + >>> 2 + 2 + 5 +""", + ) + + result = pytester.runpytest( + "guide.rst", + "--doctest-continue-on-failure", + "-q", + ) + + result.assert_outcomes(failed=1) + output = result.stdout.str() + assert "guide.rst:3" in output + assert "guide.rst:8" in output + + +def test_contributed_checker_compares_and_explains( + pytester: _pytest.pytester.Pytester, +) -> None: + """The checker that rejects output also renders the resulting failure.""" + pytester.plugins = ["pytest_doctest_docutils"] + pytester.makeconftest( + textwrap.dedent( + """ + import doctest + + from doctest_core import Provider + + class Checker(doctest.OutputChecker): + def __init__(self): + self.compared = False + + def check_output(self, want, got, optionflags): + self.compared = True + return False + + def output_difference(self, example, got, optionflags): + assert self.compared + return "CUSTOM CHECKER DIFFERENCE" + + class Contributor: + provider = Provider("pytest", "probe") + + def contribute(self, registrar): + registrar.add_output_checker( + "stdlib", Checker, replace=True + ) + + def pytest_doctest_core_contributors(): + return Contributor() + """, + ), + ) + pytester.makefile(".rst", guide=">>> 6 * 7\n42\n") + + result = pytester.runpytest("guide.rst", "-q") + + result.assert_outcomes(failed=1) + result.stdout.fnmatch_lines(["*CUSTOM CHECKER DIFFERENCE*"]) + + +def test_late_nested_contributor_is_rejected( + pytester: _pytest.pytester.Pytester, +) -> None: + """A nested conftest cannot silently miss the frozen registry.""" + pytester.plugins = ["pytest_doctest_docutils"] + pytester.makeini("[pytest]\ntestpaths = nested\n") + nested = pytester.path / "nested" + nested.mkdir() + (nested / "conftest.py").write_text( + textwrap.dedent( + """ + def pytest_doctest_core_contributors(): + return None + """, + ), + encoding="utf-8", + ) + (nested / "guide.rst").write_text(">>> 6 * 7\n42\n", encoding="utf-8") + + result = pytester.runpytest(".", "-q") + + assert result.ret is pytest.ExitCode.INTERRUPTED + result.assert_outcomes(errors=1) + result.stdout.fnmatch_lines( + ["*nested/conftest.py*doctest-core contributor*phase closed*"], + ) + + +def test_document_collection_requires_builtin_doctest( + pytester: _pytest.pytester.Pytester, +) -> None: + """Disabling the composed host fails only an affected documentation path.""" + pytester.plugins = ["pytest_doctest_docutils"] + pytester.makefile(".rst", guide=">>> 6 * 7\n42\n") + + result = pytester.runpytest("guide.rst", "-p", "no:doctest", "-q") + + assert result.ret is pytest.ExitCode.INTERRUPTED + result.assert_outcomes(errors=1) + result.stdout.fnmatch_lines( + ["*guide.rst*requires pytest's built-in doctest plugin*"], + ) + + +def test_discovered_document_requires_builtin_doctest( + pytester: _pytest.pytester.Pytester, +) -> None: + """Directory discovery reaches the same actionable disabled-host error.""" + pytester.plugins = ["pytest_doctest_docutils"] + docs = pytester.path / "docs" + docs.mkdir() + (docs / "guide.rst").write_text(">>> 6 * 7\n42\n", encoding="utf-8") + + result = pytester.runpytest("docs", "-p", "no:doctest", "-q") + + assert result.ret is pytest.ExitCode.INTERRUPTED + result.assert_outcomes(errors=1) + result.stdout.fnmatch_lines( + ["*guide.rst*requires pytest's built-in doctest plugin*"], + ) + + +def test_cleanup_exit_outranks_doctest_failure( + pytester: _pytest.pytester.Pytester, +) -> None: + """A cleanup session exit cannot be reduced to a secondary report.""" + pytester.plugins = ["pytest_doctest_docutils"] + pytester.makefile( + ".rst", + guide=textwrap.dedent( + """ + .. testcode:: shared + + print(1) + + .. testoutput:: shared + + 2 + + .. testcleanup:: shared + + import pytest + pytest.exit("cleanup requested") + """, + ), + ) + + result = pytester.runpytest("guide.rst", "-q") + + assert result.ret is pytest.ExitCode.INTERRUPTED + result.stdout.fnmatch_lines(["*Exit: cleanup requested*"]) diff --git a/tests/test_doctest_options.py b/tests/test_doctest_options.py index aa9cd68..d2537d6 100644 --- a/tests/test_doctest_options.py +++ b/tests/test_doctest_options.py @@ -204,7 +204,7 @@ def test_doctest_options( pytester.plugins = ["pytest_doctest_docutils"] # Build pytest.ini content - ini_lines = ["[pytest]", "addopts=-p no:doctest -vv"] + ini_lines = ["[pytest]", "addopts=-vv"] if ini_options: ini_lines.append(ini_options) ini_content = "\n".join(ini_lines) @@ -300,7 +300,7 @@ def test_continue_on_failure( When enabled, all doctest failures should be reported, not just the first. """ pytester.plugins = ["pytest_doctest_docutils"] - pytester.makefile(".ini", pytest="[pytest]\naddopts=-p no:doctest -vv") + pytester.makefile(".ini", pytest="[pytest]\naddopts=-vv") # Create the test file filename = f"test_doc{file_ext}" @@ -380,7 +380,7 @@ def test_custom_flags( """ pytester.plugins = ["pytest_doctest_docutils"] - ini_lines = ["[pytest]", "addopts=-p no:doctest -vv"] + ini_lines = ["[pytest]", "addopts=-vv"] if ini_options: ini_lines.append(ini_options) ini_content = "\n".join(ini_lines) @@ -484,7 +484,7 @@ def test_edge_cases( Tests empty files and files without doctests. """ pytester.plugins = ["pytest_doctest_docutils"] - pytester.makefile(".ini", pytest="[pytest]\naddopts=-p no:doctest -vv") + pytester.makefile(".ini", pytest="[pytest]\naddopts=-vv") # Create the test file filename = f"test_doc{file_ext}" @@ -494,8 +494,8 @@ def test_edge_cases( result = pytester.runpytest(str(file_path), "-v") if expected_outcome == "no_tests": - # Should collect 0 tests (file may be collected but no items) + result.assert_outcomes(errors=0) stdout = result.stdout.str() - assert "0 items" in stdout or "no tests ran" in stdout or expected_tests == 0 + assert "0 items" in stdout or "no tests ran" in stdout elif expected_outcome == "passed": result.assert_outcomes(passed=expected_tests) diff --git a/tests/test_plugin_suppression.py b/tests/test_plugin_suppression.py index 09b14cb..7042fbd 100644 --- a/tests/test_plugin_suppression.py +++ b/tests/test_plugin_suppression.py @@ -1,13 +1,14 @@ -"""Test pytest plugin suppression and precedence. +"""Test pytest doctest collector composition and precedence. -Tests for pytest plugin blocking behavior in pytest_doctest_docutils. -Ensures plugin suppression works correctly across pytest 7.x/8.x/9.x. +Documentation paths have one owner while pytest's doctest plugin remains +available for Python modules and fixture injection. Ref: pytest's test_pluginmanager.py patterns for plugin blocking tests. """ from __future__ import annotations +import importlib.metadata import re import textwrap import typing as t @@ -19,6 +20,17 @@ PYTEST_VERSION = tuple(int(x) for x in pytest.__version__.split(".")[:2]) +def test_pytest_entry_point_uses_adapter_name() -> None: + """The installed plugin can be disabled by its module-shaped name.""" + entries = [ + entry + for entry in importlib.metadata.entry_points(group="pytest11") + if entry.value == "pytest_doctest_docutils" + ] + + assert [entry.name for entry in entries] == ["pytest_doctest_docutils"] + + def requires_pytest_version( min_version: tuple[int, int], reason: str, @@ -55,18 +67,11 @@ class PluginSuppressionCase(t.NamedTuple): PLUGIN_SUPPRESSION_CASES = [ PluginSuppressionCase( - test_id="auto-blocks-builtin-doctest", + test_id="composes-with-builtin-doctest", cli_args=["--collect-only", "-q"], ini_content="", expected_tests_collected=1, - description="pytest_doctest_docutils auto-blocks built-in doctest", - ), - PluginSuppressionCase( - test_id="ini-addopts-no-doctest", - cli_args=["--collect-only", "-q"], - ini_content="addopts = -p no:doctest", - expected_tests_collected=1, - description="addopts=-p no:doctest in pytest.ini works", + description="the adapter filters only the duplicate collector", ), ] @@ -76,7 +81,7 @@ class PluginSuppressionCase(t.NamedTuple): PLUGIN_SUPPRESSION_CASES, ids=[c.test_id for c in PLUGIN_SUPPRESSION_CASES], ) -def test_plugin_suppression( +def test_collector_composition( pytester: _pytest.pytester.Pytester, test_id: str, cli_args: list[str], @@ -84,11 +89,7 @@ def test_plugin_suppression( expected_tests_collected: int, description: str, ) -> None: - """Test plugin suppression behavior. - - Verifies that pytest_doctest_docutils correctly blocks the built-in - doctest plugin to prevent duplicate test collection. - """ + """Verify documentation paths collect exactly once.""" pytester.plugins = ["pytest_doctest_docutils"] # Create pytest.ini if content provided @@ -252,18 +253,12 @@ def hello(): result.assert_outcomes(passed=expected_passed) -def test_pytest_configure_blocks_doctest( +def test_pytest_configure_keeps_doctest( pytester: _pytest.pytester.Pytester, ) -> None: - """Test that pytest_configure automatically blocks the doctest plugin. - - This tests the core behavior in pytest_doctest_docutils.pytest_configure: - if config.pluginmanager.has_plugin("doctest"): - config.pluginmanager.set_blocked("doctest") - """ + """The adapter keeps the built-in doctest plugin registered.""" pytester.plugins = ["pytest_doctest_docutils"] - # Create conftest that checks plugin state after configuration pytester.makeconftest( textwrap.dedent( """ @@ -271,31 +266,27 @@ def test_pytest_configure_blocks_doctest( @pytest.hookimpl(trylast=True) def pytest_configure(config): - # After all pytest_configure hooks run, doctest should be blocked pm = config.pluginmanager - # is_blocked exists in pytest 7+ - if hasattr(pm, 'is_blocked'): - # Store result for test to check - config._doctest_was_blocked = pm.is_blocked('doctest') - else: - config._doctest_was_blocked = None + config._doctest_was_blocked = pm.is_blocked('doctest') + config._doctest_is_loaded = pm.has_plugin('doctest') @pytest.fixture - def doctest_blocked_status(request): - return getattr(request.config, '_doctest_was_blocked', None) + def doctest_plugin_status(request): + return ( + request.config._doctest_was_blocked, + request.config._doctest_is_loaded, + ) """, ), ) - # Create test that verifies the blocking happened pytester.makepyfile( test_verify=textwrap.dedent( """ - def test_doctest_was_blocked(doctest_blocked_status): - if doctest_blocked_status is not None: - assert doctest_blocked_status is True, ( - "doctest plugin should be blocked by pytest_doctest_docutils" - ) + def test_doctest_is_composed(doctest_plugin_status): + blocked, loaded = doctest_plugin_status + assert blocked is False + assert loaded is True """, ), ) @@ -382,7 +373,7 @@ def test_collector_routing( pytester.plugins = ["pytest_doctest_docutils"] pytester.makefile( ".ini", - pytest="[pytest]\naddopts=-p no:doctest -vv", + pytest="[pytest]\naddopts=-vv", ) # Create the test file @@ -404,79 +395,6 @@ def test_collector_routing( ) -# pytest 8.1+ version-specific tests - - -@requires_pytest_version((8, 1), "pluginmanager.unblock() API") -def test_unblock_api_available( - pytester: _pytest.pytester.Pytester, -) -> None: - """Test pluginmanager.unblock() API available in pytest 8.1+. - - Verifies that the unblock() method exists and can be used to - re-enable a previously blocked plugin. - - Ref: pytest 8.1.0 changelog - pluginmanager.unblock() public API - """ - pytester.plugins = ["pytest_doctest_docutils"] - - # Create conftest that tests unblock API - pytester.makeconftest( - textwrap.dedent( - """ - import pytest - - @pytest.hookimpl(trylast=True) - def pytest_configure(config): - pm = config.pluginmanager - - # Verify unblock method exists - assert hasattr(pm, 'unblock'), "unblock() API not found" - - # doctest should be blocked by pytest_doctest_docutils - assert pm.is_blocked('doctest'), "doctest should be blocked" - - # Test unblock API - result = pm.unblock('doctest') - - # Store results for test verification - config._unblock_api_exists = True - config._unblock_result = result - config._doctest_unblocked = not pm.is_blocked('doctest') - - @pytest.fixture - def unblock_test_results(request): - return { - 'api_exists': getattr(request.config, '_unblock_api_exists', False), - 'unblock_result': getattr(request.config, '_unblock_result', None), - 'doctest_unblocked': getattr( - request.config, '_doctest_unblocked', False - ), - } - """, - ), - ) - - # Create test that verifies unblock worked - pytester.makepyfile( - test_verify=textwrap.dedent( - """ - def test_unblock_api_works(unblock_test_results): - assert unblock_test_results['api_exists'], "unblock() API should exist" - assert unblock_test_results['unblock_result'] is True, ( - "unblock() should return True when successful" - ) - assert unblock_test_results['doctest_unblocked'], ( - "doctest should be unblocked after calling unblock()" - ) - """, - ), - ) - - result = pytester.runpytest("test_verify.py", "-v") - result.assert_outcomes(passed=1) - - # pytest 8.4+ version-specific tests diff --git a/tests/test_pytest_doctest_docutils.py b/tests/test_pytest_doctest_docutils.py index 70fbd5a..577ab2d 100644 --- a/tests/test_pytest_doctest_docutils.py +++ b/tests/test_pytest_doctest_docutils.py @@ -216,7 +216,7 @@ def test_pluginDocutilsDocTestFinder( pytest=textwrap.dedent( """ [pytest] -addopts=-p no:doctest -vv +addopts=-vv """.strip(), ), @@ -250,7 +250,7 @@ def test_conftest_py( pytest=textwrap.dedent( """ [pytest] -addopts=-p no:doctest -vv +addopts=-vv """.strip(), ), @@ -321,7 +321,7 @@ def test_conftest_md( pytest=textwrap.dedent( """ [pytest] -addopts=-p no:doctest -vv +addopts=-vv """.strip(), ), @@ -443,7 +443,7 @@ def test_ignore_build_artifacts( pytest=textwrap.dedent( """ [pytest] -addopts=-p no:doctest -vv +addopts=-vv """.strip(), ), @@ -492,15 +492,6 @@ def test_hide_optionflag_py_docstring( ``ValueError: ... invalid option: '+HIDE'``. Here it must simply run. """ pytester.plugins = ["pytest_doctest_docutils"] - pytester.makefile( - ".ini", - pytest=textwrap.dedent( - """ -[pytest] -addopts=-p no:doctest - """.strip(), - ), - ) example = pytester.path / "example.py" example.write_text( textwrap.dedent( diff --git a/uv.lock b/uv.lock index f5b27ae..47ef45a 100644 --- a/uv.lock +++ b/uv.lock @@ -115,6 +115,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/77/f5/21d2de20e8b8b0408f0681956ca2c69f1320a3848ac50e6e7f39c6159675/babel-2.18.0-py3-none-any.whl", hash = "sha256:e2b422b277c2b9a9630c1d7903c2a00d0830c409c59ac8cae9081c92f1aeba35", size = 10196845, upload-time = "2026-02-01T12:30:53.445Z" }, ] +[[package]] +name = "backports-asyncio-runner" +version = "1.2.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/8e/ff/70dca7d7cb1cbc0edb2c6cc0c38b65cba36cccc491eca64cabd5fe7f8670/backports_asyncio_runner-1.2.0.tar.gz", hash = "sha256:a5aa7b2b7d8f8bfcaa2b57313f70792df84e32a2a746f585213373f900b42162", size = 69893, upload-time = "2025-07-02T02:27:15.685Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a0/59/76ab57e3fe74484f48a53f8e337171b4a2349e506eabe136d7e01d059086/backports_asyncio_runner-1.2.0-py3-none-any.whl", hash = "sha256:0da0a936a8aeb554eccb426dc55af3ba63bcdc69fa1a600b5bb305413a4477b5", size = 12313, upload-time = "2025-07-02T02:27:14.263Z" }, +] + [[package]] name = "beautifulsoup4" version = "4.15.0" @@ -382,6 +391,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/8a/0e/97c33bf5009bdbac74fd2beace167cab3f978feb69cc36f1ef79360d6c4e/exceptiongroup-1.3.1-py3-none-any.whl", hash = "sha256:a7a39a3bd276781e98394987d3a5701d0c4edffb633bb7a5144577f82c773598", size = 16740, upload-time = "2025-11-21T23:01:53.443Z" }, ] +[[package]] +name = "execnet" +version = "2.1.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/bf/89/780e11f9588d9e7128a3f87788354c7946a9cbb1401ad38a48c4db9a4f07/execnet-2.1.2.tar.gz", hash = "sha256:63d83bfdd9a23e35b9c6a3261412324f964c2ec8dcd8d3c6916ee9373e0befcd", size = 166622, upload-time = "2025-11-12T09:56:37.75Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ab/84/02fc1827e8cdded4aa65baef11296a9bbe595c474f0d6d758af082d849fd/execnet-2.1.2-py3-none-any.whl", hash = "sha256:67fba928dd5a544b783f6056f449e5e3931a5c378b128bc18501f7ea79e296ec", size = 40708, upload-time = "2025-11-12T09:56:36.333Z" }, +] + [[package]] name = "gp-furo-theme" version = "0.1.0a37" @@ -407,6 +425,7 @@ dependencies = [ { name = "docutils" }, { name = "myst-parser", version = "4.0.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, { name = "myst-parser", version = "5.1.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11'" }, + { name = "packaging" }, { name = "pytest" }, ] @@ -423,10 +442,12 @@ dev = [ { name = "gp-sphinx" }, { name = "mypy" }, { name = "pytest" }, + { name = "pytest-asyncio" }, { name = "pytest-cov" }, { name = "pytest-mock" }, { name = "pytest-rerunfailures" }, { name = "pytest-watcher" }, + { name = "pytest-xdist" }, { name = "ruff" }, { name = "sphinx-autobuild", version = "2024.10.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, { name = "sphinx-autobuild", version = "2025.8.25", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11'" }, @@ -450,15 +471,18 @@ lint = [ testing = [ { name = "gp-libs" }, { name = "pytest" }, + { name = "pytest-asyncio" }, { name = "pytest-mock" }, { name = "pytest-rerunfailures" }, { name = "pytest-watcher" }, + { name = "pytest-xdist" }, ] [package.metadata] requires-dist = [ - { name = "docutils", specifier = ">=0.20" }, - { name = "myst-parser" }, + { name = "docutils", specifier = ">=0.20.1" }, + { name = "myst-parser", specifier = ">=2.0.0" }, + { name = "packaging" }, { name = "pytest", specifier = ">=8.3.3" }, ] @@ -475,10 +499,12 @@ dev = [ { name = "gp-sphinx", specifier = "==0.1.0a37" }, { name = "mypy" }, { name = "pytest" }, + { name = "pytest-asyncio" }, { name = "pytest-cov" }, { name = "pytest-mock" }, { name = "pytest-rerunfailures" }, { name = "pytest-watcher" }, + { name = "pytest-xdist" }, { name = "ruff", specifier = ">=0.16.1" }, { name = "sphinx-autobuild" }, { name = "sphinx-autodoc-api-style", specifier = "==0.1.0a37" }, @@ -500,9 +526,11 @@ lint = [ testing = [ { name = "gp-libs" }, { name = "pytest" }, + { name = "pytest-asyncio" }, { name = "pytest-mock" }, { name = "pytest-rerunfailures" }, { name = "pytest-watcher" }, + { name = "pytest-xdist" }, ] [[package]] @@ -983,6 +1011,20 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/24/25/1de2678b631f5a49215c6c96fff41ba892b0a34df68d6d80292b1b48aa7f/pytest-9.1.1-py3-none-any.whl", hash = "sha256:37a86b45efb9a47a61a36449063e8e18d0cab3161329fc099eb21783169c4f0c", size = 386536, upload-time = "2026-06-19T10:58:31.347Z" }, ] +[[package]] +name = "pytest-asyncio" +version = "1.4.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "backports-asyncio-runner", marker = "python_full_version < '3.11'" }, + { name = "pytest" }, + { name = "typing-extensions", marker = "python_full_version < '3.13'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/43/7c/d36d04db312ecf4298932ef77e6e4a9e8ad017906e24e34f0b0c361a2473/pytest_asyncio-1.4.0.tar.gz", hash = "sha256:c6c0d2259945122819f171a32ecea2c349ead889ee28176caaf492143424be42", size = 58514, upload-time = "2026-05-26T09:56:04.083Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/03/e2/08a497ef684b88559c9cc5f4ad53a37e7b99e727094a86d6ea32536d5d3c/pytest_asyncio-1.4.0-py3-none-any.whl", hash = "sha256:933ca923a23075a87fb7070c0ec272a6848489824d887c85c812670932835aa1", size = 16930, upload-time = "2026-05-26T09:56:02.576Z" }, +] + [[package]] name = "pytest-cov" version = "7.1.0" @@ -1035,6 +1077,19 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/fc/3f/172d73600ad2771774cda108efb813fc724fc345e5240a81a1085f1ade5d/pytest_watcher-0.6.3-py3-none-any.whl", hash = "sha256:83e7748c933087e8276edb6078663e6afa9926434b4fd8b85cf6b32b1d5bec89", size = 12431, upload-time = "2026-01-10T23:28:17.64Z" }, ] +[[package]] +name = "pytest-xdist" +version = "3.8.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "execnet" }, + { name = "pytest" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/78/b4/439b179d1ff526791eb921115fca8e44e596a13efeda518b9d845a619450/pytest_xdist-3.8.0.tar.gz", hash = "sha256:7e578125ec9bc6050861aa93f2d59f1d8d085595d6551c2c90b6f4fad8d3a9f1", size = 88069, upload-time = "2025-07-01T13:30:59.346Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ca/31/d4e37e9e550c2b92a9cbc2e4d0b7420a27224968580b5a447f420847c975/pytest_xdist-3.8.0-py3-none-any.whl", hash = "sha256:202ca578cfeb7370784a8c33d6d05bc6e13b4f25b5053c30a152269fd10f0b88", size = 46396, upload-time = "2025-07-01T13:30:56.632Z" }, +] + [[package]] name = "pyyaml" version = "6.0.3" From 774a1d26e671a636edd6dbd7babe5f54a857c38f Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sat, 8 Aug 2026 08:54:00 -0500 Subject: [PATCH 28/31] ci(test[matrix]): Pin supported versions why: Broad major-version constraints hid invalid dependency combinations and did not prove the declared Python and pytest floor. what: - Pin docutils, pytest, asyncio, and rerun plugin matrix versions - Select the matrix interpreter for every uv command - Check installed dependency consistency before static and runtime gates --- .github/workflows/tests.yml | 27 ++++++++++++++++++--------- 1 file changed, 18 insertions(+), 9 deletions(-) diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 7cf416c..34c6661 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -7,11 +7,20 @@ jobs: if: github.event_name == 'push' || github.event.pull_request.head.repo.full_name != github.repository runs-on: ubuntu-latest + env: + UV_PYTHON: ${{ matrix.python-version }} strategy: matrix: python-version: ['3.10', '3.11', '3.12', '3.13', '3.14'] - docutils-version: ['0.20', '0.22.4'] - pytest-version: ['8', '9'] + docutils-version: ['0.20.1', '0.22.4'] + pytest-version: ['8.4.2', '9.1.1'] + include: + - pytest-version: '8.4.2' + pytest-asyncio-version: '1.4.0' + pytest-rerunfailures-version: '16.4' + - pytest-version: '9.1.1' + pytest-asyncio-version: '1.4.0' + pytest-rerunfailures-version: '16.4' exclude: # docutils 0.22 removed the bundled docutils.utils.roman. Sphinx 8.1 # imports it, so its latex builder fails to load. Sphinx 8.2 dropped @@ -19,11 +28,6 @@ jobs: # supporting 3.10, leaving no working pair on that interpreter. - python-version: '3.10' docutils-version: '0.22.4' - # Overrides the repo's .python-version so every uv command in this job - # uses the matrix interpreter. `uv python install` only downloads one; - # without this the environment is built against .python-version instead. - env: - UV_PYTHON: ${{ matrix.python-version }} steps: - uses: actions/checkout@v7 @@ -41,11 +45,16 @@ jobs: # Every step below runs --no-sync. A plain `uv run` re-syncs the # environment to uv.lock first, which would undo these pins and run # every matrix leg against the locked versions. - - name: Install matrix pytest and docutils + - name: Install matrix versions run: >- uv pip install - "pytest~=${{ matrix.pytest-version }}.0" "docutils==${{ matrix.docutils-version }}" + "pytest==${{ matrix.pytest-version }}" + "pytest-asyncio==${{ matrix.pytest-asyncio-version }}" + "pytest-rerunfailures==${{ matrix.pytest-rerunfailures-version }}" + + - name: Check dependency consistency + run: uv pip check - name: Print python, pytest and docutils versions run: | From ec203005c56e3c410a7fb4084f717c314f7f7a5a Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sat, 8 Aug 2026 09:35:21 -0500 Subject: [PATCH 29/31] doctest_core(docs[adr]): Record spike evidence why: The proposed architecture contained compatibility and lifecycle claims that changed when exercised against real hosts and version floors. what: - Record the typed-core and thin-adapter bakeoff result - Narrow vanilla, diagnostics, line, and pytest compatibility promises - Define runner and registry contracts proven by the implementation - List deferred host lifecycle, reporting, async, and facade work --- docs/adrs/0001-typed-vanilla-doctest-core.md | 414 +++++++++--------- .../0002-runner-conformance-across-cpython.md | 235 +++++----- docs/adrs/0004-diagnostics-as-data.md | 26 +- .../0005-line-recovery-for-nested-blocks.md | 15 +- .../0006-pytest-private-api-compatibility.md | 70 ++- ...0007-host-plugin-registration-lifecycle.md | 52 ++- notes/analyses/24-implementation-bakeoff.md | 178 ++++++++ notes/analyses/README.md | 3 +- 8 files changed, 646 insertions(+), 347 deletions(-) create mode 100644 notes/analyses/24-implementation-bakeoff.md diff --git a/docs/adrs/0001-typed-vanilla-doctest-core.md b/docs/adrs/0001-typed-vanilla-doctest-core.md index 44e813f..33988c4 100644 --- a/docs/adrs/0001-typed-vanilla-doctest-core.md +++ b/docs/adrs/0001-typed-vanilla-doctest-core.md @@ -50,7 +50,7 @@ pytest's `repr_failure` reconstruct locations by slicing that single string. So the blocks must be laid out on a synthetic page with blank-line padding and a clamp, and a wholly-skipped block must be lifted back out to report at all. -**Prompt-free `{testcode}` needs a CPython private.** The per-example loop +**Prompt-free `{testcode}` needs a second execution lane.** The per-example loop [`DocTestRunner.__run`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1344) hard-codes `"single"` ([`Lib/doctest.py:1400`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1400)), @@ -94,9 +94,9 @@ The decoupling is not "group versus block". It is **scheduling identity versus diagnostic identity**: pytest schedules the group, while each `DocTest` keeps its own source location, examples and failure gutter. -One {class}`pytest.Item` per (document, group). Inside it, a tuple of per-block -`DocTest`s run in phase order against one live `globs` mapping that never leaves -the item. +One {class}`pytest.Item` per (document, group). Inside it, immutable per-block +recipes materialize fresh `DocTest`s just before execution. They run in phase +order against one live `globs` mapping that never leaves the item. The execution shape is partly precedented. `sphinx.ext.doctest` runs several `DocTest`s against one shared group namespace — but only for the *test* phase. @@ -112,10 +112,11 @@ why `SphinxDocTestRunner` overrides a private stdlib method to swallow the resulting `IndexError`. Mapping the group onto one {class}`pytest.Item` while each block keeps its own identity is the contribution. -With the default checker that buys, for free and with no override of -`repr_failure` or `reportinfo`: per-block failure locations, per-block gutters, -and per-block "location unknown". -Meanwhile `-k`, `--lf`, `-x`, `--reruns` and every `--dist` mode are structurally +With the default checker that buys per-block failure locations, per-block +gutters, and per-block "location unknown" without synthetic merged source. The +adapter uses its narrow `repr_failure` renderer to retain the comparison-time +checker and does not override `reportinfo`. +Meanwhile `-k`, `--lf`, `-x`, `--reruns` and xdist scheduling are structurally incapable of splitting the shared state, because there is only one item to schedule. @@ -153,47 +154,27 @@ With one `DocTest` per block, every failure therefore carries its own `filename` and `lineno` for free. A block reached through `.. include::` reports the *included* file. A block docutils could not locate carries `lineno=None` and takes pytest's honest `EXAMPLE LOCATION UNKNOWN` branch **without poisoning its -siblings**. The default-checker path requires no override of `repr_failure` or -`reportinfo`. +siblings**. The adapter's narrow failure renderer retains those same per-failure +locations while reusing the checker instance that made each comparison. It does +not override `reportinfo`. This is what makes merging unnecessary: the synthetic page, its blank-line padding and its clamp exist only to reconstruct locations from a single spliced docstring, and there is no spliced docstring here. -**2. `_DocTestRunner__run` is an ordinary attribute override.** Name mangling -rewrites the *call site* at compile time, so the `self.__run(...)` lookup inside -`run()` -([`Lib/doctest.py:1571`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1571)) -resolves through the instance's MRO like any other attribute. A subclass that -defines `_DocTestRunner__run` takes over the loop, with `run()` untouched. - -This page is itself a doctest, so the claim is checked on every run: - -```{doctest} ->>> import doctest ->>> import io ->>> fired = [] ->>> class Runner(doctest.DocTestRunner): -... def _DocTestRunner__run(self, test, compileflags, out): -... fired.append(test.name) -... return super()._DocTestRunner__run(test, compileflags, out) ->>> example = doctest.Example("1 + 1\n", "2\n") ->>> test = doctest.DocTest([example], {}, "demo", "demo.py", 0, None) ->>> Runner().run(test, out=io.StringIO().write) -TestResults(failed=0, attempted=1) - -The subclass method ran, and `run()` was never overridden: - ->>> fired -['demo'] -``` - -Keeping `run()` as stdlib's matters: it owns the save-and-restore of +**2. The ordinary lane does not need to own CPython's loop.** A reporter +subclass can retain failures through `report_failure` and +`report_unexpected_exception` while inheriting the per-example loop unchanged +([`Lib/doctest.py:1286-1314`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1286-L1314)). +Keeping `run()` and its private loop as stdlib's matters: `run()` owns the +save-and-restore of `sys.stdout`, `pdb.set_trace`, `linecache.getlines`, `sys.displayhook`, `_colorize.can_colorize` and the `PYTHON_COLORS`/`FORCE_COLOR` environment variables, all in its own `finally` ([`Lib/doctest.py:1534-1573`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1534-L1573)). -That contract is not reproduced; it is inherited. +That contract is inherited for prompt blocks. Extended `exec` profiles use a +separate bounded runtime and make no claim to inherit it; see +{doc}`0002-runner-conformance-across-cpython`. **3. Making the item the sharing unit dissolves the *affinity* problem.** A live mapping never crosses a process boundary, so there is nothing for xdist to split, @@ -214,7 +195,7 @@ stated here rather than discovered later. | Signal | Granularity | Notes | |---|---|---| -| Failure location, `want`/`got`, gutter | **per block** | default `repr_failure` iterates failures and reads each one's own `DocTest`; custom checkers use the same locations in the narrow rendering branch | +| Failure location, `want`/`got`, gutter | **per block** | the adapter renderer iterates failures, reads each one's own `DocTest`, and uses its retained comparison-time checker | | `EXAMPLE LOCATION UNKNOWN` | **per block** | a block with `lineno=None` does not affect its siblings | | `passed` / `failed` / `skipped` | **per item** | `TestReport.outcome` is one scalar | | JUnit `` | **per item** | node reporters are keyed by node id | @@ -233,16 +214,20 @@ detail, not as a pytest outcome.** Setup and cleanup are infrastructure and do not contribute a passed or skipped test. No extra reports are synthesized. "Typed block detail" needs a channel, or implementers will re-invent skip lifting -or write to stderr. The channel is a `GroupResult` — one `BlockResult` per block, -each carrying phase, outcome, gate reason and location — attached to the item and -rendered in two places: the failure longrepr when the group fails, and a -terminal-summary line at `-rs`. It is **not** visible at default verbosity, and -it never becomes a JUnit `` entry. +or write to stderr. The channel begins with a `GroupResult` — one `BlockResult` +per block, each carrying phase, outcome, gate reason and location — attached to +the item. A versioned, JSON-safe projection must then cross the worker boundary +for terminal rendering. It never becomes a JUnit `` entry. That is a real product loss relative to lifting a gated block into its own item, and it is accepted deliberately: **a gated block inside a mixed group gives up its selectable skip row.** The information survives; the addressable unit does not. +The spike retains `GroupResult` worker-local and reports secondary cleanup +outcomes, but does not yet transport partial-skip detail to the controller or +terminal summary. That projection remains an acceptance gate rather than an +implemented claim. + `subtests` — a builtin pytest plugin since 9.0, exporting `pytest.Subtests` and `pytest.SubtestReport` — can emit per-block outcomes, and is the only sanctioned mechanism that can. It is not adopted here: pytest documents it as experimental, @@ -272,29 +257,29 @@ contracts settings model | Layer | Owns | Must not know | |---|---|---| | `contracts` | Public protocols and immutable contribution records: `DocumentParser`, `ExecutionProfile`, `ExecutionRuntime`, `CheckerFactory`, `Contributor` and `Registrar` | Sphinx, pytest, xdist and host lifecycle objects. Only stdlib and public parser types cross this boundary | -| `settings` | Three immutable facets — `ParseSettings`, `ProjectionSettings`, `RunSettings`. `None` sentinels at the resolve boundary make a future default change announceable | registries, pytest's `Config`, argparse, ini format, Sphinx's `app`. The host extracts; this resolves | -| `model` | `ParsedBlock`, `ParsedOutput`, `BlockKind`, `Phase`, `Diagnostic`, `ProjectedBlock`, `GroupPlan`, `RunContext` and the result types. **No stdlib subclasses.** | docutils, MyST, Sphinx, pytest, xdist, the filesystem. Stdlib imports only | +| `settings` | Three immutable facets — `ParseSettings`, `ProjectionSettings`, `RunSettings`. Hosts resolve their own input and instantiate final settings | registries, pytest's `Config`, argparse, ini format, Sphinx's `app` | +| `model` | `ParsedBlock`, `ParsedOutput`, `BlockKind`, `Phase`, `Diagnostic`, `ProjectedBlock`, `GroupPlan` and the result types. **No stdlib subclasses.** | docutils, MyST, Sphinx, pytest, xdist, the filesystem. Stdlib imports only | | `registry` | A private mutable builder and the public immutable `RegistrySnapshot` consumed by every later layer | host lifecycle objects after the snapshot is frozen | -| `markup/` | Text → `(blocks, diagnostics)`. **The whole docutils vocabulary**: which node classes each kind may arrive as, the `BlockAttributes` stamp, line-number recovery and its per-front-end meaning, `.. include::` attribution, `nodes.comment` traversal, reporter capture, idempotent directive registration, per-kind `option_spec` | Groups as a runtime concept, `DocTest`, pytest, pairing | +| `markup/` | Text → `(blocks, diagnostics)`. Recognized docutils and Sphinx node vocabulary: field-level stamp validation, line-number recovery, `.. include::` attribution, `nodes.comment` traversal, reporter capture, idempotent built-in directive registration, and preservation of custom stamped kind names | Groups as a runtime concept, `DocTest`, pytest, pairing | | `project` | The **only** place grouping exists: `*` expansion, anonymous naming, phase order, `testcode`/`testoutput` pairing, option defaults, name minting. A pure function | docutils, pytest, the filesystem, whether anything will run. Evaluates no user code | -| `runner` | `_DocTestRunner__run`; option merge, `SKIP`-after-merge, `FAIL_FAST`, `report_*` dispatch, version shims. `run_group()` owns phase sequencing, run-time `:skipif:`, the profile's context manager, and the `try`/`finally` guaranteeing `testcleanup` | docutils, markup, pytest. Never overrides `run()` | +| `runner` | Stock CPython execution for prompt blocks and a bounded independent loop for extended profiles. `run_group()` owns materialization, phase sequencing, run-time gates, profile lifetimes, and cleanup after block or gate failures | docutils, markup, pytest. Never overrides CPython's `run()` or private loop | | `pytest_doctest_docutils` | Options, `Document(pytest.Module)`, `DocutilsItem`, group `globs` lifetime, the outcome contract, built-in-plugin composition, surfacing diagnostics | docutils node classes, MyST configuration, grouping rules | -`pytest_doctest_docutils._compat` is the only module that imports +`_pytest_doctest_compat` is the only module that imports `_pytest.doctest`, behind a pinned support matrix. See {doc}`0006-pytest-private-api-compatibility`. ### Settings and the frozen registry -Settings have **lifetimes**, not just precedence. "Resolve exactly once at session -start" cannot be true of document front matter, which does not exist until that -document is parsed. Three scopes: +Settings have **lifetimes**, not just precedence. The spike establishes three +immutable facets and leaves document front matter for a later decision: -| Scope | Owns | Resolved | +| Facet | Owns | Resolved | |---|---|---| -| `SessionSettings` | defaults plus normalized host configuration | once, before collection or direct execution | -| `DocumentSettings` | the front-matter overlay a page is permitted to set | per document, after parsing | -| block / example policy | directive options, then inline `# doctest:` flags | per block, at projection and run | +| `ParseSettings` | diagnostic suppression | before parsing or extracting one document | +| `ProjectionSettings` | unlabelled-block grouping policy | before projecting one document | +| `RunSettings` | runner flags, failure continuation and checker selection | before executing one group attempt | +| block / example policy | directive options, gates, then inline `# doctest:` flags | per block, at projection and run | Not every field shares one ladder, so the precedence is stated per axis. For option flags it follows Sphinx: **runner defaults → directive or output @@ -307,11 +292,13 @@ not user-configurable knobs — exposing them would let a project produce node i no other project can read. `ProjectionSettings.ungrouped` defaults to `"default"`. An unlabelled runnable -block therefore joins the page's `default` group unless the user explicitly asks -for block isolation. This clean-slate default follows Sphinx's author vocabulary -and makes state-sharing opt-out rather than a project-specific surprise. +block therefore joins the page's `default` group unless the caller explicitly +asks for block isolation. This clean-slate core default follows Sphinx's author +vocabulary. The pytest adapter defaults to `"block"` to preserve gp-libs' +released per-block isolation; an explicit, unargumented Sphinx directive still +stamps `groups=["default"]` and shares under either adapter setting. -The **registry** is a separate input resolved after `SessionSettings`. "Frozen" +The **registry** is a separate input resolved before pipeline use. "Frozen" means the public `RegistrySnapshot` contains immutable mappings and records; the mutable builder is private and discarded. Registering after the host freezes its snapshot is an error. Keeping these values separate matters under xdist: settings @@ -325,9 +312,10 @@ builder and every consumer receives the same `RegistrySnapshot`. Host-specific registration timing is a separate decision; the core contract does not import a pytest hook or a Sphinx application. -The direct, pytest and Sphinx lifecycles, including the xdist consistency check, -are specified in {doc}`0007-host-plugin-registration-lifecycle` so this record -does not mistake a host bootstrap policy for a core dependency. +The direct and pytest lifecycles are specified in +{doc}`0007-host-plugin-registration-lifecycle`. Sphinx contribution timing and +an xdist registry manifest remain proposals there; the first spike proves only +resolved-doctree extraction and homogeneous-worker execution. ### Item lifecycle @@ -339,7 +327,7 @@ an implementer cannot get it wrong by omission: `setup()`, `reportinfo()` and `_check_all_skipped()`, so the subclass must define it even though a group holds many tests. `self.dtest` is a synthetic **zero-example** `DocTest` for the group, and its `globs` **is** the canonical - live mapping — the same object every `RunnableBlock`'s test is given. That + live mapping — the same object every freshly materialized test is given. That makes the inherited `setup()` inject fixtures into exactly the right place with no override of the injection itself. @@ -350,23 +338,24 @@ an implementer cannot get it wrong by omission: `super().setup()` so fixtures inject into that same object. Clearing in place rather than rebinding is what keeps `item.globs is run.globs` true for every block, and what stops attempt two of a `--reruns` run from reading attempt - one's mutations. Each `RunnableBlock` is materialized against this object — - its `DocTest.globs` assigned *after* construction, because `DocTest.__init__` - copies. + one's mutations. It does not materialize block tests; `run_group()` does that + immediately before each block runs, after its gates and paired output have + been resolved. 3. **`runtest()`** is overridden. It must not delegate to `DoctestItem.runtest`, which runs a single `dtest` with `clear_globs` defaulting to `True` — that would empty the shared mapping after the first block. It calls `run_group()`, which materializes and runs each - `RunnableBlock` in phase order with `clear_globs=False`, evaluates `:skipif:` + `ProjectedBlock` in phase order with `clear_globs=False`, evaluates `:skipif:` and `:pyversion:` against the live mapping and interpreter, - finalizes each paired `want` from its gated `ExpectedOutput`, and wraps the - body in a `try`/`finally` so cleanup runs whether or not the body raised. When - cleanup *also* fails, the body's failure is the one raised; cleanup's is - recorded in the `GroupResult`. - - The `OutcomeException` re-raise, the `bdb.BdbQuit` → `outcomes.exit` - conversion and `continue_on_failure` are reimplemented here, because - `PytestDoctestRunner` is nested inside a factory and cannot be imported. + finalizes each paired `want` from its gated `ExpectedOutput`, and preserves + cleanup after setup, test, or gate failure. When cleanup *also* fails, the + body's failure is the one raised; cleanup's is recorded in the `GroupResult`. + Profile context entry and exit failures still need the representation decision + in {doc}`0002-runner-conformance-across-cpython`. + + A host-neutral `ExceptionPolicy` classifies exceptions that must propagate. + The pytest adapter supplies pytest's outcome and debugger-exit policy; the + core does not import pytest or reproduce its private runner class. 4. **Outcome** follows [](#the-outcome-contract): only `Phase.TEST` blocks determine pass versus skip. A plan with no test block yields no item. Setup and cleanup are infrastructure: an error there may fail or abort the item, but a @@ -377,22 +366,20 @@ an implementer cannot get it wrong by omission: with grouping. 5. **Failure projection** flattens every `Failed.failures` tuple in block order - and raises pytest's `MultipleDoctestFailures`. With the default pytest checker, - `repr_failure` is inherited unchanged and therefore preserves pytest's exact - output. A contributed checker must supply both `check_output` and - `output_difference`; the item takes one small custom-rendering branch so the - checker that decided the failure also explains it. `repr_failure` does not - render `GroupResult`. Partial-skip detail goes to a report section and the - terminal summary; see [](#the-outcome-contract). - -6. **Reporting across processes.** The controller never sees the item — it - receives serialized `TestReport` dictionaries. So `pytest_runtest_makereport` - copies a **versioned, JSON-safe** block summary onto the report. The summary - contains only the schema version, group name, block name, phase, outcome, - counts and structured skip reason — never exceptions or live objects. pytest - serializes arbitrary report attributes and xdist reconstructs them - controller-side. The rich `GroupResult`, and every exception in it, stays - worker-local. + and raises pytest's `MultipleDoctestFailures`. The item uses one quarantined + renderer for both pytest's checker and contributed checkers so the instance + that decided each failure also explains it. The renderer preserves pytest's + report choice, gutter and per-failure location shape; it does not render + `GroupResult`. Secondary cleanup failures use a report section. The + partial-skip terminal channel described in [](#the-outcome-contract) remains + an acceptance gate. + +6. **Reporting across processes is deferred.** The controller never sees the + item; it receives serialized `TestReport` dictionaries. A complete adapter + therefore needs `pytest_runtest_makereport` to copy a versioned, JSON-safe + block summary onto the report. The spike keeps the rich `GroupResult` and its + exceptions worker-local, so partial-skip detail does not yet reach the + controller or terminal summary. ### Vocabulary @@ -411,7 +398,7 @@ referents. Each term below is decided once and used only that way. | directive | inline `# doctest: +FLAG` | — | a docutils directive with an `option_spec` | Reserved for the docutils meaning. doctest's form is "inline flags" | | optionflags | an int bitmask; `register_optionflag` | `doctest_optionflags` ini | `:options:` plus `doctest_default_flags` | Keep verbatim. `register_optionflag` is the one genuinely cross-library extension point | | setup / cleanup | `setUp`/`tearDown` on the suite builders | fixtures | `testsetup`/`testcleanup` directives | Author-facing names stay Sphinx's; `phase` is the internal ordering axis; a fixture is never "setup" | -| name | a dotted path; `__lt__` compares it as **text** | node id is `parent.nodeid + "::" + name` | the *group* name, shared by every block in it | `DocTest.name` is a unique machine-independent id. The absolute path lives only in `filename` | +| name | a dotted path; `__lt__` compares it as **text** | node id is `parent.nodeid + "::" + name` | the *group* name, shared by every block in it | `DocTest.name` is unique within one document plan; pytest's parent path makes the node id suite-wide. The source path lives in `filename` | ### Data model @@ -431,12 +418,20 @@ Failure: t.TypeAlias = doctest.DocTestFailure | doctest.UnexpectedException class RuntimeOutcome(t.NamedTuple): results: doctest.TestResults failures: tuple[Failure, ...] + skipped: int # explicit because Python 3.10 TestResults cannot carry it + + +class ExceptionPolicy(t.Protocol): + def should_propagate(self, error: BaseException) -> bool: ... + + def is_abort(self, error: BaseException) -> bool: ... class RuntimeSettings(t.NamedTuple): optionflags: int continue_on_failure: bool checker: doctest.OutputChecker + exception_policy: ExceptionPolicy class CheckerFactory(t.Protocol): @@ -458,7 +453,7 @@ class ExecutionProfile(t.Protocol): class ParsedBlock(t.NamedTuple): kind: str # registered BlockKind name - source: str # dedented body, verbatim author text + source: str # dedented, outer-newline-normalized extracted text path: pathlib.Path # the file the text lives in, not the collected document line: int | None # None when docutils could not recover one document_order: int # position among blocks AND outputs; the pairing key @@ -471,8 +466,9 @@ class ParsedBlock(t.NamedTuple): class ParsedOutput(t.NamedTuple): - """A `testoutput` body. Not a block: it never runs.""" + """An expected-output body. Not a block: it never runs.""" + kind: str # the BlockKind.pairs_with name stamped on the source node text: str path: pathlib.Path line: int | None @@ -487,7 +483,6 @@ class BlockKind(t.NamedTuple): phase: Phase profile_name: str # resolved against the frozen registry, not held here pairs_with: str | None - grouped: bool # --- projected: one per block, with everything the runner needs ------------ @@ -534,35 +529,17 @@ class GroupPlan(t.NamedTuple): seed: t.Mapping[str, t.Any] # initial namespace; copied per attempt -# --- run: one attempt materializes fresh stdlib objects -------------------- - - -class RunnableBlock(t.NamedTuple): - """Materialized per attempt. Its `DocTest` is never retained by a plan.""" - - recipe: ProjectedBlock - test: doctest.DocTest # fresh stock objects, built this attempt - runtime: ExecutionRuntime # from the profile factory, this attempt - - -class RunContext: - """One execution attempt. Owns the live mapping; a plan never does.""" - - plan: GroupPlan - globs: dict[str, t.Any] # cleared in place and reseeded per attempt - runtimes: dict[str, ExecutionRuntime] # one per profile used by the group - - # --- results: a discriminated union, so invalid states cannot be built ----- class Counts(t.NamedTuple): + failed: int # may exceed len(Failed.failures) under report-only-first attempted: int skipped: int # a PASSING block can still carry skipped examples class SkipReason(t.NamedTuple): - kind: t.Literal["skipif", "inline-flag", "pyversion", "profile"] + kind: t.Literal["skipif", "inline-flag", "pyversion"] detail: str # the gate expression, the flag, the specifier @@ -576,6 +553,7 @@ class Failed(t.NamedTuple): counts: Counts # PLURAL: continue_on_failure yields several from one block failures: tuple[Failure, ...] + checker: doctest.OutputChecker # the instance that made the comparison class Skipped(t.NamedTuple): @@ -601,9 +579,11 @@ class GroupResult(t.NamedTuple): `ParsedBlock` carries no `want`, because neither owner of a `want` is the parsed block: for a prompt-form block it is *inside* `source` and -{class}`doctest.DocTestParser` extracts it at projection, and for a -`testcode`/`testoutput` pair it is a separate `ParsedOutput`. Conflating the two -was what made "projection owns pairing" untrue. +{class}`doctest.DocTestParser` extracts it at projection, and for a paired block +it is a separate `ParsedOutput`. The output retains its stamped `kind`, so a +contributed `BlockKind.pairs_with` relationship survives extraction without +hard-coding `testoutput`. Conflating the two was what made "projection owns +pairing" untrue. `document_order` is one monotonic sequence shared by runnable blocks and `ParsedOutput` records. Pairing therefore follows the source stream even when an @@ -621,8 +601,9 @@ registration name resolve against the frozen registry. assigns `globs` to it after construction, and a run mutates that mapping — so a plan retaining one would not be a recipe, it would be last attempt's state. Under `--reruns` that is the false-green this design exists to prevent. `ProjectedBlock` -therefore carries the *ingredients*, and `RunContext` materializes fresh stock -`Example` and `DocTest` objects for every attempt. +therefore carries the *ingredients*, and `run_group()` materializes fresh stock +`Example` and `DocTest` objects for every block in every attempt. Attempt-local +runtimes remain local implementation state rather than a public context object. **The ingredients are per example, not per block.** One prompt block routinely yields several {class}`doctest.Example` objects, each with its own `source`, @@ -664,6 +645,9 @@ Four result details are load-bearing: - **`Failed.failures` is plural.** Under `continue_on_failure` one block reports several failures; a singular field silently keeps the first. +- **`Failed.checker` is the comparison-time instance.** A contributed checker + may carry configuration or state, so reconstructing one during pytest failure + rendering can explain the result differently from the object that decided it. - **`Passed` carries counts.** A block can pass *and* have skipped examples — `failed=0 attempted=2 skipped=1` — and a result type without counts loses the skip entirely, which is the same information ADR 0001's outcome contract @@ -672,8 +656,8 @@ Four result details are load-bearing: while an all-`SKIP` doctest has parsed examples and reports them skipped. The reason alone cannot distinguish those cases. - **`SkipReason` is typed.** A skip originates from `:skipif:`, an inline - `# doctest: +SKIP`, `:pyversion:`, or a profile declining to run — and "the gate - expression" describes only the first. + `# doctest: +SKIP`, or `:pyversion:` — and "the gate expression" describes only + the first. Profile decline is not in the initial runtime contract. **Exception precedence is phase-aware, not a single ladder.** Grouping {exc}`KeyboardInterrupt`, a debugger quit, {exc}`pytest.skip`, `xfail` and @@ -682,21 +666,23 @@ a real test failure, while a session exit must never be converted into block dat | Class | Examples | Rule | |---|---|---| -| process, debugger or session abort | {exc}`KeyboardInterrupt`, `SystemExit`, `bdb.BdbQuit`, `pytest.exit` | always propagates from every phase; cleanup still runs, but cannot replace it | +| process, debugger or session abort | core: {exc}`KeyboardInterrupt`; pytest policy also classifies `SystemExit`, `bdb.BdbQuit`, and `pytest.exit` | propagates from every block phase; cleanup runs and cannot replace it | | host outcome from setup or test | `pytest.skip`, `pytest.xfail` | propagates as the host outcome after cleanup | -| doctest mismatch from test | `DocTestFailure`, `UnexpectedException` | retained in source order and projected to the host after cleanup | -| runtime error from setup or test | a gate that raises, or a profile that cannot start | recorded as `Errored`; becomes the primary failure when no abort or host outcome exists | -| cleanup exception or cleanup host outcome | any exception, including `pytest.skip` or `pytest.xfail` | recorded as `secondary` when a primary exists; otherwise becomes the item failure, never a skip or xfail | +| doctest mismatch or ordinary executed exception | `DocTestFailure`, `UnexpectedException` in any phase | retained in source order and projected as doctest failures after cleanup | +| gate or host-owned error from setup or test | a gate that raises, `pytest.fail`, or another propagated host outcome | recorded as `Errored`; becomes the primary failure when no abort or host outcome exists | +| cleanup gate or host-owned error | a propagated error, including `pytest.skip` or `pytest.xfail` | recorded as `secondary` when a primary exists; otherwise becomes the item failure, never a skip or xfail | Profile runtimes are entered through {class}`contextlib.ExitStack`, so a partial -startup unwinds deterministically in reverse. Classifying which exceptions are -pytest outcomes is the pytest adapter's job; the core knows only the phase. +startup unwinds deterministically in reverse. The adapter supplies an +`ExceptionPolicy` that identifies host outcomes and the smaller set of aborts +that outrank every recorded result; the core remains host-neutral while +preserving their propagation semantics. `ParsedBlock.line` being nullable is load-bearing, not defensive. A bare `>>>` block nested in a `.. note::`, a list item or a block quote reports `line=None, source=None` from docutils, and an `.. include::`-ed block numbers against the -*included* file. Both propagate to `DocTest.lineno=None` and pytest's honest -"location unknown", rather than to a fabricated number. +*included* file. The first propagates to `DocTest.lineno=None` and pytest's +honest "location unknown"; the second retains the included path and line. `ProjectedBlock` carries phase and gate because `run_group()` owns phase sequencing, run-time `:skipif:` evaluation and a cleanup `finally` — and cannot do @@ -714,7 +700,7 @@ assertion. **A wildcard block is projected separately per group it joins.** Projection clones the recipe and mints a group-qualified name for each destination. Each -`RunContext` then builds its own `DocTest` from its own recipe. Reusing one +`run_group()` then builds its own `DocTest` from its own recipe. Reusing one `ProjectedBlock` would make its name ambiguous; sharing one materialized `DocTest` would be worse, because `DocTest.globs` is mutable and the second group's assignment would win. @@ -727,7 +713,7 @@ it is not what `sphinx-build` does. Recorded rather than hidden. **`ExecutionProfile` is an immutable factory; `ExecutionRuntime` is per attempt.** A group can mix prompt, `exec` and async blocks, so there is no single per-group -profile. The profile is chosen per block and names a factory; `RunContext` creates +profile. The profile is chosen per block and names a factory; `run_group()` creates one runtime *per distinct profile* the group uses, and blocks sharing a profile share its runtime. An async runtime therefore owns one event loop for the whole group, which is what lets awaited state cross block boundaries. @@ -739,9 +725,9 @@ event-loop lifetime a mode string cannot express. The runtime's context manager what `run_group()` enters, so that lifetime is served without overriding `run()` and stdlib's save-and-restore `finally` stays inherited. -**The ordinary lane does not use the owned loop at all.** For prompt-form blocks — +**The ordinary lane does not use an owned loop at all.** For prompt-form blocks — the overwhelming majority — the runner is a plain reporter subclass over CPython's -*untouched* per-example loop. `_DocTestRunner__run` is invoked only for extended +*untouched* per-example loop. A separate bounded runtime handles extended profiles: `exec` bodies, top-level await, and whatever comes next. Ordinary doctests are then compatible **by construction** rather than by differential testing, and {doc}`0002-runner-conformance-across-cpython`'s harness shrinks to @@ -749,11 +735,11 @@ guarding the extended lane. **A checker owns both comparison and explanation.** The default pytest registration constructs pytest's checker, preserving `ALLOW_UNICODE`, -`ALLOW_BYTES`, `NUMBER` and its inherited failure representation exactly. A -contributed `CheckerFactory` constructs a fresh checker for each runtime. The -same instance performs `check_output()` and `output_difference()`; using pytest's -private `_get_checker()` only at rendering time would let one checker reject the -example and another explain why. +`ALLOW_BYTES` and `NUMBER`. A contributed `CheckerFactory` constructs a fresh +checker for each runtime. The same instance performs `check_output()` and +`output_difference()` through the adapter's pytest-shaped renderer; using +pytest's private `_get_checker()` only at rendering time would let one checker +reject the example and another explain why. **Which docutils node classes a kind may arrive as is a front-end concern, not a `BlockKind` field.** `testsetup`, `testcleanup` and any `:hide:` block are @@ -785,7 +771,10 @@ layer that never changes what is constructed. def extract_blocks( - doctree: nodes.document, *, settings: ParseSettings + doctree: nodes.document, + *, + settings: ParseSettings, + registry: RegistrySnapshot, ) -> ParseResult: ... ``` @@ -793,7 +782,10 @@ layer that never changes what is constructed. one extractor is the *point* of the split, and a second implementation would reintroduce the standalone-versus-Sphinx divergence it exists to prevent. Standalone reST and MyST use both halves; a Sphinx extension calls only the - extractor, on the doctree it already resolved. + extractor, on the doctree it already resolved. Passing the same frozen + registry is load-bearing: extraction derives expected-output stamp names from + registered `BlockKind.pairs_with` relationships before projection resolves + them. **`DocumentParser` is not a {class}`doctest.DocTestParser`.** The two signatures are incompatible — stdlib's is `parse(self, string, name='')` @@ -836,19 +828,19 @@ layer that never changes what is constructed. One genuine nominal edge does exist: `DocTestSuite` sorts its results, and `DocTest.__lt__` returns `NotImplemented` for a non-`DocTest`, so a custom finder must return real `DocTest` objects. -- **`TypedDict` at the docutils boundary.** `BlockAttributes` types what a - directive stamps on a node, with one narrowing accessor that validates once. - This is where `Any` currently enters: `str(node.get("test") or ...)` and - `dict(node.get("options") or {})` are runtime coercions paid for a static hole, - and typeshed's docutils stub makes it worse — its `get(key, failobj: _T) -> _T` - claims `_T` even when the key is present holding something else. +- **Field-level narrowing at the docutils boundary.** External directives stamp + dynamically typed node attributes, so a `TypedDict` would falsely imply that + producers honor an owned schema. Small accessors validate each consumed field; + `ParsedBlock` and `ParsedOutput` are the first trusted typed boundary. - **`t.Literal` for closed vocabularies**, derived from one source of truth so a public signature and a config field cannot diverge. - **Plain `int` keys for optionflags.** {class}`enum.IntFlag` was considered and rejected; see [](#alternatives-rejected). -- **`py.typed` ships.** The project already runs mypy strict over `src` and - `tests`, and the marker file does not exist, so every consumer sees the package - as untyped. This is a packaging defect independent of the rest of this ADR. +- **`doctest_core/py.typed` ships.** The project already runs mypy strict over + `src` and `tests`; the wheel and sdist include the marker so consumers see the + core's public types. The legacy flat `doctest_docutils` and + `pytest_doctest_docutils` facades remain untyped compatibility surfaces unless + they later move behind typed packages or stubs. ### What "vanilla-compatible" promises @@ -859,22 +851,23 @@ several different strengths. |---|---| | `Example` / `DocTest` runtime types | **Exact.** Stock instances, never subclassed for metadata | | plain-text parsing | **Exact.** The stdlib lane uses `DocTestParser` unmodified | -| option flags and checkers | **Exact.** `register_optionflag` and the stdlib checker contract, with pytest's `ALLOW_UNICODE`/`ALLOW_BYTES`/`NUMBER` reachable | +| option flags and checkers | **Exact for the stdlib contract.** `register_optionflag` and `OutputChecker` remain stock; pytest's `ALLOW_UNICODE`/`ALLOW_BYTES`/`NUMBER` are available through its adapter | | prompt-block execution | **Exact.** CPython's own per-example loop, unmodified | -| `DocTestFinder`-shaped Python-object discovery | **Shaped**, as a separate adapter | -| `DocFileSuite` / `DocTestSuite` | **Façade only**, over the stdlib lane. A group plan cannot be expressed through an API that returns one `DocTest` per parser call | +| `DocTestFinder`-shaped Python-object discovery | **Deferred.** The spike implements document-text discovery only | +| `DocFileSuite` / `DocTestSuite` | **Not implemented by the spike.** A future stdlib-shaped façade can cover the plain lane, but cannot express one shared group through an API returning independent `DocTest`s | | `{testcode}`, async, groups, phases, Sphinx gates | **Deliberate extension.** No stdlib equivalent to be compatible with | | pytest collection, fixtures, reporting | pytest's own contracts, composed with rather than replaced | -| Sphinx **node** vocabulary | **Exact.** The `BlockAttributes` stamp is byte-compatible | +| Sphinx **node** vocabulary | **Extractor-compatible.** The core accepts Sphinx's stamps and may retain a metadata superset | | Sphinx **execution** | **Not promised.** See below | **The Sphinx promise is narrow, and this record narrows it deliberately.** What is offered is an *extractor over a Sphinx-resolved doctree* — a pure function from a doctree to blocks, callable from an extension. -{doc}`0007-host-plugin-registration-lifecycle` defines how Sphinx extensions -contribute capabilities, but not a Sphinx execution lifecycle or result channel. -Inventing those would be the builder that [](#alternatives-rejected) turns down. -The record promises doctree consumption and nothing more. +{doc}`0007-host-plugin-registration-lifecycle` proposes how Sphinx extensions +could contribute capabilities, but the spike implements neither that lifecycle +nor a Sphinx execution or result channel. Inventing a result channel would be the +builder that [](#alternatives-rejected) turns down. The implemented promise is +doctree consumption and nothing more. ## Constraints @@ -895,8 +888,9 @@ cited. The full derivation is in `notes/analyses/`. | Custom flag names must be registered at import; ints are `1 << len(OPTIONFLAGS_BY_NAME)` and an unregistered name makes a page fail to **parse** | [`doctest.py:153`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L153) | `report_skip` does not exist at v3.14.2 — the runner has only `report_start`, -`report_success`, `report_failure` and `report_unexpected_exception`. A runner -that owns the loop must probe for it rather than assume it. +`report_success`, `report_failure` and `report_unexpected_exception`. The prompt +lane inherits that surface. The extended runtime does not emulate reporter-hook +events. ### pytest (9.1.1) @@ -905,7 +899,7 @@ that owns the loop must probe for it rather than assume it. | `repr_failure` reads each failure's own `test` — the fact this design is built on | [`doctest.py:317-344`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L317-L344) | | `DoctestItem.setup()` does `self.dtest.globs.update(globs)`, so the mapping must be mutable and survive collection → setup → run | [`doctest.py:288-293`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L288-L293) | | `runtest()` calls `run(self.dtest, out=failures)` with `clear_globs` defaulting to `True` — which would empty a shared mapping after the first block | [`doctest.py:295-303`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L295-L303) | -| `PytestDoctestRunner` is defined *inside* `_init_runner_class()` and is not importable, so the `OutcomeException` re-raise, `BdbQuit` → `outcomes.exit` and `continue_on_failure` handling do **not** come for free | [`doctest.py:178-181`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L178-L181) | +| `PytestDoctestRunner` is defined *inside* `_init_runner_class()` and is not importable, so its outcome and continuation policy must be mapped at the adapter boundary rather than inherited | [`doctest.py:178-181`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L178-L181) | | A page must be a `pytest.Module` with `obj = None` as a **class** attribute, or the `Module` machinery tries to import the `.rst`/`.md` file | [`doctest.py:420-421`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L420-L421) | | Conftest autouse fixtures reach page items through `FixtureManager.pytest_plugin_registered`, **not** through a collector calling `parsefactories` — that call is `DoctestModule`-only, for fixtures defined in the collected `.py` itself | [`fixtures.py`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/fixtures.py), [`doctest.py:556`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L556) | | `_is_doctest` claims any `.txt`/`.rst` **initial path before consulting `--doctest-glob`**, so `pytest docs/page.rst` is claimed by the built-in plugin regardless of glob | [`doctest.py:148-152`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L148-L152) | @@ -960,7 +954,7 @@ an answer; the position taken and its price are recorded. **A vanilla `DocTest` cannot carry the front-end's metadata.** It has exactly `(examples, globs, name, filename, lineno, docstring)`. *Position:* all extension -metadata stays on `ProjectedBlock`, `RunContext` and the result records. Stock +metadata stays on `ProjectedBlock` and the result records. Stock `DocTest` and `Example` objects remain exact compatibility objects, not metadata carriers. *Price:* a consumer holding only the stdlib object sees only stdlib semantics; it must retain the core recipe to inspect groups, profiles or gates. @@ -985,9 +979,8 @@ True ``` A block's execution policy is **uniform across its examples**, so it belongs on -`ProjectedBlock`, not on the examples. The runner establishes an *active -execution request* immediately before delegating to stdlib `run()` and clears it -in a `finally`; `_DocTestRunner__run` reads it. Stock `Example` objects stay +`ProjectedBlock`, not on the examples. The selected execution profile receives a +fresh stock `DocTest` immediately before execution. Stock `Example` objects stay stock, and nothing in the compatibility kernel is subclassed for metadata at all. **Node-id granularity versus shared state.** *Position:* decouple them — N @@ -1024,22 +1017,27 @@ impossible. **Sphinx compatibility versus silent-loss behaviours.** Sphinx silently discards an orphan `testoutput`, silently discards a `testoutput` following a `doctest` block, silently overwrites a duplicate `testoutput`, and silently ignores -`:pyversion:` on a `testcode`. *Position:* keep the behaviour, add a diagnostic -with a stable code for each of the four. *Price:* a page warns under pytest and -is silent under `sphinx-build`; the results still match. - -**Guaranteed cleanup versus Sphinx's setup-failure short-circuit.** When setup +`:pyversion:` on a `testcode`. *Position:* enforce `:pyversion:` consistently on +extended blocks rather than preserve Sphinx's silent loss. The spike still +ignores orphan and misplaced outputs without diagnostics; duplicate outputs use +Sphinx's last-one-wins rule, also without a diagnostic. *Price:* +`testcode :pyversion:` can run differently from `sphinx-build`; diagnostics for +the silent cases remain an acceptance gap. + +**Guaranteed block cleanup versus Sphinx's setup-failure short-circuit.** When setup fails, Sphinx returns before the cleanup phase, skipping page `testcleanup` -blocks and `doctest_global_cleanup` alike. *Position:* run cleanup -unconditionally in a `try`/`finally`, because a page that spawns a server in -setup and fails mid-way should not leak it. *Price:* a page whose setup fails +blocks and `doctest_global_cleanup` alike. *Position:* once profile runtimes have +opened, run cleanup after block and gate failures because a page that spawns a +server in setup and fails mid-way should not leak it. Profile context entry and +exit failures remain open. *Price:* a page whose setup fails leaves different residue under pytest than under `sphinx-build`, and that is a deliberate divergence rather than an oversight. -**Owning the loop versus tracking CPython.** *Position:* own it, because -constraint 4 makes it the only way to control compile mode without a -process-global patch or a code-object clone. *Price:* a version shim and a -conformance harness. See {doc}`0002-runner-conformance-across-cpython`. +**Owning the extended loop versus tracking CPython.** *Position:* inherit the +prompt loop unchanged and own only the bounded extended runtime, because that is +the only way to control compile mode without a process-global patch or a +code-object clone. *Price:* each extended profile needs an explicit semantic +matrix. See {doc}`0002-runner-conformance-across-cpython`. ## What this avoids @@ -1148,24 +1146,31 @@ conformance test in CI. ### Positive - Failure locations are correct by construction, including through `.. include::`; - the default-checker path needs no `repr_failure` or `reportinfo` override. + the adapter's renderer needs no synthetic merged source or `reportinfo` + override. - A block docutils cannot locate degrades to an honest disclaimer instead of a fabricated line, and does not affect its siblings. -- Every `--dist` mode works, because there is no shared state to split. +- Shared groups require no affinity scheduler. The spike exercises xdist's + `load` and `worksteal` modes; other modes retain the same one-item boundary but + remain outside its evidence. - No CPython code-object clone, and no process-global rebinding of anything. -- Collection runs no author-supplied Python, so `--collect-only` has no side - effects and the largest source of worker divergence is removed. +- Collection does not execute collected doctest Python or evaluate gates, so + `--collect-only` removes the largest source of worker divergence. Parser + directives, includes, and plugin registration may still have side effects. - Grouping is one pure function with no docutils, pytest or filesystem dependency, and is testable without any of them. -- A new block kind is a registration, not an edit to a method branching on string - literals. -- Parse diagnostics become values with stable codes rather than stderr writes and - mid-parse aborts. +- A new block kind can provide projection policy and name a custom expected-output + stamp through registration. A new markup spelling also needs a parser or + stamped-node contribution; the first spike proves preservation and pairing of + custom stamps, not directive registration by name alone. +- Parse diagnostics become values rather than stderr writes and mid-parse + aborts. Project-owned codes are stable; docutils-originated classification is + provisional as recorded in {doc}`0004-diagnostics-as-data`. ### Tradeoffs -- The per-example loop is this project's to maintain across supported - interpreters, including two version-shaped divergences. +- The extended per-example loop is this project's to maintain across supported + interpreters. Prompt-form doctests continue to inherit CPython's loop. - A page containing Sphinx `{testcode}` blocks produces `DocTest`s a stock runner cannot run, because `compile("a = 1\nb = 2\n", "", "single")` raises. Prompt-form blocks — the overwhelming majority — run perfectly on an unmodified @@ -1176,20 +1181,21 @@ conformance test in CI. ### Risks -**Runner drift.** A CPython refactor that inlines the loop into `run()` would -silently route execution back to stdlib — invisible for prompt-form blocks, -immediately broken for `{testcode}`. Mitigated by the conformance harness in -{doc}`0002-runner-conformance-across-cpython`, gating on capability probes rather -than a `sys.version_info` ladder. +**Runner drift.** Prompt profiles inherit CPython changes directly. Extended +profiles deliberately reproduce only a bounded subset, so a new doctest behavior +must be considered explicitly rather than assumed. Mitigated by the conformance +matrix in {doc}`0002-runner-conformance-across-cpython` and capability probes for +version-shaped result objects. -**pytest private API.** Four private helpers and a subclassed item. Mitigated by +**pytest private API.** Collector, runner-option, failure and representation +helpers remain private. Mitigated by quarantining them in one module behind a pinned matrix; see {doc}`0006-pytest-private-api-compatibility`. **Foreign directive registration.** `Sphinx.add_directive` overrides existing registrations unconditionally, so `sphinx.ext.doctest` loaded in the same interpreter can replace these directive classes. Mitigated by reading -`BlockAttributes` off the node — byte-compatible with what Sphinx stamps — rather +extractor metadata off the node — compatible with what Sphinx stamps — rather than depending on this project's own classes having run. **Over-suppressed diagnostics.** Suppressing one code too many turns a broken page @@ -1199,8 +1205,8 @@ the narrow default set in {doc}`0004-diagnostics-as-data`. ## Relationship to other ADRs This ADR fixes the architecture. Six decisions it defers get their own records: -{doc}`0002-runner-conformance-across-cpython` (how the owned loop is proven -equivalent), {doc}`0003-rejecting-per-block-items` (why shared per-block items +{doc}`0002-runner-conformance-across-cpython` (the stock prompt lane and bounded +extended-runtime matrix), {doc}`0003-rejecting-per-block-items` (why shared per-block items are rejected), {doc}`0004-diagnostics-as-data` (what is reported and what is suppressed), {doc}`0005-line-recovery-for-nested-blocks` (the optional last step), {doc}`0006-pytest-private-api-compatibility` (the quarantine and its @@ -1213,10 +1219,12 @@ points). The core produces real {class}`doctest.DocTest` objects holding real {class}`doctest.Example` objects. `Example.source` is the stdlib-normalized executable body — prompts and indentation stripped, trailing newline added, the -stripped column recorded in `Example.indent` — not a synthesized wrapper and not -the author's verbatim text, which is what `ParsedBlock.source` holds. Everything -else — groups, phases, pairing, diagnostics, distribution — is a layer above that -fact, and no layer reaches around another. +stripped column recorded in `Example.indent` — not a synthesized wrapper. +`ParsedBlock.source` is the dedented, outer-newline-normalized body extracted +from markup. Prompt projection applies stdlib normalization to it; the exec lane +uses it as one indent-zero recipe. Everything else — groups, phases, pairing, +diagnostics, distribution — is a layer above that fact, and no layer reaches +around another. The unit that shares a `globs` mapping is the unit pytest schedules. That is the one invariant every other property in this document follows from, and it is not diff --git a/docs/adrs/0002-runner-conformance-across-cpython.md b/docs/adrs/0002-runner-conformance-across-cpython.md index afd1889..d699c7d 100644 --- a/docs/adrs/0002-runner-conformance-across-cpython.md +++ b/docs/adrs/0002-runner-conformance-across-cpython.md @@ -7,106 +7,141 @@ Date: 2026-08-02 ## Context -{doc}`0001-typed-vanilla-doctest-core` decides that the runner owns the -per-example loop by defining `_DocTestRunner__run` in a subclass, rather than -cloning CPython's code object or rebinding `doctest.compile` process-wide. - -Owning the loop means owning the private state it writes into, and that state has -changed shape inside this project's supported interpreter range. Three -divergences are known: - -**The outcome accumulator changed name and arity.** On 3.10 through 3.12 it is -`__record_outcome(self, test, f, t)` writing into `self._name2ft`; on 3.13 and -later it is `__record_outcome(self, test, failures, tries, skips)` writing into -`self._stats` -([`Lib/doctest.py:1485`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1485)). -A loop that calls the wrong one leaves `summarize()` reporting zeros for a -passing file — a silent, total failure of the reporting path. - -**`TestResults` gained a third value that is not a tuple field.** It carries -`skipped` as an extra instance attribute -([`Lib/doctest.py:114`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L114)), -so `TestResults(f, a, skipped=s)` works on 3.13+ and raises on earlier versions. - -**`report_skip` does not exist at v3.14.2.** The runner has only `report_start`, -`report_success`, `report_failure` and `report_unexpected_exception` -([`Lib/doctest.py:1286-1314`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1286-L1314)). -It appears in later prereleases, so a loop must probe rather than assume in -either direction. - -A fourth risk has no current instance but would be silent: a CPython refactor -that inlines the loop into `run()` would route execution back to stdlib. That is -invisible for prompt-form blocks and immediately broken for `{testcode}`. - -## Question - -How is an owned per-example loop proven equivalent to the interpreter's own, -continuously, without a `sys.version_info` ladder? - -## Direction - -A differential conformance harness, run in CI on every supported interpreter, -gating the build step that lands the runner. - -**Scoped to the extended lane.** {doc}`0001-typed-vanilla-doctest-core` runs -ordinary prompt blocks on CPython's untouched per-example loop, so those need no -differential proof — they *are* the reference. The owned `__run` is invoked only -for `exec` bodies, top-level await and future profiles, and that is what this -harness guards. It is a smaller obligation than an unconditionally owned loop, -and it is the reason owning the loop is affordable at all. - -A fixed case matrix — pass, fail, unexpected exception, `SyntaxError`, all -examples skipped, partially skipped, `FAIL_FAST`, `REPORT_ONLY_FIRST_FAILURE`, -`IGNORE_EXCEPTION_DETAIL`, and an exec-mode body — is run through both this -runner and a stock {class}`doctest.DocTestRunner`, asserting the captured -`report_*` text, `summarize()` output, the accumulator contents, and the result -as `(failed, attempted, skipped)`. - -**Assert the triple, not `TestResults` equality.** `TestResults` is a two-field -namedtuple carrying `skipped` off-tuple, so `==` compares only two of the three -values and a skip-count regression passes silently. `attempted` is also -incremented *before* the `SKIP` check, so a skip that wrongly executes moves -neither counter — it is invisible to both the tuple and to `summarize()` at zero -failures, and only the `report_*` text distinguishes it. - -The exec-mode case is the one the two runners are *meant* to disagree on, and it -still compares against stock. `compile()` raises on a multi-statement body, but -that call sits inside the loop's own `try` -([`Lib/doctest.py:1398-1408`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1398-L1408)), -so a stock {class}`doctest.DocTestRunner` catches the `SyntaxError` and records -it as an unexpected exception rather than propagating it: one -`report_unexpected_exception` call, `TestResults(failed=1, attempted=1)`, and -`_stats` at `(1, 1, 0)`. Only {class}`doctest.DebugRunner` — and pytest's runner -beneath it — converts that into a raise, as -{exc}`doctest.UnexpectedException`. So the case is asserted as a pair: stock -records the failure, this runner records a pass. A regression that silently -reverts to `"single"` mode shows up as the two converging. - -**What else belongs in the matrix, and what does not.** Add `report_*` hook -events — the only channel that distinguishes a skip which wrongly *executed*, -since `attempted` increments before the `SKIP` check and neither counter moves — -and repeated runs of one test, which exercise accumulator arithmetic across -calls. - -Cross-block `FAIL_FAST` and cleanup aggregation stay out. Both are properties of -`run_group()` rather than of the per-example loop, so a stock runner offers -nothing to compare them against; they belong to -{doc}`0001-typed-vanilla-doctest-core`'s item-lifecycle tests. A -{exc}`pytest.skip` raised inside an example and a debugger exit are likewise -pytest-layer concerns, testable only through a pytest session. - -Version handling is by capability probe, never by version comparison, so a -backport, a vendored interpreter or a fork behaves correctly rather than by -coincidence. {doc}`0001-typed-vanilla-doctest-core` rejects an import-time guard -that raises: a `pytest11` plugin that aborts at import takes down suites whose -majority of tests never touch a doctest. +{doc}`0001-typed-vanilla-doctest-core` has two execution lanes with different +compatibility claims. + +Prompt-form blocks are ordinary {class}`doctest.DocTest` objects executed by +CPython's own {class}`doctest.DocTestRunner` loop. The core subclasses only the +reporting hooks that retain failures for an embedding host. It does not override +`run()` or `_DocTestRunner__run`. + +Extended blocks such as Sphinx `testcode` cannot use that loop unchanged. CPython +compiles each example in `"single"` mode, while a `testcode` body may contain +several statements and requires `"exec"`. There is no stdlib execution mode to +select and therefore no exact-compatibility claim to make. + +The supported interpreters also expose different result semantics. Python 3.10 +increments `tries` only after an example passes its `SKIP` gate +([`Lib/doctest.py:1326-1337`](https://github.com/python/cpython/blob/v3.10.19/Lib/doctest.py#L1326-L1337)). +Python 3.14 increments `attempted` before the gate and records `skips` separately +([`Lib/doctest.py:1353-1379`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L1353-L1379)). +`TestResults` gained its `skipped` attribute with that newer shape +([`Lib/doctest.py:114-126`](https://github.com/python/cpython/blob/v3.14.2/Lib/doctest.py#L114-L126)). +The prompt lane must expose skipped examples without silently rewriting the +interpreter's own `attempted` value. The extended lane has no CPython count to +inherit and defines its own stable count below. + +## Decision + +Keep the two lanes structurally separate. + +The prompt runtime delegates to CPython's untouched per-example loop with +`clear_globs=False`. Its reporter subclass may retain +{class}`doctest.DocTestFailure` and {class}`doctest.UnexpectedException`, and may +propagate exceptions selected by the host's `ExceptionPolicy`; it does not own +compilation, option merging, comparison, debugger setup, display hooks, linecache +patching, or result accounting. + +An extended execution profile owns an independent, deliberately smaller runtime. +It accepts a fresh stock `DocTest` plus resolved `RuntimeSettings` and returns a +`RuntimeOutcome`. The initial `exec` runtime owns these semantics: + +- merge runner flags with per-example options, then honor `SKIP` and fail-fast; +- derive active future flags from the live `globs`, compile in `"exec"` mode, + and pass `dont_inherit=True` so the core module's future imports cannot leak; +- capture and restore stdout around every example; +- compare expected exceptions against the exception-only tail, including + `SyntaxError` normalization and `IGNORE_EXCEPTION_DETAIL`, while retaining + captured stdout for failure rendering; +- use the injected checker for comparison and retain stock failure objects; +- propagate host-owned outcomes through `ExceptionPolicy`; and +- leave group phase ordering, cleanup, and exception precedence to + `run_group()`. + +It does not update a `DocTestRunner` accumulator, call `report_*`, implement +`summarize()`, patch the debugger, or claim byte-for-byte output parity with the +prompt lane. A direct stdlib-shaped facade may translate `GroupResult` into the +version-specific accumulator needed by `summarize()`; that compatibility shim is +separate from execution. + +## Conformance gate + +The prompt lane is compatible by construction, but still runs on every supported +Python to catch subclass-state collisions and changes to reporter signatures. +Its tests assert stock object types, per-example option merging, fail-fast, +partial skips, repeated fresh materialization, and restoration of the shared +mapping contract. + +The extended lane has a behavioral matrix rather than a comparison against +CPython's `"single"` compiler mode. Before it is accepted, the matrix covers: + +| Behavior | Required assertion | +|---|---| +| pass and mismatch | stock failure objects, counts, and checker identity | +| future flags | no ambient inheritance; an explicitly imported feature persists through group `globs` | +| unexpected exception and `SyntaxError` | stock exception shape and stable traceback ownership | +| output before exception | defined capture and rendering behavior | +| all and partial skip | examples examined, including skips, plus an explicit skipped count on every interpreter | +| fail-fast and report-only-first | execution and reporting policies remain distinct | +| checker options | `IGNORE_EXCEPTION_DETAIL` and contributed checker behavior | +| process state | stdout is restored; debugger, display-hook, and linecache support is explicitly accepted or excluded | +| repeated calls | runtime-local state cannot leak between attempts | + +Group cleanup after failure, pytest outcomes, fixture injection, reruns, and xdist +belong to host and `run_group()` acceptance tests. They are not evidence about an +individual execution profile. + +Version handling uses capability probes, not `sys.version_info`. The prompt +runtime preserves CPython's own `attempted` value. The extended runtime counts +each example it examines, including an example skipped before compilation; on +interpreters whose `TestResults` cannot carry `skipped`, `run_group()` reconstructs +that value from the materialized test and stores it in `Counts`. + +## Alternative rejected + +Defining `_DocTestRunner__run` for extended profiles was rejected by the +implementation bakeoff. It couples a small `"exec"` requirement to private +accumulators, private outcome-recording arity, report-hook sequencing, debugger +machinery, and `summarize()` behavior that the host-neutral runtime does not use. +It is more code and a larger compatibility promise without making extended +syntax vanilla. + +Cloning and patching CPython's code object or rebinding `doctest.compile` +process-wide remain rejected. Both make unrelated doctest execution depend on +global mutable state. + +The spike still has two smaller CPython-private parser dependencies: +`DocTestParser._EXAMPLE_RE` recognizes prompt-form literal blocks and +`DocTestParser._EXCEPTION_RE` extracts the expected exception tail from paired +output. They do not couple execution to private runner state, but they are still +compatibility debt and need explicit probes across the supported Python matrix. +The legacy direct facade's use of `doctest._load_testfile` is outside the typed +core but belongs in the facade's own compatibility inventory. + +## Consequences + +- Ordinary doctests inherit CPython behavior directly rather than through a + differential approximation. +- Extended profiles state their semantic subset and can manage attempt-scoped + resources through their context manager. +- CPython's pre-3.13 and current prompt-lane skip counters remain observable; + extended profiles expose their separate version-independent count through + `Counts`. +- The direct compatibility facade needs a small version-shaped statistics shim + if it promises stdlib `summarize()` and `master.merge()` behavior. +- The direct facade cannot reproduce the complete verbose + `Trying`/`Expecting`/`ok` stream from `GroupResult`, because the core retains + failures but not successful per-example reporter events. Failure and summary + rendering remain stock-shaped. +- Each new execution profile owns its own behavioral matrix; adding async does + not expand the prompt lane's maintenance surface. ## Open -- Whether the harness asserts on `report_*` text verbatim, or on a normalized - form — verbatim is stricter and will churn when CPython adjusts wording. -- Whether a probe failure degrades to stdlib's loop with a diagnostic, or fails - the affected items loudly. Degrading is silent for prompt-form blocks, which is - the argument against it. -- The floor: whether supporting 3.10's `_name2ft` shape is worth its shim once - that version reaches end of life. +- Complete the extended matrix for report-only-first and repeated runtime calls. +- Probe the two private parser regex contracts on every supported Python. +- Decide whether extended runtimes should reproduce doctest's debugger, + display-hook, and linecache behavior or explicitly exclude interactive + debugging. +- Define how a profile context-manager entry or exit failure is represented while + still allowing an already-open cleanup profile to run. diff --git a/docs/adrs/0004-diagnostics-as-data.md b/docs/adrs/0004-diagnostics-as-data.md index 45ed820..173bc03 100644 --- a/docs/adrs/0004-diagnostics-as-data.md +++ b/docs/adrs/0004-diagnostics-as-data.md @@ -77,17 +77,35 @@ code to a docutils message is unsettled, which is why this record stays `Draft`. Every diagnostic raised by this project's own layers defaults to visible, and `level="error"` from those layers fails collection with the file and line named. -A page whose only block fails to parse, and a page with a malformed `:options:` -value, must both produce a collection error rather than collecting nothing and -passing. +A page whose only block fails to parse must produce a collection error rather +than collecting nothing and passing. A malformed `:options:` value follows +Sphinx's warning severity, but that warning must be visible rather than silently +discarded by the host. Expose promotion and suppression by code so a project can tune the set without a global on/off switch. +## Spike result + +The spike captures reporter messages as typed values, deduplicates messages seen +through both the observer and the doctree, and suppresses both messages emitted +for an unknown role. It provisionally classifies docutils messages by normalized +message substrings because docutils supplies no stable code. Unknown directives +remain visible in `ParseResult.diagnostics`. + +That proves capture and normalization, not host disposition. The pytest adapter +does not yet fail collection for an unsuppressed error or surface warnings with +source attribution. Until that policy and its reST/MyST wording matrix exist, a +malformed body-owning directive can still collect no tests without failing the +session, and malformed doctest options can warn only inside the retained parse +result. The direct facade also does not render those diagnostics. This record +therefore remains `Draft`. + ## Open - Whether diagnostics surface as {class}`pytest.PytestWarning` subclasses, giving - `-W error::` control for free, or as a dedicated report section. + `-W error::` control for free, or as collection errors and dedicated report + sections. Errors must not be silently ignored by the host. - **What classifies a code-less docutils message.** The options are an owned, version-pinned message-text table with a test that fails on upstream rewording (and which must handle two dialects — reST's `Unknown directive type "x".` at diff --git a/docs/adrs/0005-line-recovery-for-nested-blocks.md b/docs/adrs/0005-line-recovery-for-nested-blocks.md index 784f7c7..6397d5e 100644 --- a/docs/adrs/0005-line-recovery-for-nested-blocks.md +++ b/docs/adrs/0005-line-recovery-for-nested-blocks.md @@ -10,8 +10,8 @@ Date: 2026-08-02 docutils does not report a usable line for every node, and what it reports differs by front-end and by version. -At **docutils 0.21.2** — which this project does not pin but does resolve, via -its Sphinx and myst-parser constraints — a bare `>>>` block +At **docutils 0.21.2** — the newest line convention in this project's current +`docutils >= 0.20.1, < 0.22` range — a bare `>>>` block nested in a `.. note::`, a list item, a block quote or a `{tab}` directive reports `line=None, source=None`. A top-level reStructuredText `doctest_block` reports its **last** line. A MyST fence reports its **first** line. An @@ -79,7 +79,10 @@ Resolved versions per interpreter, with `docutils >= 0.22` requested: | 3.12–3.14 | 0.22.4 | 5.1.0 | 9.1.0 | Today's lock resolves Sphinx 8.1.3 on Python 3.10 and 8.2.3 elsewhere, and -neither permits docutils 0.22. +neither permits docutils 0.22. The package therefore caps docutils below 0.22 +until those support-policy steps can move together; silently accepting 0.22 +would apply the old last-line correction to its already-correct first-line +nodes. ## Consequences @@ -93,6 +96,12 @@ both reStructuredText and MyST report the first line — but the normalization layer stays, because the conventions still differ below the floor and a front-end is the right place to know which it is dealing with. +The standalone MyST parser recovers root-document body lines by scanning the +root source text. It deliberately does not apply that stamp to nodes whose +physical source is an included file: the root bytes cannot prove an included +line. Exact standalone MyST include-line fidelity remains open unless the parser +retains the included source text or supplies an absolute body line itself. + Every line-convention claim elsewhere in these records is version-qualified. A statement about "docutils" that does not name a version is a bug in the statement. diff --git a/docs/adrs/0006-pytest-private-api-compatibility.md b/docs/adrs/0006-pytest-private-api-compatibility.md index 175bba4..5aeb7d0 100644 --- a/docs/adrs/0006-pytest-private-api-compatibility.md +++ b/docs/adrs/0006-pytest-private-api-compatibility.md @@ -52,22 +52,29 @@ or already reported a collection error, which deselection cannot retract. What private surface is depended on, and how does a pytest release that changes it fail? -The current surface is `_get_checker`, `get_optionflags`, -`_get_continue_on_failure`, `_get_report_choice` and `MultipleDoctestFailures`. -Not everything in the quarantine is equally risky: {class}`pytest.DoctestItem` is -**public** — exported from `pytest` — so subclassing it is ordinary API use. The -collector class filtered out of the multicall result is private, and that filter -is the part that needs a version matrix. `_init_runner_class` is explicitly *not* usable: +The spike's private surface is `_get_checker`, `get_optionflags`, +`_get_continue_on_failure`, `_get_report_choice`, `DoctestTextfile`, +`MultipleDoctestFailures`, `ReprFailDoctest`, `_pytest._code` representation +classes, and the Darwin capture method on the public item. Not everything in the +quarantine is equally risky: +{class}`pytest.DoctestItem` is **public** from pytest 7.2 onward, so subclassing +it is ordinary API use and establishes the adapter's minimum pytest. The +collector class filtered out of the +multicall result and the representation helpers are private and need a version +matrix. `_init_runner_class` is explicitly not used: `PytestDoctestRunner` is defined inside it ([`_pytest/doctest.py:178-181`](https://github.com/pytest-dev/pytest/blob/9.1.1/src/_pytest/doctest.py#L178-L181)) -and is unreachable by name, which is why the `OutcomeException` re-raise, -`BdbQuit` → `outcomes.exit` and `continue_on_failure` handling must be -reimplemented rather than inherited. +and is unreachable by name. The carrier runner never executes; the adapter maps +the core's result records and host exception policy directly, while +`continue_on_failure` is read through the quarantined helper. ## Direction -Quarantine every private import in one module, `pytest_doctest_docutils._compat`, -with a pinned support matrix. +Quarantine every private import in one adapter-owned module with a pinned +support matrix. The clean-slate package spelling is +`pytest_doctest_docutils._compat`. The spike retains the released flat facade and +therefore uses the top-level `_pytest_doctest_compat`; that is an implementation +compromise, not the preferred namespace. **Filter the built-in's collector out of the `pytest_collect_file` result, in a hook wrapper.** The directory collector consumes the multicall result directly, @@ -82,11 +89,11 @@ under pytest 7 with a new enough pluggy. But pytest 7 declares only `pluggy>=0.12,<2.0`, so a resolver may legally install pluggy 1.0 or 1.1, where `wrapper=True` raises `TypeError` *while importing the plugin* — a session-wide abort, which this record and {doc}`0001-typed-vanilla-doctest-core` both forbid. -Old style needs no floor at all and was verified working on both pytest 7 and 9. +Old style adds no pluggy floor and was verified from pytest 7.2 through 9. -Whichever spelling is used, **name the minimum supported pytest**. The CI matrix -floor is 7 and the package declares no pytest dependency, so today the support -statement exists only in the workflow file. +Whichever spelling is used, **name the minimum supported pytest**. The package +declares `pytest>=7.2`, the first release exporting `pytest.DoctestItem`, and the +CI matrix pins that exact floor. **Fail on an unsupported pytest only when an affected document is collected**, not at plugin registration. A `pytest11` plugin that raises at import takes down @@ -96,23 +103,38 @@ sessions whose majority of tests never touch a doctest, and failures for the same reason. The error names the pytest version and the missing symbol, and it names the document that triggered it. -CI carries a job pinned to the minimum supported pytest and one tracking its -prerelease. +The acceptance matrix must carry a job pinned to the minimum supported pytest +and one tracking its prerelease. The spike implements the floor job; the +prerelease probe remains open. Registry construction is not a pytest-private-API concern. The host-neutral -contract, pytest hookspec, Sphinx adapter and xdist manifest are specified in +contract and host lifecycle proposals are specified in {doc}`0007-host-plugin-registration-lifecycle`. +## Spike result + +The old-style collection wrapper works across pytest 7.2, 8.4 and 9.1 and removes only +the built-in collector for documents this plugin claims. The built-in plugin is +required: collecting an affected documentation file, or a Python module through +this adapter's doctest-module mode, raises an actionable usage error when it has +been disabled. Fixture injection and pytest's checker/report options continue to +come from the built-in plugin. The adapter limits its claim to suffixes in the +frozen document-parser registry, so a separate `--doctest-glob=*.foo` remains +owned by pytest's text collector unless a contributor actually registers a +`.foo` parser. + +The quarantine is effective as an import boundary, but its symbol binding is +still eager. A supported or newer pytest release missing one of those private +names would fail while the plugin imports rather than when an affected document +is collected. Pytest below the declared 7.2 floor may likewise fail at the public +base-class import. The CI matrix covers released pytest 7.2, 8.4 and 9.1; it does +not yet include a prerelease probe. Those are remaining acceptance gaps, so this +record stays `Draft`. + ## Open -- Whether requiring the built-in plugin should be stated as a hard dependency. - `-p no:doctest` already fails today with a raw `ValueError` about an unknown - option, so this is not a regression — but the message should become actionable. - Whether the probe should accept a *newer* pytest it has not been tested against, or refuse it. Refusing is safer and more annoying; for a private-API quarantine with a small matrix, safer probably wins. -- What the filtering wrapper should do when the built-in's collector is the *only* - one for a path — that is the ordinary `--doctest-glob` case this plugin has no - business touching, so the filter must be scoped to paths it actually claims. - Whether any of these helpers can be promoted upstream, which would delete the quarantine entirely. diff --git a/docs/adrs/0007-host-plugin-registration-lifecycle.md b/docs/adrs/0007-host-plugin-registration-lifecycle.md index be3b515..122f02e 100644 --- a/docs/adrs/0007-host-plugin-registration-lifecycle.md +++ b/docs/adrs/0007-host-plugin-registration-lifecycle.md @@ -34,7 +34,8 @@ class Provider(t.NamedTuple): version: str | None -class Registration(t.NamedTuple, t.Generic[T]): +@dataclasses.dataclass(frozen=True, slots=True) +class Registration(t.Generic[T]): name: str value: T provider: Provider @@ -83,6 +84,11 @@ bound to that contributor's `Provider`; registrations cannot claim a different origin. A contributor retaining that registrar cannot retain a mutation path: every method raises `RegistryClosedError` after the snapshot is made. +`Registration` is a frozen, slotted dataclass rather than a generic +`NamedTuple`. The latter declaration fails while importing on Python 3.10, which +is inside the package's support range; immutability is the contract, not the +tuple representation. + ### Names, collisions and order Registration names are case-sensitive ASCII identifiers matching @@ -102,6 +108,11 @@ accepted only when the challenger uses the incumbent parser's name and passes `replace=True`; two differently named parsers cannot both win `.md` by incidental plugin load order. +Freeze also validates cross-references. Every block kind must name an existing +execution profile; a non-`None` expected-output kind must follow the registry +name grammar and cannot also be a runnable block kind. Errors identify the block +kind and provider before parsing begins. + ## Host adapters ### Direct API @@ -134,9 +145,24 @@ Nested conftests load during collection and are outside this lifecycle. A and raises `pytest.UsageError` naming that plugin and the closed registration phase. Fixtures and unrelated hooks in nested conftests remain valid. -### Sphinx +### Spike boundary + +The direct and pytest paths above are implemented. The pytest snapshot is frozen +once, custom checker contribution is exercised end to end with one +comparison-and-rendering instance, a custom block can pair with a custom output +stamp, and a late nested conftest contributor fails with an actionable usage +error. Low-level parse, extract, project, and run functions retain a convenience +`registry=None` default; registry identity across stages is guaranteed only when +a caller passes the same snapshot, as both host adapters do. + +The Sphinx contributor lifecycle and xdist manifest below were not needed to +test the core boundary and are deferred until an external contributor requires +them. The spike proves Sphinx-resolved doctree extraction and homogeneous xdist +execution, not these two bootstrap protocols. + +### Proposed Sphinx lifecycle -The Sphinx adapter exposes +The proposed Sphinx adapter exposes `add_doctest_core_contributor(app, contributor)`. Extensions call it from their `setup(app)` function. At `config-inited`, after extension setup and before any document is read, the adapter emits a `doctest-core-contributors` event, appends @@ -152,24 +178,24 @@ extension name. This lifecycle makes the extractor usable on Sphinx-resolved doctrees. It does not add a builder or claim parity with `sphinx-build -b doctest` execution. -## xdist consistency +## Proposed xdist consistency -The controller sends a JSON-safe manifest through `workerinput` from +The controller would send a JSON-safe manifest through `workerinput` from `pytest_configure_node`. Each worker builds its own snapshot during `pytest_configure` and compares before collection. The manifest has a schema version and contains: -- a JSON-safe projection of normalized `SessionSettings` +- JSON-safe projections of normalized parse, projection, and run settings - every registry category, name, provider and provider version in declared order - `doctest.OPTIONFLAGS_BY_NAME`, sorted by flag name -A mismatch aborts the session with the controller and worker manifests. This is +A mismatch would abort the session with the controller and worker manifests. This is an extension-set consistency check, not proof that two workers are semantically identical. Equal provider names and versions do not prove equal source code, and the manifest does not hash included documents, directive implementations or MyST plugins. -Version 1 therefore supports homogeneous worker environments. Equal source +The initial contract therefore supports homogeneous worker environments. Equal source closure and equal installed provider code are preconditions, while xdist's own identical-collection check remains authoritative for node ids. Stronger support for deliberately heterogeneous SSH or socket workers would require content or @@ -180,11 +206,13 @@ environment attestation and is deferred. - Core extension authors implement one `Contributor` regardless of host. - Settings remain serializable inputs; discovered objects remain in the registry. - Parse and execution code cannot mutate capabilities after collection starts. -- pytest and Sphinx own timing and diagnostics in their native idioms without - leaking their lifecycle types into the core. +- pytest owns registration timing in its native idioms without leaking lifecycle + types into the core. Sphinx can adopt the same contract when its lifecycle is + implemented. - Replacement is possible but visible, attributed and deterministic. -- Supporting heterogeneous xdist workers is explicitly outside the first - contract rather than implied by a weak manifest. +- Supporting heterogeneous xdist workers remains outside the first contract; + the proposed manifest would diagnose capability mismatches without pretending + to attest worker code or source closures. ## Open diff --git a/notes/analyses/24-implementation-bakeoff.md b/notes/analyses/24-implementation-bakeoff.md new file mode 100644 index 0000000..451fd5e --- /dev/null +++ b/notes/analyses/24-implementation-bakeoff.md @@ -0,0 +1,178 @@ +# Typed doctest core implementation bakeoff + +## Question + +Can the architecture in ADR 0001 be implemented as a typed, host-neutral core +while retaining stock doctest objects and composing with pytest, docutils, MyST, +Sphinx-resolved doctrees, reruns, and xdist? + +## Candidates + +### Extend the existing modules + +Keeping extraction, grouping, execution, pytest collection, and reporting in the +two existing modules minimizes import changes. It also preserves the current +coupling: projection cannot be tested without docutils, pytest policy leaks into +execution, and mutable `DocTest` instances are likely to survive across reruns. +This shape was rejected. + +### Typed core with compatibility adapters + +The successful shape is a new `doctest_core` package with thin direct and pytest +adapters: + +```text +contributors -> frozen registry + | +text/doctree -> extraction -> projection -> group runner + | | | + inert records recipes fresh DocTests + | + direct / pytest adapters +``` + +Extraction returns inert typed records and diagnostics. Projection is pure and +owns grouping, wildcard expansion, phase ordering, pairing, and names. The group +runner materializes fresh stock `doctest.Example` and `doctest.DocTest` objects +for every attempt and keeps the shared mapping inside one scheduled item. Host +adapters own collection, fixtures, exception policy, and presentation. + +This shape was selected. It preserves the one invariant that mattered under +reruns and xdist: the unit sharing mutable globals is also the unit the host +schedules. + +### Replace doctest semantics wholesale + +Owning parsing, examples, comparison, and reporting would make every extension +easy to express, but would discard the compatibility goal. It would also require +reimplementing pytest's checker extensions and CPython's process-state behavior. +The bakeoff found no benefit that justified that compatibility surface. + +## What implementation changed in the ADRs + +The ordinary prompt lane should delegate to CPython's untouched runner. An +extended `exec` lane should be a separate, bounded runtime. Rebinding +`doctest.compile`, cloning CPython's code object, or overriding its private loop +all attach extended syntax to global or private behavior that the core does not +otherwise need. ADR 0002 now records the two-lane contract. + +Practice also required contracts absent from the original data model: + +- `ExceptionPolicy` lets a host distinguish ordinary exceptions, host outcomes, + and aborts that must outrank prior failures without importing pytest into the + core. +- `Failed` retains the exact checker that compared output so a contributed + checker also explains its own failure. +- `Registration` is a frozen generic dataclass. A generic `NamedTuple` fails at + import on Python 3.10. +- A proposed `BlockAttributes` `TypedDict` was rejected as false precision over + third-party node stamps. Field-level validation narrows into `ParsedBlock` and + `ParsedOutput`, which are the first owned schema. +- Gates execute inside the group failure boundary, and cleanup runs after setup, + test, and gate failures. +- Extended compilation uses `dont_inherit=True` and only future flags explicitly + present in the live group mapping. +- An inline doctest `FAIL_FAST` flag stops the current runtime's example loop; + a runner-level flag also stops later group blocks despite the host's continue + policy. +- The core defaults unlabelled blocks to Sphinx's `default` group, while the + pytest adapter preserves gp-libs' released per-block isolation default. +- The core's failure-continuation default follows doctest and Sphinx; direct and + pytest hosts override it only for explicit fail-fast or debugger policy. +- The pytest adapter composes with the built-in doctest plugin and filters only + its duplicate documentation collector. It no longer unregisters the plugin + whose fixture, checker, options, and rendering it uses. +- The distribution declares its actual pytest 7.2 floor and direct `packaging` + dependency, and uses `pytest_doctest_docutils` as the pytest entry-point name + so the standard `-p no:pytest_doctest_docutils` spelling works. +- Sphinx compatibility is extractor compatibility over resolved doctrees, not + byte-identical directive stamps. +- Expected-output records retain their stamp name. A custom `pairs_with` + relationship therefore works through extraction and projection rather than + being nominal registry metadata. +- Freeze validates profile and expected-output references before parsing, and + anonymous group identities cannot collide with an author-written `block-N` + group. +- Prompt-free `doctest` directives project no group and produce no passing + carrier item. Collector filtering is limited to registered parser suffixes, + leaving unrelated `--doctest-glob` paths to pytest. + +## Evidence + +| Boundary | Result | +|---|---| +| Full repository suite on Python 3.14 and pytest 9 | 227 passed | +| Full repository suite on Python 3.12 and pytest 8.4 | 227 passed | +| Python 3.10, docutils 0.20.1, and pytest 7.2 floor suite | 224 passed, 3 skipped | +| Rerun isolation | a failed first attempt cannot pass from retained globals | +| xdist | stateful groups pass under `load` and `worksteal` without affinity | +| Sphinx | resolved doctrees retain hidden setup/cleanup nodes and include attribution | +| Extension seam | a contributed checker compares and renders with the same instance | +| pytest-asyncio | a 1.x async autouse fixture populates the doctest namespace | +| Packaging | the core package and `py.typed` are present in wheel-from-sdist validation | + +The xdist result proves the item boundary under two schedulers. It does not prove +heterogeneous workers or every distribution mode. The Sphinx result proves +extraction from its resolved tree, not a Sphinx execution lifecycle. + +## ADR shortcomings exposed by the bakeoff + +The architecture is usable, but these claims remain incomplete: + +- The diagnostics core captures, deduplicates, and suppresses known noise, but + the pytest adapter does not yet fail unsuppressed errors or render warnings. + Message-substring classification is provisional because docutils supplies no + stable diagnostic codes. The direct facade also drops the channel, so a + malformed option's Sphinx-compatible warning is not yet user-visible. +- Partial block skips remain worker-local in `GroupResult`; there is no versioned + JSON-safe pytest report projection or controller-side terminal summary. +- Sphinx contributor timing and the xdist registry/settings manifest are designs, + not implemented lifecycle contracts. +- The extended runtime matrix still lacks report-only-first, repeated-call, and + interactive debugger coverage. Expected-exception output, `SyntaxError`, + `IGNORE_EXCEPTION_DETAIL`, and inline fail-fast are covered. +- The core avoids CPython's private runner loop but still uses the private + `DocTestParser._EXAMPLE_RE` and `_EXCEPTION_RE` contracts. Their behavior is + exercised indirectly, not yet guarded by focused compatibility probes. +- The Python 3.10/Sphinx 8 stack constrains docutils to its pre-0.22 line + convention. The package now states `<0.22`; supporting docutils 0.22 requires + the coordinated Python/Sphinx policy change in ADR 0005. +- Standalone MyST root-line recovery cannot prove exact locations inside + included Markdown files. It refuses to stamp a root line onto an included + source and retains the parser's ambiguous fallback. +- Profile context-manager entry and exit failures do not yet have phase-aware + result semantics, and the initial runtime contract has no separate + profile-decline outcome. +- The direct facade cannot reproduce doctest's complete verbose + `Trying`/`Expecting`/`ok` stream because successful per-example events are not + retained. Failure and summary output remain stock-shaped. A cleanup error that + follows an ordinary doctest mismatch is also retained only in the core result; + the direct facade has no secondary-outcome rendering channel yet. +- The pytest private-API quarantine binds its symbols eagerly and has no + prerelease CI probe, so an unsupported pytest can still fail at plugin import. +- The pytest 7.2 floor also requires an older pytest-asyncio test dependency; + the matrix must pin those versions together rather than installing each + plugin's newest release independently. +- The legacy adapter is still a flat module, which leaves its private quarantine + as the top-level `_pytest_doctest_compat` module. A packaged adapter namespace + would contain that private surface more cleanly. +- A registered block kind and its custom expected-output stamp are preserved + through projection, but registration alone does not teach reST or MyST a new + directive. +- Orphan, misplaced, and duplicate `testoutput` records still need explicit + diagnostics. Pairing itself is group-local and duplicate output follows + Sphinx's last-one-wins rule. `testcode :pyversion:` is deliberately enforced + rather than silently ignored as Sphinx does. +- Async pytest fixtures compose with documentation items under pytest-asyncio + 1.x. Its pytest-7-compatible 0.21 line does not await an async autouse fixture + for this item shape. Async block execution is only represented by the + execution-profile seam; no async profile was implemented in this spike. +- Document front-matter settings were premature and are deferred. + +## Conclusion + +Keep the typed core and thin adapters. Do not return to the monolith and do not +base extended execution on CPython's private loop. The ADR's central +one-item-per-shared-group decision survived implementation; its overclaims were +mostly in conformance, diagnostics, host bootstrap, and reporting rather than in +the core boundary itself. diff --git a/notes/analyses/README.md b/notes/analyses/README.md index 72b5187..f6a679d 100644 --- a/notes/analyses/README.md +++ b/notes/analyses/README.md @@ -52,7 +52,8 @@ What exactly does each of them require, and where do they contradict each other? - Cross-cutting: [`20-data-structures.md`](20-data-structures.md), [`21-data-flows.md`](21-data-flows.md), [`22-extension-seams.md`](22-extension-seams.md), - [`23-namespace-scope-and-test-identity.md`](23-namespace-scope-and-test-identity.md). + [`23-namespace-scope-and-test-identity.md`](23-namespace-scope-and-test-identity.md), + and [`24-implementation-bakeoff.md`](24-implementation-bakeoff.md). - [`90-bibliography.md`](90-bibliography.md) — every pinned anchor cited by the ADRs, in one place. From 36f1a1fc9a5af6bb25ae11a7dab3558dc1fddbcb Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sat, 8 Aug 2026 09:39:39 -0500 Subject: [PATCH 30/31] docs(doctest): Describe grouped adapters why: Public guidance must distinguish the direct command from pytest's one-item-per-group behavior without promising full stdlib transcripts. what: - Describe direct doctest command and summary behavior - Explain built-in pytest doctest plugin composition - Clarify bare-block isolation and the Sphinx default directive group - State the supported docutils range --- README.md | 23 ++++++++++--------- docs/modules/doctest_docutils/how-to.md | 4 ++-- docs/modules/doctest_docutils/index.md | 7 +++--- .../modules/pytest_doctest_docutils/how-to.md | 16 ++++++------- docs/modules/pytest_doctest_docutils/index.md | 6 +++-- .../pytest_doctest_docutils/tutorial.md | 8 ++++--- 6 files changed, 34 insertions(+), 30 deletions(-) diff --git a/README.md b/README.md index 850dfc9..b8d7c26 100644 --- a/README.md +++ b/README.md @@ -12,9 +12,10 @@ git-pull projects, e.g. [cihai], [vcs-python], or [tmux-python]. Two components: -1. `doctest_docutils` module: Same specification as `doctest`, but can parse reStructuredText - and markdown -2. `pytest_doctest_docutils`: Pytest plugin, collects test items for pytest for reStructuredText and markdown files +1. `doctest_docutils`: a doctest-shaped direct API and CLI for reStructuredText + and Markdown +2. `pytest_doctest_docutils`: a pytest plugin that collects shared-state groups + from reStructuredText and Markdown files This means you can do: @@ -24,8 +25,8 @@ Two components: ### doctest module -This extends standard library `doctest` to support anything docutils can parse. -It can parse reStructuredText (.rst) and markdown (.md). +This uses standard-library `doctest` prompt and comparison conventions while +parsing reStructuredText (`.rst`) and Markdown (`.md`). See more: @@ -64,7 +65,7 @@ It supports two barebones directives: #### Usage -The `doctest_docutils` module preserves standard library's usage conventions: +The `doctest_docutils` module preserves the standard library's command shape: ##### reStructuredText @@ -84,10 +85,10 @@ $ python -m doctest_docutils README.md -v ### pytest plugin -_This plugin disables [pytest's standard `doctest` plugin]._ - -This plugin integrates `doctest_docutils` with pytest so documentation examples -run with the surrounding `conftest.py` setup. +This plugin runs documentation examples as pytest items. It composes with +[pytest's standard `doctest` plugin]: gp-libs owns matching documentation files, +while pytest continues to supply fixtures, checker and report options, and +Python-module doctest collection. ```console $ pytest docs/ @@ -154,7 +155,7 @@ You can test the unpublished version of g before its released. To lift the development burden of supporting legacy APIs, as this package is lightly used, minimum constraints have been pinned: -- docutils: 0.20.1+ +- docutils: >=0.20.1,<0.22 - myst-parser: 2.0.0+ If you have even passing interested in supporting legacy versions, file an diff --git a/docs/modules/doctest_docutils/how-to.md b/docs/modules/doctest_docutils/how-to.md index da5be6f..fbbbfbf 100644 --- a/docs/modules/doctest_docutils/how-to.md +++ b/docs/modules/doctest_docutils/how-to.md @@ -18,9 +18,9 @@ Use the same command for `.rst` files: $ python -m doctest_docutils README.rst ``` -## See collected examples +## See the run summary -Pass `-v` for verbose standard-library doctest output: +Pass `-v` to list each tested group in the final summary: ```console $ python -m doctest_docutils README.md -v diff --git a/docs/modules/doctest_docutils/index.md b/docs/modules/doctest_docutils/index.md index e63866e..70d8f48 100644 --- a/docs/modules/doctest_docutils/index.md +++ b/docs/modules/doctest_docutils/index.md @@ -23,7 +23,7 @@ Run your first documentation doctest from a Markdown page. :::{grid-item-card} How-to :link: how-to :link-type: doc -Choose files, run verbose output, and map the command to stdlib doctest. +Choose files, inspect run summaries, and map the command to stdlib doctest. ::: :::{grid-item-card} Examples @@ -35,7 +35,7 @@ See the supported Markdown and reStructuredText example shapes. :::{grid-item-card} API Reference :link: reference :link-type: doc -Inspect finder, runner, directive, and CLI APIs. +Inspect finder, directive, and CLI APIs. ::: :::: @@ -48,8 +48,7 @@ Run a Markdown page: $ python -m doctest_docutils README.md ``` -No output means the examples passed. Add `-v` when you want the standard -doctest transcript. +No output means the examples passed. Add `-v` for a final group summary. ```{toctree} :hidden: diff --git a/docs/modules/pytest_doctest_docutils/how-to.md b/docs/modules/pytest_doctest_docutils/how-to.md index aaeef3c..6e2fe75 100644 --- a/docs/modules/pytest_doctest_docutils/how-to.md +++ b/docs/modules/pytest_doctest_docutils/how-to.md @@ -53,13 +53,13 @@ parses in `.rst`, `.md`, and Python-module doctests. The standalone `python -m doctest_docutils` command does not register it, so use the marker when you run examples through pytest. -## Keep pytest's built-in doctest plugin disabled +## Keep pytest's built-in doctest plugin enabled -The gp-libs plugin blocks pytest's built-in doctest plugin by default. Keep -`-p no:doctest` in local examples when you are demonstrating explicit pytest -configuration: +This plugin composes with pytest's built-in doctest plugin. The built-in plugin +supplies `doctest_namespace`, output-checker extensions, report options, and +Python-module collection, while gp-libs owns the documentation-file collector. -```ini -[pytest] -addopts = -p no:doctest -``` +Do not pass `-p no:doctest` when collecting documentation through gp-libs. If +the built-in plugin is disabled, collecting an affected documentation file or +using `--doctest-docutils-modules` raises a usage error instead of silently +collecting no tests. diff --git a/docs/modules/pytest_doctest_docutils/index.md b/docs/modules/pytest_doctest_docutils/index.md index 309c68d..53705dd 100644 --- a/docs/modules/pytest_doctest_docutils/index.md +++ b/docs/modules/pytest_doctest_docutils/index.md @@ -8,8 +8,10 @@ parses each page through {ref}`doctest_docutils` before pytest runs the examples. -The plugin blocks {ref}`pytest's standard doctest plugin ` by -default so the same examples are not collected twice. +The plugin composes with {ref}`pytest's standard doctest plugin `. +gp-libs owns matching documentation paths and filters pytest's duplicate text +collector before it parses the page; pytest continues to supply fixtures, +checker and report options, and Python-module doctest collection. ::::{grid} 1 1 2 2 :gutter: 2 2 3 3 diff --git a/docs/modules/pytest_doctest_docutils/tutorial.md b/docs/modules/pytest_doctest_docutils/tutorial.md index 3018fcc..8e6579c 100644 --- a/docs/modules/pytest_doctest_docutils/tutorial.md +++ b/docs/modules/pytest_doctest_docutils/tutorial.md @@ -16,8 +16,10 @@ $ pytest docs/ ``` {mod}`pytest_doctest_docutils` parses each matching documentation file with -{mod}`doctest_docutils`, then reports each collected doctest as a pytest item. -That gives documentation examples the same pass/fail surface as the rest of -your suite. +{mod}`doctest_docutils`, then reports each projected shared-state group as one +pytest item. Bare prompt blocks without a group stamp are isolated by default; +unargumented `doctest` directives join Sphinx's `default` group. Named blocks in +the same group share fixtures and Python globals, and execute together as one +schedulable item. [pytest]: https://docs.pytest.org/en/stable/ From d362ae35d650da80a17303cdaaec65b841277e7f Mon Sep 17 00:00:00 2001 From: Tony Narlock Date: Sat, 15 Aug 2026 19:34:41 -0500 Subject: [PATCH 31/31] ci(test[matrix]): Restore the docutils ceiling why: The rebase onto master dropped this branch's `docutils<0.22` cap, on the reasoning that master tests 0.22.4 successfully. That was wrong. Sphinx declares `docutils>=0.20,<0.22`, so 0.22.4 leaves the environment inconsistent, and this branch's `uv pip check` step fails on it. Master has no such step, which is why it never noticed. what: - Restore `docutils>=0.20.1,<0.22` - Return the matrix to docutils 0.20.1 and 0.21.2 - Drop the Python 3.10 exclusion, which only existed to keep 3.10 away from docutils 0.22 and cannot trigger under the ceiling Verified `uv pip check` clean and 227 tests passing on docutils 0.20.1 and 0.21.2 against pytest 8.4.2 and 9.1.1. --- .github/workflows/tests.yml | 9 +-------- pyproject.toml | 2 +- uv.lock | 2 +- 3 files changed, 3 insertions(+), 10 deletions(-) diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 34c6661..97e2f19 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -12,7 +12,7 @@ jobs: strategy: matrix: python-version: ['3.10', '3.11', '3.12', '3.13', '3.14'] - docutils-version: ['0.20.1', '0.22.4'] + docutils-version: ['0.20.1', '0.21.2'] pytest-version: ['8.4.2', '9.1.1'] include: - pytest-version: '8.4.2' @@ -21,13 +21,6 @@ jobs: - pytest-version: '9.1.1' pytest-asyncio-version: '1.4.0' pytest-rerunfailures-version: '16.4' - exclude: - # docutils 0.22 removed the bundled docutils.utils.roman. Sphinx 8.1 - # imports it, so its latex builder fails to load. Sphinx 8.2 dropped - # that import but requires Python 3.11, and 8.1 is the newest release - # supporting 3.10, leaving no working pair on that interpreter. - - python-version: '3.10' - docutils-version: '0.22.4' steps: - uses: actions/checkout@v7 diff --git a/pyproject.toml b/pyproject.toml index d17c496..4f31cf8 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -35,7 +35,7 @@ readme = 'README.md' keywords = [] homepage = "https://gp-libs.git-pull.com" dependencies = [ - "docutils>=0.20.1", + "docutils>=0.20.1,<0.22", "myst-parser>=2.0.0", "packaging", "pytest>=8.3.3", diff --git a/uv.lock b/uv.lock index 47ef45a..39192a6 100644 --- a/uv.lock +++ b/uv.lock @@ -480,7 +480,7 @@ testing = [ [package.metadata] requires-dist = [ - { name = "docutils", specifier = ">=0.20.1" }, + { name = "docutils", specifier = ">=0.20.1,<0.22" }, { name = "myst-parser", specifier = ">=2.0.0" }, { name = "packaging" }, { name = "pytest", specifier = ">=8.3.3" },