Skip to content

Classify plugin startup import failures (#993) - #14824

Open
RonnyPfannschmidt wants to merge 1 commit into
pytest-dev:mainfrom
RonnyPfannschmidt:plugin-import-error-exit-codes-993
Open

Classify plugin startup import failures (#993)#14824
RonnyPfannschmidt wants to merge 1 commit into
pytest-dev:mainfrom
RonnyPfannschmidt:plugin-import-error-exit-codes-993

Conversation

@RonnyPfannschmidt

@RonnyPfannschmidt RonnyPfannschmidt commented Aug 1, 2026

Copy link
Copy Markdown
Member

Closes #993.

On main a plugin failing to import at startup escapes _main as a plain ImportError: raw traceback, exit 1 — colliding with EXIT_TESTSFAILED by accident — and pytest.main() raises instead of returning its documented exit code. A bare argless ImportError additionally crashes pytest's own e.args[0] handling with IndexError.

#7290 stalled on whether a plugin failing under a conftest's pytest_plugins is a conftest problem or a plugin problem. This PR splits on what actually went wrong instead of on which mechanism asked:

  • plugin not found (-p, pytest_plugins, PYTEST_PLUGINS) → misuse → USAGE_ERROR (4), same as a failing conftest.py import
  • found but raises while importing — including a missing transitive dependency and a broken pytest11 entry point → plugin defect → INTERNAL_ERROR (3), traceback preserved in the report and on the exception chain (raise ... from) (losing it was the other objection to Issue 993 - improve handling exceptions in conftest #7290)

The split keys on ModuleNotFoundError.name matching the requested spec or a parent package, so a plugin whose own import foo fails is not blamed on the user. conftest.py import failures stay at 4, unchanged.

before after
missing plugin via -p / pytest_plugins / PYTEST_PLUGINS 1 + raw traceback 4
plugin raises at import (incl. missing dependency, broken entry point) 1 + raw traceback 3 + formatted traceback
conftest.py fails to import 4 4
pytest.main() in-process raised ImportError returns the exit code

Draft because the classification is a judgement call and exit codes are observable API:

  1. Is INTERNAL_ERROR (3) right for a broken third-party plugin, or does this warrant a new exit code? wrap_session already uses 3 for plugin exceptions mid-session.
  2. Should conftest.py follow the same split? Left alone here to keep the blast radius small.
  3. The load_setuptools_entrypoints override widens scope — but without it a broken installed plugin still exits 1 with a raw traceback.
  4. Changelog: bugfix or breaking? Filed as breaking per review.

Tests: new TestStartupPluginImportErrors covers every entry path plus both regressions; six existing tests updated from ImportError to UsageError/PluginImportFailure. Full suite and pre-commit (incl. mypy) pass locally.

🤖 Written by Claude Code (Claude Fable 5); reviewed and submitted by @RonnyPfannschmidt.

@psf-chronographer psf-chronographer Bot added the bot:chronographer:provided (automation) changelog entry is part of PR label Aug 1, 2026
@RonnyPfannschmidt
RonnyPfannschmidt force-pushed the plugin-import-error-exit-codes-993 branch 5 times, most recently from 434b9ab to 363aea6 Compare August 26, 2026 22:12
@RonnyPfannschmidt
RonnyPfannschmidt force-pushed the plugin-import-error-exit-codes-993 branch from 363aea6 to baff0df Compare August 27, 2026 07:02
@RonnyPfannschmidt
RonnyPfannschmidt marked this pull request as ready for review August 27, 2026 07:20
@RonnyPfannschmidt
RonnyPfannschmidt force-pushed the plugin-import-error-exit-codes-993 branch from baff0df to 875f10f Compare August 27, 2026 07:21
Comment thread changelog/993.breaking.rst
Comment thread src/_pytest/config/__init__.py Outdated
A plugin failing to load during startup escaped `_main` as an unhandled
exception: Python printed a raw traceback and exited 1, which is
indistinguishable from EXIT_TESTSFAILED. Meanwhile a conftest.py failing
to import already returned EXIT_USAGEERROR, which is the inconsistency
pytest-dev#993 was filed about.

Split the failure into the two things it can actually mean:

- the plugin cannot be found at all -- pytest was pointed at something
  which is not there, so this is a usage error (exit 4), matching what
  conftest.py import failures already do.

- the plugin was found but raised while importing -- including a missing
  transitive dependency and a broken pytest11 entry point -- which is a
  defect in the plugin rather than a misuse of pytest, so it is reported
  as an internal error (exit 3).

The plugin traceback is preserved in both the report and the exception
chain (PluginImportFailure is always raised `from` the original error);
losing it was the main objection to the earlier attempt in pytest-dev#7290.

Side effects:

- pytest.main() now returns these exit codes instead of propagating the
  exception to its caller, matching its documented contract.
- a bare `raise ImportError` in a plugin no longer crashes pytest's own
  internals with `IndexError: tuple index out of range` from `e.args[0]`.

conftest.py import failures are deliberately left alone and keep
returning exit 4.

Co-Authored-By: Claude Opus 5 <ai@anthropic.com>
Co-Authored-By: Claude Fable 5 <ai@anthropic.com>
Co-Authored-By: Claude Code <ai@anthropic.com>
@RonnyPfannschmidt
RonnyPfannschmidt force-pushed the plugin-import-error-exit-codes-993 branch from 875f10f to 98fcf26 Compare August 27, 2026 19:57
@RonnyPfannschmidt RonnyPfannschmidt added this to the 10.0 milestone Aug 28, 2026
@RonnyPfannschmidt

Copy link
Copy Markdown
Member Author

might make sense to wait 1-2 majro releases before putting this in as is

@nicoddemus

Copy link
Copy Markdown
Member

Why wait? Going into the next major release should be fine?

@RonnyPfannschmidt

Copy link
Copy Markdown
Member Author

Sorry I meant feature release

I recall we have some new deprecation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bot:chronographer:provided (automation) changelog entry is part of PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ImportError raised by plugins make pytest return EXIT_TESTSFAILED

2 participants