How to write negative typing tests?

[dimaqq]

Let's say I'm writing a pypi package and providing types for it.

I would like to ensure that my users catch certain classes of typing errors with pyright, for example, here's my best manual attempt to validate that type hints for main both allow a good argument and reject a bad argument:

  GOOD_ARGUMENT = ...
  BAD_ARGUMENT = 42  # no ints, please!


  def foo():
      from apackage import main
      main(GOOD_ARGUMENT)
E     main(BAD_ARGUMENT) # E: Argument of type "..." cannot be assigned to parameter "first_arg" ...
E     main() # E: Argument missing for parameter "first_arg"


  def bar():
      from apackage import Client
      main = Client().main
      main(GOOD_ARGUMENT)
E     main(BAD_ARGUMENT) # E: Argument of type "..." cannot be assigned to parameter "first_arg" ...
E     main() # E: Argument missing for parameter "first_arg"

I suppose positive tests are trivial: dump the code into a file somewhere under ./tests/. If there's a regression, pyright will catch this in CI.

What about negative tests though? Is there a best practice to follow?

Or, perhaps, is there a way to ask pyright a question like this:

• given my library
• given its use in hypothetical user code (e.g. a simple import)
• tell me the exact type you've computed

The key here being that some are direct type hints, but others, like typeof[ Client().main ] are computed or inferred.

[erictraut]

The common way to write negative typing tests is to use a combination of a # type: ignore or # pyright: ignore along with the reportUnnecessaryTypeIgnoreComment check in pyright. The former will suppress the typing error, and the latter will trigger a diagnostic if the # pyright: ignore is no longer serving a purpose.

The assert_type function is also useful for typing tests.

Code sample in pyright playground

# pyright: reportUnnecessaryTypeIgnoreComment=true

from typing import assert_type


def func(x: int) -> int:
    return x


v1 = func(3)
assert_type(v1, int)

func("bad param") # pyright: ignore[reportArgumentType]

func(1) # pyright: ignore[reportArgumentType]