feat(schematic): evaluate Pyright for type checking in schematic-api

[tschaffter]

Description

This PR sets up a comparison of type checking in sandbox-api using both mypy and Pyright. The best of the two will be used in example Python projects like sandbox-py-app and sandbox-py-lib.

A couple of notes about both linters:

• Mypy is currently used to lint Python files in the schematic-api project.
• Mypy is a Python package that needs to be installed in each Python project where you want to use it.
• Pyright is available as a Python package and npm package. It is possible to install Pyright only once using the npm package.
• Pyright is designed to power the Python language server, Pylance, which is available as a VS Code extension. This means that Python files opened in VS Code are automatically linted using Pyright, and any errors are immediately visible to the user.
  • A similar behavior can be achieved for mypy by installing its VS Code extension.

Changelog

• Add the task schematic-api:type-check

References

• https://marketplace.visualstudio.com/items?itemName=ms-pyright.pyright

Notes

About Pyright VS Code extension:

For most VS Code users, we recommend using the Pylance extension rather than Pyright. Pylance incorporates the Pyright type checker but features additional capabilities such as semantic token highlighting and symbol indexing. When Pylance is installed, the Pyright extension will disable itself.

Preview

Note

Here we are type checking the entire schematic_api folder instead of the subset of files checked by the task schematic-api:mypy.

Type checking with mypy

Output:

$ cd apps/schematic/api
$ poetry run mypy --non-interactive --install-types --disallow-untyped-defs schematic_api
schematic_api/typing_utils.py:22: error: Function is missing a type annotation  [no-untyped-def]
schematic_api/typing_utils.py:26: error: Function is missing a type annotation  [no-untyped-def]
schematic_api/typing_utils.py:30: error: Function is missing a type annotation  [no-untyped-def]
schematic_api/controllers/security_controller_.py:4: error: Function is missing a type annotation  [no-untyped-def]
schematic_api/util.py:8: error: Function is missing a type annotation  [no-untyped-def]
schematic_api/util.py:36: error: Function is missing a type annotation  [no-untyped-def]
schematic_api/util.py:54: error: Function is missing a type annotation  [no-untyped-def]
...
schematic_api/controllers/manifest_generation_controller.py:13: error: Function is missing a type annotation  [no-untyped-def]
schematic_api/controllers/manifest_generation_controller.py:54: error: Function is missing a type annotation  [no-untyped-def]
schematic_api/test/conftest.py:17: error: Incompatible return value type (got "bytes", expected "str")  [return-value]
Found 314 errors in 50 files (checked 75 source files)

In strict mode:

$ poetry run mypy --non-interactive --install-types --strict schematic_api
...
schematic_api/test/test_storage_controller_endpoints2.py:275: error: Call to untyped function "DatasetMetadata" in typed context  [no-untyped-call]
schematic_api/test/test_manifest_generation_endpoints.py:29: error: Unused "type: ignore" comment  [unused-ignore]
schematic_api/test/test_manifest_generation_endpoints.py:55: error: Unused "type: ignore" comment  [unused-ignore]
Found 483 errors in 58 files (checked 75 source files)

Runtime (non-strict mode; the strict mode features virtually the same runtime (data not shown)):

$ hyperfine --warmup 1 --runs 10 'poetry run mypy --non-interactive --install-types --disallow-untyped-defs schematic_api' --ignore-failure
Benchmark 1: poetry run mypy --non-interactive --install-types --disallow-untyped-defs schematic_api
 ⠸ Performing warmup runs         ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
  Time (mean ± σ):      9.135 s ±  0.422 s    [User: 7.626 s, System: 0.679 s]
  Range (min … max):    8.784 s … 10.204 s    10 runs

Type checking with Pyright

Output:

$ nx type-check schematic-api
/workspaces/sage-monorepo/apps/schematic/api/schematic_api/__main__.py
  /workspaces/sage-monorepo/apps/schematic/api/schematic_api/__main__.py:8:9 - error: Cannot assign to attribute "json_encoder" for class "None"
    Attribute "json_encoder" is unknown (reportAttributeAccessIssue)
/workspaces/sage-monorepo/apps/schematic/api/schematic_api/controllers/manifest_generation_controller.py
  /workspaces/sage-monorepo/apps/schematic/api/schematic_api/controllers/manifest_generation_controller.py:46:9 - error: Argument of type "Unknown | None" cannot be assigned to parameter "add_annotations" of type "bool" in function "generate_excel_manifest"
    Type "Unknown | None" is not assignable to type "bool"
      "None" is not assignable to "bool" (reportArgumentType)
  /workspaces/sage-monorepo/apps/schematic/api/schematic_api/controllers/manifest_generation_controller.py:48:9 - error: Argument of type "Unknown | None" cannot be assigned to parameter "display_label_type" of type "DisplayLabelType" in function "generate_excel_manifest"
    Type "Unknown | None" is not assignable to type "DisplayLabelType"
      Type "None" is not assignable to type "DisplayLabelType"
        "None" is not assignable to "Literal['class_label']"
        "None" is not assignable to "Literal['display_label']" (reportArgumentType)
/workspaces/sage-monorepo/apps/schematic/api/schematic_api/controllers/paging.py
  /workspaces/sage-monorepo/apps/schematic/api/schematic_api/controllers/paging.py:30:26 - error: Attribute type cannot use type variable "ITEM_TYPE@__init__" scoped to local method (reportGeneralTypeIssues)
/workspaces/sage-monorepo/apps/schematic/api/schematic_api/controllers/schema_controller_impl.py
  /workspaces/sage-monorepo/apps/schematic/api/schematic_api/controllers/schema_controller_impl.py:140:36 - error: Type "OutEdgeView[Unknown]" is not assignable to declared type "list[tuple[str, str]]"
    "OutEdgeView[Unknown]" is not assignable to "list[tuple[str, str]]" (reportAssignmentType)
/workspaces/sage-monorepo/apps/schematic/api/schematic_api/controllers/tangled_tree_controller_impl.py
  /workspaces/sage-monorepo/apps/schematic/api/schematic_api/controllers/tangled_tree_controller_impl.py:46:30 - error: Type "str | list[str]" is not assignable to declared type "list[str]"
    Type "str | list[str]" is not assignable to type "list[str]"
      "str" is not assignable to "list[str]" (reportAssignmentType)
/workspaces/sage-monorepo/apps/schematic/api/schematic_api/controllers/utils.py
  /workspaces/sage-monorepo/apps/schematic/api/schematic_api/controllers/utils.py:289:42 - warning: Unsupported escape sequence in string literal (reportInvalidStringEscapeSequence)
/workspaces/sage-monorepo/apps/schematic/api/schematic_api/test/__init__.py
  /workspaces/sage-monorepo/apps/schematic/api/schematic_api/test/__init__.py:13:17 - error: Cannot assign to attribute "json_encoder" for class "None"
    Attribute "json_encoder" is unknown (reportAttributeAccessIssue)
/workspaces/sage-monorepo/apps/schematic/api/schematic_api/test/conftest.py
  /workspaces/sage-monorepo/apps/schematic/api/schematic_api/test/conftest.py:17:12 - error: Type "bytes" is not assignable to return type "str"
    "bytes" is not assignable to "str" (reportReturnType)
8 errors, 1 warning, 0 informations

In strict mode:

$ nx type-check schematic-api
...
  /workspaces/sage-monorepo/apps/schematic/api/schematic_api/test/test_version_endpoints.py:39:13 - error: Type of "assert500" is partially unknown
    Type of "assert500" is "(response: Unknown, message: Unknown | None = None) -> None" (reportUnknownMemberType)
  /workspaces/sage-monorepo/apps/schematic/api/schematic_api/test/test_version_endpoints.py:40:17 - error: Argument type is unknown
    Argument corresponds to parameter "response" in function "assert500" (reportUnknownArgumentType)
  /workspaces/sage-monorepo/apps/schematic/api/schematic_api/test/test_version_endpoints.py:40:49 - error: Type of "data" is unknown (reportUnknownMemberType)
  /workspaces/sage-monorepo/apps/schematic/api/schematic_api/test/test_version_endpoints.py:40:49 - error: Type of "decode" is unknown (reportUnknownMemberType)
3417 errors, 0 warnings, 0 informations 
Warning: command "pyright" exited with non-zero status code

Runtime (strict mode):

$ hyperfine --warmup 1 --runs 10 'nx type-check schematic-api' --ignore-failure
Benchmark 1: nx type-check schematic-api
  Time (mean ± σ):     11.779 s ±  0.599 s    [User: 20.254 s, System: 2.055 s]
  Range (min … max):   11.171 s … 13.212 s    10 runs

Observations

• Pyright found 3417 errors in strict mode in schematic_api/. mypy found 483 errors in strict mode.
• Pyright is about 1s faster than mypy in non-strict mode (data not shown).
• Pyright takes about 2-3 seconds more than mypy in strict mode.

Comparison by ChatGPT

Here's a comparison table between mypy and pyright, two popular static type checkers for Python:

Feature	mypy	pyright
Language Support	Supports all standard Python syntax and features.	Supports Python 3.7 and above, and most Python features.
Performance	Slower for large projects due to single-threaded execution.	Faster performance with multi-threaded processing.
Ease of Use	Requires configuration file (mypy.ini or setup.cfg).	Easier to get started; config optional but uses pyrightconfig.json.
Editor Integration	Supported in most editors via plugins or extensions.	Integrated in Visual Studio Code with built-in support; also available in other editors.
Error Reporting	Detailed error messages, but can be verbose.	Clear and concise error messages, often easier to read.
Type Inference	Strong type inference, but may need type hints in complex scenarios.	Excellent type inference, often requires fewer annotations.
Third-Party Library Support	Strong support, but sometimes needs *.pyi stub files.	Excellent support, with automatic stubs for many libraries.
Configuration Flexibility	Highly configurable with options like strict mode, plugins.	Configurable with strict and custom rules, but fewer plugins.
Type Checking Features	Supports @no_type_check, @type: ignore, and more for finer control.	Similar controls with # pyright: ignore, type: ignore comments.
Active Development	Actively maintained, backed by Python Software Foundation.	Actively maintained, backed by Microsoft, with regular updates.
Community and Documentation	Strong community, extensive documentation and examples.	Growing community, good documentation, but fewer examples compared to mypy.
CLI Features	Rich command-line options for various checks.	Fewer CLI options, mainly geared towards editor integration.
IDE Integration	Available as a plugin or extension for most IDEs.	Best integrated with VS Code, works with others through LSP (Language Server Protocol).
Linting Capability	Focuses on type checking; separate linters like flake8 needed for style issues.	Provides some linting features out-of-the-box, but not as comprehensive as dedicated linters.

Overall, both tools are powerful for static type checking in Python. The choice between them often comes down to project requirements, editor preference, and performance needs.

[dpulls]

🎉 All dependencies have been resolved !