Skip to content

Integrate python template tooling + additional fixes #53

Description

@emapuljak

Description

The library is missing linting, CI pipeline, coverage and unified style for documentation so that autodoc can be used for easier handling in the future. Also add additional fixes that are useful.

  • add rulesets in the project.

Things to add

  1. Linting: migrate to Ruff (lint + format + import-sort, replacing black/isort/flake8/pyupgrade), but keep line-length = 125. Add mypy + bandit.
  2. CI gates (all hard-blocking): tests + ruff, mypy, coverage, and the changelog gate.
  3. Add coverage requirement to fall under 60% for now (later update to 80%) - includes adding pytest-cov to the test extra
  4. Add docs extra (optional-dependencies) that covers everything docs related: sphinx, furo/sphinx_rtd_theme, myst-parser, sphinx-autodoc-typehints, etc. Currently the github/workflow for sphinx-build fails because there is no docs extra.
  5. Add dev extra - every package a developer needs for developing: ruff, mypy, bandit[toml], pre-commit, nbmake, ipykernel + [test,docs].
  6. Replace hardcoded packages = ["pquant"] with auto-discovery: [tool.setuptools.packages.find] where = ["src"] — currently subpackages (pquant.core, pquant.pruning_methods, pquant.data_models, pquant.configs) may be dropped from wheels.
  • Add [tool.setuptools.package-data] to ship the configs/*.yaml (and finetuning.yaml) files.
  1. .pre-commit-config.yaml - swap linters for Ruff, add mypy + bandit - add kynan/nbstripout (strip notebook outputs) - Drop check-manifest (moving to packages.find + package-data makes MANIFEST unnecessary).
  2. Add community health files + wire changelog into docs.
  3. Set up API autodoc, enforce full docstring coverage and standardize docs on markdown files.
  4. Fix pre-existing issues with the library.
  5. Add tool caches/artifacts to .gitignore. The new tooling generates local caches and coverage artifacts that must not be committed. Add: .ruff_cache/ (Ruff), .mypy_cache/ (mypy), .pytest_cache/ (pytest/nbmake), and .coverage, coverage.xml, htmlcov/ (coverage).
  6. Narrow the overly broad *.txt ignore in .gitignore. Line 3 is a blanket *.txt. It isn't hiding anything today (the only .txt, docs/requirements.txt, is already tracked, and .gitignore doesn't untrack tracked files), but it will silently skip any new .txt file from git add/git status - problem for the future. Narrow it to the paths actually meant to be ignored (e.g. logs/*.txt) or remove it and ignore intended files explicitly.

Implementation details

CI pipeline + pre-merge

  • Copy ci.yml and pre-merge.yml from the template, but adapt the pytest/coverage/mypy jobs to the dual backend:
  • (dual backend solution) CI test matrix: Run the pytest job as a matrix crossing KERAS_BACKEND (tensorflow, torch) × python-version (3.10, 3.11, 3.12) — 6 combinations. Each job sets the KERAS_BACKEND env var and installs the matching backend via pip install -e ".[test,tensorflow]" or ".[test,torch]" (the plain .[test] extra has no Keras backend, so tests can't run without this). This mirrors tests/run_tests.sh across all supported Python versions.
  • Keep the changelog, ruff, mypy, bandit, coverage, nbmake, and ci-ok gate jobs. Point branch protection's required check at CI · all checks passed (ci-ok).
  • Add pre-merge.yml: same lighter subset (ruff/mypy/bandit/pytest), backend-adapted - this is without coverage or notebooks, which will run on PRs and pushes to feature branch.
  • CI changelog gate: the planned ci.yml will include a changelog job requiring every PR to touch CHANGELOG.md (bypassable with a skip-changelog label). CHANGELOG.md must be created in the same PR that adds this gate, or that PR fails its own check.

Community health files + changelog

  • Add CONTRIBUTING.md (contribution workflow: issue → branch → PR → review) and SECURITY.md (how to report vulnerabilities privately).
  • Add CHANGELOG.md at repo root, following Keep a Changelog + semantic versioning, seeded with an ## [Unreleased] section and the current release version.
  • Surface the changelog in the docs: add docs/source/changelog.md that pulls in the root file via MyST include ({include} ../../CHANGELOG.md ) and add changelog to the toctree in docs/source/index.rst.

Documentation
Today docs/source/reference.md is hand-written tables - the actual public API (layers, pruning methods, quantizers) is not auto-documented, so docs drift from code. Wire up automatic API-reference generation from docstrings:

  • Use the already-enabled sphinx.ext.autodoc + autosummary + napoleon (in conf.py to generate an API reference from source.
  • Fix the broken sphinx-apidoc call in docs/Makefile (it points at ../src/HGQ - a template leftover - instead of ../src/pquant).
  • Standardize docstring style on Google — it's already the de-facto convention in the codebase (only src/pquant/core/torch/hgq_quantizer.py uses NumPy style; convert it). This matches the Ruff pydocstyle convention = "google" setting.
  • Every public function/method/class MUST have a docstring (Args, Returns, Raises where applicable). Current coverage is ~230 docstrings across ~900 defs - the bulk is undocumented. Ruff's D rule family (already in the lint config) enforces this and will flag every missing docstring; the CI ruff gate makes it blocking.

Standardize docs on MyST Markdown

  • The docs currently mix one reStructuredText file docs/source/index.rst with all-Markdown content pages, enabled by MyST-Parser + source_suffix = ['.rst', '.md']. Convert index.rst → index.md using MyST's Markdown toctree directive so the entire docs tree is a single format:
:maxdepth: 2

status
install
getting_started
reference
faq

Once converted, .rst support is no longer needed and source_suffix can be narrowed to ['.md'].

Fix pre-exiting bugs

  • tox.ini: [testenv] extras = testingtest (the real extra name; today it installs nothing).
  • .readthedocs.yaml: sphinx.configuration: docs/conf.pydocs/source/conf.py (real location); uncomment the python.install docs-requirements block.
  • docs/Makefile: sphinx-apidoc ... ../src/HGQ../src/pquant (HGQ is a template leftover). Fix the sphinx-build.yml docs workflow which cd docs; make html but the real conf is under docs/source/ - align it (and note it triggers on dev, not main).
  • README.md: fix links to non-existent docs/pruning_methods.md / docs/quantization_parameters.md.
  • Remove stray root __init__.py (accidental, not part of the package) - actual package is src/pquant
  • Reconcile version signals: docs/source/conf.py hardcodes release = "1.0.0" while setuptools-scm drives the real version - have conf read from the installed package metadata instead.

Execution: sub-issues & PR grouping

The work is split into 7 sub-issues (each independently reviewable), delivered in 3 PRs
straight to dev:

Sub-issues

  1. Packaging & tooling config (pyproject.toml): Ruff/mypy/bandit/coverage config, packages.find, package-data, test/docs/dev extras. Foundation.
  2. Pre-commit migration: swap linters → Ruff, add mypy/bandit/nbstripout, drop check-manifest.
  3. CI pipeline + pre-merge: ci.yml + pre-merge.yml, dual-backend matrix, ci-ok gate, changelog gate.
  4. Community health files: CONTRIBUTING.md, SECURITY.md, CHANGELOG.md + changelog wired into docs.
  5. Documentation overhaul: autodoc + full Google-docstring coverage + MyST-Markdown standardization + docs-build CI job.
  6. Pre-existing bug fixes: tox / readthedocs / Makefile / README / version / stray root __init__.py.
  7. .gitignore cleanup: add tool caches; narrow *.txt.

PR grouping

  • PR A - "Tooling": sub-issues 1 + 4 + 6 + 7. Config, community files (incl. seeding CHANGELOG.md), bug fixes, gitignore. Green on arrival. Don't include the Ruff D docstring rule here (leave it out of select) - enabling it before the docstring backfill (PR C) would flood with errors. Land the one-time ruff format reformat as its own commit.
  • PR B - "Enforcement": sub-issues 2 + 3. Pre-commit + CI/pre-merge. The changelog gate is safe because CHANGELOG.md already exists from PR A. D rule is still not included.
  • PR C - "Docs": sub-issue 5. Autodoc, Google-docstring backfill (now turn on the D rule), MyST standardization.

Order: PR A → PR B → PR C.

Docs-build CI note (deliverable of Sub-issue 5 / PR C): Add a build-only docs job to ci.yml
(pip install -e ".[docs]"cd docs && make html, fail on error with -W, no deploy), wired into
the ci-ok gate. Deferred to PR C (not PR B) because the docs don't build cleanly until autodoc + the
Sub-issue 6 fixes land — adding it earlier would be red on arrival. Deploy stays in the separate
(fixed) sphinx-build.yml. Runnable examples are covered by nbmake instead.

Acceptance Criteria

  • PR A passed
  • sub-issue PR B passed - fixing tests
  • PR B passed
  • PR C passed

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions