Description
The library is missing linting, CI pipeline, coverage and unified style for documentation so that autodoc can be used for easier handling in the future. Also add additional fixes that are useful.
- add rulesets in the project.
Things to add
- Linting: migrate to Ruff (lint + format + import-sort, replacing black/isort/flake8/pyupgrade), but keep line-length = 125. Add mypy + bandit.
- CI gates (all hard-blocking): tests + ruff, mypy, coverage, and the changelog gate.
- Add coverage requirement to fall under 60% for now (later update to 80%) - includes adding
pytest-cov to the test extra
- Add docs extra (optional-dependencies) that covers everything docs related:
sphinx, furo/sphinx_rtd_theme, myst-parser, sphinx-autodoc-typehints, etc. Currently the github/workflow for sphinx-build fails because there is no docs extra.
- Add dev extra - every package a developer needs for developing:
ruff, mypy, bandit[toml], pre-commit, nbmake, ipykernel + [test,docs].
- Replace hardcoded
packages = ["pquant"] with auto-discovery: [tool.setuptools.packages.find] where = ["src"] — currently subpackages (pquant.core, pquant.pruning_methods, pquant.data_models, pquant.configs) may be dropped from wheels.
- Add
[tool.setuptools.package-data] to ship the configs/*.yaml (and finetuning.yaml) files.
.pre-commit-config.yaml - swap linters for Ruff, add mypy + bandit - add kynan/nbstripout (strip notebook outputs) - Drop check-manifest (moving to packages.find + package-data makes MANIFEST unnecessary).
- Add community health files + wire changelog into docs.
- Set up API autodoc, enforce full docstring coverage and standardize docs on markdown files.
- Fix pre-existing issues with the library.
- Add tool caches/artifacts to
.gitignore. The new tooling generates local caches and coverage artifacts that must not be committed. Add: .ruff_cache/ (Ruff), .mypy_cache/ (mypy), .pytest_cache/ (pytest/nbmake), and .coverage, coverage.xml, htmlcov/ (coverage).
- Narrow the overly broad
*.txt ignore in .gitignore. Line 3 is a blanket *.txt. It isn't hiding anything today (the only .txt, docs/requirements.txt, is already tracked, and .gitignore doesn't untrack tracked files), but it will silently skip any new .txt file from git add/git status - problem for the future. Narrow it to the paths actually meant to be ignored (e.g. logs/*.txt) or remove it and ignore intended files explicitly.
Implementation details
CI pipeline + pre-merge
- Copy
ci.yml and pre-merge.yml from the template, but adapt the pytest/coverage/mypy jobs to the dual backend:
- (dual backend solution) CI test matrix: Run the pytest job as a matrix crossing KERAS_BACKEND (tensorflow, torch) × python-version (3.10, 3.11, 3.12) — 6 combinations. Each job sets the KERAS_BACKEND env var and installs the matching backend via pip install -e ".[test,tensorflow]" or ".[test,torch]" (the plain .[test] extra has no Keras backend, so tests can't run without this). This mirrors tests/run_tests.sh across all supported Python versions.
- Keep the
changelog, ruff, mypy, bandit, coverage, nbmake, and ci-ok gate jobs. Point branch protection's required check at CI · all checks passed (ci-ok).
- Add
pre-merge.yml: same lighter subset (ruff/mypy/bandit/pytest), backend-adapted - this is without coverage or notebooks, which will run on PRs and pushes to feature branch.
- CI changelog gate: the planned
ci.yml will include a changelog job requiring every PR to touch CHANGELOG.md (bypassable with a skip-changelog label). CHANGELOG.md must be created in the same PR that adds this gate, or that PR fails its own check.
Community health files + changelog
- Add CONTRIBUTING.md (contribution workflow: issue → branch → PR → review) and SECURITY.md (how to report vulnerabilities privately).
- Add CHANGELOG.md at repo root, following Keep a Changelog + semantic versioning, seeded with an ## [Unreleased] section and the current release version.
- Surface the changelog in the docs: add docs/source/changelog.md that pulls in the root file via MyST include (
{include} ../../CHANGELOG.md ) and add changelog to the toctree in docs/source/index.rst.
Documentation
Today docs/source/reference.md is hand-written tables - the actual public API (layers, pruning methods, quantizers) is not auto-documented, so docs drift from code. Wire up automatic API-reference generation from docstrings:
- Use the already-enabled sphinx.ext.autodoc + autosummary + napoleon (in
conf.py to generate an API reference from source.
- Fix the broken sphinx-apidoc call in docs/Makefile (it points at ../src/HGQ - a template leftover - instead of ../src/pquant).
- Standardize docstring style on Google — it's already the de-facto convention in the codebase (only
src/pquant/core/torch/hgq_quantizer.py uses NumPy style; convert it). This matches the Ruff pydocstyle convention = "google" setting.
- Every public function/method/class MUST have a docstring (Args, Returns, Raises where applicable). Current coverage is ~230 docstrings across ~900 defs - the bulk is undocumented. Ruff's D rule family (already in the lint config) enforces this and will flag every missing docstring; the CI ruff gate makes it blocking.
Standardize docs on MyST Markdown
- The docs currently mix one reStructuredText file
docs/source/index.rst with all-Markdown content pages, enabled by MyST-Parser + source_suffix = ['.rst', '.md']. Convert index.rst → index.md using MyST's Markdown toctree directive so the entire docs tree is a single format:
:maxdepth: 2
status
install
getting_started
reference
faq
Once converted, .rst support is no longer needed and source_suffix can be narrowed to ['.md'].
Fix pre-exiting bugs
tox.ini: [testenv] extras = testing → test (the real extra name; today it installs nothing).
.readthedocs.yaml: sphinx.configuration: docs/conf.py → docs/source/conf.py (real location); uncomment the python.install docs-requirements block.
docs/Makefile: sphinx-apidoc ... ../src/HGQ → ../src/pquant (HGQ is a template leftover). Fix the sphinx-build.yml docs workflow which cd docs; make html but the real conf is under docs/source/ - align it (and note it triggers on dev, not main).
README.md: fix links to non-existent docs/pruning_methods.md / docs/quantization_parameters.md.
- Remove stray root
__init__.py (accidental, not part of the package) - actual package is src/pquant
- Reconcile version signals:
docs/source/conf.py hardcodes release = "1.0.0" while setuptools-scm drives the real version - have conf read from the installed package metadata instead.
Execution: sub-issues & PR grouping
The work is split into 7 sub-issues (each independently reviewable), delivered in 3 PRs
straight to dev:
Sub-issues
- Packaging & tooling config (
pyproject.toml): Ruff/mypy/bandit/coverage config, packages.find, package-data, test/docs/dev extras. Foundation.
- Pre-commit migration: swap linters → Ruff, add mypy/bandit/nbstripout, drop check-manifest.
- CI pipeline + pre-merge:
ci.yml + pre-merge.yml, dual-backend matrix, ci-ok gate, changelog gate.
- Community health files:
CONTRIBUTING.md, SECURITY.md, CHANGELOG.md + changelog wired into docs.
- Documentation overhaul: autodoc + full Google-docstring coverage + MyST-Markdown standardization + docs-build CI job.
- Pre-existing bug fixes: tox / readthedocs / Makefile / README / version / stray root
__init__.py.
.gitignore cleanup: add tool caches; narrow *.txt.
PR grouping
- PR A - "Tooling": sub-issues 1 + 4 + 6 + 7. Config, community files (incl. seeding
CHANGELOG.md), bug fixes, gitignore. Green on arrival. Don't include the Ruff D docstring rule here (leave it out of select) - enabling it before the docstring backfill (PR C) would flood with errors. Land the one-time ruff format reformat as its own commit.
- PR B - "Enforcement": sub-issues 2 + 3. Pre-commit + CI/pre-merge. The changelog gate is safe because
CHANGELOG.md already exists from PR A. D rule is still not included.
- PR C - "Docs": sub-issue 5. Autodoc, Google-docstring backfill (now turn on the
D rule), MyST standardization.
Order: PR A → PR B → PR C.
Docs-build CI note (deliverable of Sub-issue 5 / PR C): Add a build-only docs job to ci.yml
(pip install -e ".[docs]" → cd docs && make html, fail on error with -W, no deploy), wired into
the ci-ok gate. Deferred to PR C (not PR B) because the docs don't build cleanly until autodoc + the
Sub-issue 6 fixes land — adding it earlier would be red on arrival. Deploy stays in the separate
(fixed) sphinx-build.yml. Runnable examples are covered by nbmake instead.
Acceptance Criteria
Description
The library is missing linting, CI pipeline, coverage and unified style for documentation so that autodoc can be used for easier handling in the future. Also add additional fixes that are useful.
Things to add
pytest-covto thetestextrasphinx,furo/sphinx_rtd_theme,myst-parser,sphinx-autodoc-typehints, etc. Currently the github/workflow for sphinx-build fails because there is no docs extra.ruff, mypy, bandit[toml], pre-commit, nbmake, ipykernel+[test,docs].packages = ["pquant"]with auto-discovery:[tool.setuptools.packages.find] where = ["src"]— currently subpackages (pquant.core,pquant.pruning_methods,pquant.data_models,pquant.configs) may be dropped from wheels.[tool.setuptools.package-data]to ship theconfigs/*.yaml(andfinetuning.yaml) files..pre-commit-config.yaml- swap linters for Ruff, add mypy + bandit - addkynan/nbstripout(strip notebook outputs) - Dropcheck-manifest(moving topackages.find+ package-data makes MANIFEST unnecessary)..gitignore. The new tooling generates local caches and coverage artifacts that must not be committed. Add: .ruff_cache/ (Ruff), .mypy_cache/ (mypy), .pytest_cache/ (pytest/nbmake), and .coverage, coverage.xml, htmlcov/ (coverage).*.txtignore in.gitignore. Line 3 is a blanket*.txt. It isn't hiding anything today (the only .txt, docs/requirements.txt, is already tracked, and .gitignore doesn't untrack tracked files), but it will silently skip any new .txt file from git add/git status - problem for the future. Narrow it to the paths actually meant to be ignored (e.g. logs/*.txt) or remove it and ignore intended files explicitly.Implementation details
CI pipeline + pre-merge
ci.ymlandpre-merge.ymlfrom the template, but adapt the pytest/coverage/mypy jobs to the dual backend:changelog,ruff,mypy,bandit,coverage,nbmake, andci-okgate jobs. Point branch protection's required check atCI · all checks passed(ci-ok).pre-merge.yml: same lighter subset (ruff/mypy/bandit/pytest), backend-adapted - this is without coverage or notebooks, which will run on PRs and pushes to feature branch.ci.ymlwill include a changelog job requiring every PR to touch CHANGELOG.md (bypassable with a skip-changelog label).CHANGELOG.mdmust be created in the same PR that adds this gate, or that PR fails its own check.Community health files + changelog
{include} ../../CHANGELOG.md) and add changelog to the toctree indocs/source/index.rst.Documentation
Today
docs/source/reference.mdis hand-written tables - the actual public API (layers, pruning methods, quantizers) is not auto-documented, so docs drift from code. Wire up automatic API-reference generation from docstrings:conf.pyto generate an API reference from source.src/pquant/core/torch/hgq_quantizer.pyuses NumPy style; convert it). This matches the Ruffpydocstyle convention = "google"setting.Standardize docs on MyST Markdown
docs/source/index.rstwith all-Markdown content pages, enabled by MyST-Parser +source_suffix = ['.rst', '.md']. Convert index.rst → index.md using MyST's Markdown toctree directive so the entire docs tree is a single format:Once converted,
.rstsupport is no longer needed and source_suffix can be narrowed to['.md'].Fix pre-exiting bugs
tox.ini:[testenv] extras = testing→test(the real extra name; today it installs nothing)..readthedocs.yaml:sphinx.configuration: docs/conf.py→docs/source/conf.py(real location); uncomment thepython.installdocs-requirements block.docs/Makefile:sphinx-apidoc ... ../src/HGQ→../src/pquant(HGQ is a template leftover). Fix thesphinx-build.ymldocs workflow whichcd docs; make htmlbut the real conf is underdocs/source/- align it (and note it triggers ondev, notmain).README.md: fix links to non-existentdocs/pruning_methods.md/docs/quantization_parameters.md.__init__.py(accidental, not part of the package) - actual package issrc/pquantdocs/source/conf.pyhardcodesrelease = "1.0.0"while setuptools-scm drives the real version - have conf read from the installed package metadata instead.Execution: sub-issues & PR grouping
The work is split into 7 sub-issues (each independently reviewable), delivered in 3 PRs
straight to
dev:Sub-issues
pyproject.toml): Ruff/mypy/bandit/coverage config,packages.find,package-data,test/docs/devextras. Foundation.ci.yml+pre-merge.yml, dual-backend matrix,ci-okgate, changelog gate.CONTRIBUTING.md,SECURITY.md,CHANGELOG.md+ changelog wired into docs.__init__.py..gitignorecleanup: add tool caches; narrow*.txt.PR grouping
CHANGELOG.md), bug fixes, gitignore. Green on arrival. Don't include the RuffDdocstring rule here (leave it out ofselect) - enabling it before the docstring backfill (PR C) would flood with errors. Land the one-timeruff formatreformat as its own commit.CHANGELOG.mdalready exists from PR A.Drule is still not included.Drule), MyST standardization.Order: PR A → PR B → PR C.
Docs-build CI note (deliverable of Sub-issue 5 / PR C): Add a build-only docs job to
ci.yml(
pip install -e ".[docs]"→cd docs && make html, fail on error with-W, no deploy), wired intothe
ci-okgate. Deferred to PR C (not PR B) because the docs don't build cleanly until autodoc + theSub-issue 6 fixes land — adding it earlier would be red on arrival. Deploy stays in the separate
(fixed)
sphinx-build.yml. Runnable examples are covered by nbmake instead.Acceptance Criteria