RFC setup of the Micropython library documentation

[Josverl]

Introduction and Motivation:

This proposal is to change the way that the documentation for the MicroPython library is maintained.

Currently all documentation is manually created in .rST format, which is error prone and makes it difficult to keep the documentation up-to-date with changes in the implementation of libraries. Also different notation preferences and the .rST oddities have let to numerous unclarities or onprecise documentation.

My proposal is to permanently add type stubs for the library modules, and generate (parts of the) library documentation from these stubs.

I propose to use type-stubs from the[ micropython-stubs](Josverl/micropython-stubs: Stubs of most MicroPython ports, boards and versions to make writing code that much simpler. (github.com)) project as the inital basis for this, and would continue to use these in the process to build the micropython-*-stubs I publish to PyPI.

To allow Sphinx generate documentation from type-stubs, a Sphinx add-in is needed. The best option I have found thus far is Sphinx AutoAPI 3.1.2 ). This is a full featured OSS project with active maintenance and an MIT licence.
It also supports documenting both .pyi type-stubs and .py modules and without the need to import them into CPython, which for many hardware based modules is essential.

Benefits:

• Verifiable correct basis for the documentation of modules functions and classes
• Documentation that uses consistent and unambiguous notation for functions, classes and methods.
• Improve the quality of the micropython-*-stubs published to PyPI, which will:
  • Detect more errors via static type checking
  • Further Improve IDE auto-completion
• Allow more MicroPython users to contribute to the above.

Scope:

The inital scope would be limited to the documentation under :

• repo: /docs/library
• rtd : MicroPython libraries — MicroPython latest documentation

Possible futures:

After initial integration of AutoAPI, it would be simple to add additional documentation pages that were impractical to add before, such as :

• Adding references to port specific modules to the Quick References, with the aim to de-duplicate maintenance effort.
• select or all modules from micropython-lib
• a complete and up-to-date overview of all modules
• modules per port or per bundle

Background Information:

• This has been worked on previously by Jim, an I have spoken to him on occasion but that work appears to have stalled.
  I spend a few days to try to understand his approach, and see if I could make that work by changing the way I generate what I refer to as "Doc-Stubs", but sphinx.ext.autodoc is incapable of working with .pyi stubs.
  MicroPython Season of Docs 2023 - Proposal - Google Docs
  In other words , the concept was sound , but autodoc just can't handle stub files or modules that are intended for MCUs

• Currently I maintain micropython-stubber which reads the dics/library/.rst files and converts then to .pyi files. This takes approx 1800 lines of single purpose code and 150+ code tests that I can mostly retire after this has completed.

• I may be overly optimistic, however I think micropython-stubs and stubber are taking care of of the heavy lifing needed to create the relevant .pyi files in the original docs proposal, and the set of 400+ stub quality tests that should give assurance of their initial quality.

• The current micropython type stubs are not perfect, nor complete.I strive them to be not wrong, and that already takes a lot of time.
  I have been looking for ways to apply things like adding @Overloads on methods, and to mkae it simpler to contribute, without needing to understand all the complexities of building a stub package that I have added over the years.

• Blog: Generating beautiful Python API documentation with Sphinx AutoAPI | bylr.info

Detailed Description:

To Implement this the following steps are needed

• Create a documentation guide that helps contributors decide how to best document things. This can also be used as the basis to create Jinja templates that are used by AutoAPI.
• Select the relevant type-stubs and add them to the MicroPython repo
  ( docs/stubs/library)
• Add the Sphinx AutoAPI and relevant configuration

• Customize AutoAPI's jinja templates used to generate documentation

• Update the existing .rST documents.
  Per document in docs/library:

  • Update the variuous python refencences to their autoapi equivalents:
    • .. module:: --> .. autoapimodule::
    • .. function::-->.. autoapifunction::
    • .. class::-->.. autoapiclass::
    • etc...
  • Remove associated details for that topic
  • regenerate the docs using make -C ./docs
  • check for errors and completeness

• Provide a guide-level explanation of the proposed change. What will it achieve?

• Include a reference-level explanation with technical details1.

Topics for discussion:

1. Toolset
   Is Sphinx-autoapi the correct tool, or are there better ones ?
   I have looked at :

   • Standard Sphinx autodoc sphinx.ext.autodoc

     • autodoc provides several directives that are versions of the usual py:module, py:class and so forth. On parsing time, they import the corresponding module and extract the docstring of the given objects, inserting them into the page source under a suitable py:module, py:class etc. directive.
     • The above directives work very well to (gradually) update the current documentation to use auto-generated documentation.
     • unable to import .pyi stubs , need complex workarouds and changes to the type-stub files

   • sphinx-autodoc2

     • Can import .pyi files
     • no support for autodoc directives

   • Sphinx AutoAPI

     • Can generate documentation for both .py and .pyi files
     • Does support autoapi-like directives
     • Can use custom templates
     • Can use custom filters which could be used to filter port or board specific modules , functions and classes.

2. Writing guidelines
   Currently all is documenteded in the .rST files.
   In the future the maintenance of that documentation will be split across two sources:

   1. The .rST for the Context and the Why to use a module, function or class
   2. The type-stub .pyi file tfor the How and the deatils , inculding parameter typing and return types.

   What is a common effective structure and we will need to document that as well.

3. Reviewing changes - Type-Stubs
   How to verifying correctness of changes to type-stubs.
   typestubs can be automatically verified in CI/CD, but there are limited standard tools for this.

   • A simple validation using Pyright and MyPy can be added to pytest with ease , but that will
   • complex cases by running typecheckers agains known valid code samples from different ports and version.
     micropython-stubs/tests/quality_tests at main · Josverl/micropython-stubs (github.com)

4. Reviewing changes - Sphinx
   A review is needed as always, but the review will be a bit more challenging as in the source the change is spread across different files, but for the result should be similar or the same. I know that RTD support building and publishing [PR previews](Pull request previews — Read the Docs user documentation) so perhaps that is a good option to allow efficient and effective reviewer

5. Documentation Format
   Should we use .rST(restructured text) or .MD (Markdown) for the documentation.
   Current docs are written in .rST , but .MD is more accessible.
   Currently Sphinx AutoAPI does not directy support .MD, but it is possible and on [their roadmap](Markdown support · Issue #287 · readthedocs/sphinx-autoapi · GitHub).
   If we indeed use Sphinx AutoAP, It seems sensible to wait

6. Publishing type-stubs
   I do not think it makes sense to publish the setof library type-stubs independently, as they do not match the set of stubs needed for any port or board.
   I do not want to inflate the scope of this with the added complexity of building , testing and publishing distibution packages for type stubs, but It is clearly related.
   That being said, I'm currently the sole owner of PyPI of these micropython-stubs, and that is not perfect either.

7. Documenting Constants/Attributes/Data
   There is no formal standard to add docstrings to module or class level properties, but there is a common practice that is also used in the micropython stubs.
   This method does not appear to be recognised by either of the API-documentation tools.

   String literals occurring elsewhere in Python code may also act as documentation. They are not recognized by the Python bytecode compiler and are not accessible as runtime object attributes (i.e. not assigned to doc), but two types of extra docstrings may be extracted by software tools:

   1. String literals occurring immediately after a simple assignment at the top level of a module, class, or init method are called “attribute docstrings”.

Sorry that this a bit of a longer post, but I have been mulling this for quite some time,
and would like to move forward on this, but do not want to do that entirely on my own.

[Josverl]

See https://josverl.github.io/autodoc201/ for a quick preview

[stinos]

Can you add a direct link to a source code sample which shows this new way of dcumenting?

[Josverl]

Thanks for your reaction Stinos,
I was beginning to wonder if anyone cared.

It is in that same repo that the pages are : https://github.com/Josverl/autodoc201
This is a partial copy of micropython/micropython:docs

Code - there is not too much of it actually, that is the point.

the Sphinxs conf.py needs:

• additional config for autoapi

• some logic to point to the stubs

• additional logic to pull in modules from micropython-lib [Option, added for exploration]

• a few hooks for for some pre/post processing.

  • docstring pre-processing to
    • Add a note for all modules from micropython-lib
    • can be extended to add other rule bases annotations
  • html postprocessing
  • to tidy up _typeshed.Incomplete ( I'd prefer another way but have not yet found one)

• i have been moving some functions into a separate module to avoid bloating conf.py too much

autoapi needs:

• customizable .rst / Jinja templates
  • as an example I have used these to move the Submodules section in the module/package pages down to keep it similar to the current pages.
• the stubs (created by a custom version of createstubs , still some details in the works)

Example module documentation:

• current plain .rst

• autoapi style

The results are docpages

• the existing library/array docpage
  these pages are either generated or individually customizable
• Full module page generated by autoapi
  These pages all follow the autoapi_templates, to provide simple consistency.

:mod:`array` -- arrays of numeric data
======================================

.. autoapimodule:: array
    :no-index:
    :members:
    :undoc-members:
    :special-members:
    :exclude-members: __init__, __weakref__

It is also possible to mix information autogenerated from the stubs, with additional text/documentation written in .rst ( or .md)
some examples are:

• https://github.com/Josverl/autodoc201/blob/Sphinx_AutoAPI/docs/library/btree.rst
• https://github.com/Josverl/autodoc201/blob/Sphinx_AutoAPI/docs/library/esp32.rst

[Josverl]

Documentation for overloads
As I am working through, and comparing the outcome of a few modules I notice that the current documentation has a few occurrences of what I understand are essentially @overloads of the same methods, including documentation on each overload.
One example is esp.osdebug(...):

• esp.osdebug(uart_no) for esp8622
• esp.osdebug(uart_no[, level]) for esp32

As according to the standards, overloads may not have docstrings, and Shpinxs AutoAPI does not generate documentation for any but the last,
these will need to be combined into a single docstring.

Note: VSCode's Pylance does show both docstrings, and even shows the one that matches the overload best.

[stinos]

Checked the high level and some details and overall I'd say this is better than the current situation, mainly because from my point of view having the type hints in the docs is rather valueable plus it bridges docs and stubs and imo the stubs should be made an official part of micropython.

Still with respect to

Currently all documentation is manually created in .rST format, which is error prone and makes it difficult to keep the documentation up-to-date with changes in the implementation of libraries. Also different notation preferences and the .rST oddities have let to numerous unclarities or onprecise documentation.

I don't think this changes much. Not that that is a problem, it's inherent to the Python language probably, but all of these issues still mostly remain, right? Now the stubs have to be kept up-to-date, still use .rst, there can still be different styles.

And there's one thing (which can maybe solved though) which actually seems a bit worse: opening a .rst file in most editors gives syntax highlighting. Opening a .pyi file in VSCode just shows the dosctrings as docstrings so none of the .rst in those strings is recognized. However when Python linting with pretty much default settings kicks in it'll complain about line length, lack of newline between functions etc. So as far as writing the documentation goes this might be a bit harder, or maybe there's tooling which can solve that.

[Josverl]

I agree that mixing languages in the same document is something that even modern IDEs struggle with.
in the current .rst files the most mistakes are made with the notation in the calls and function defs.
Today in the documentation this also shows, In quite a lot of cases the notation is just plain illegal python, with wrong / mismatched quotes and backticks , multiple **, use of reserved keywords , and all types of parameter mistakes.
Currently I maintain a rather long list of algorithmic and (regex) search & replaces to try to correct these., and the result can be validated as syntactically correct. ( sending PRs for these could not keep keep up with the rate of newly found errors).
The advantage is that each of the docstring are shorter than the average page, but agree that I've needed to copy/paste existing docstrings from micropython-lib, to a native .rst editor to allow it to educate me on where the errors were.

Also currently the return annotations are, at best , sometimes described in the text. I think this is another thing that can better be described in python-stubs - with embedded .rst, rather than the other way around.
I read somewhere that there are recent VScode improvements to displaying .rst in docstrings ( and thus also in stubs with docstrings) but I cant find the link anymore.

Another benefit of using stubs is that different documentation pages can refer to/ include parts ( functions / classes/ methods ) from a common source.
For instance the time.time method is used in quite a few samples and pages.
Using stubs & autoapi, each page could 'copy in' the signature using .. autoapifunction:: time.time which would render the complete function signature , including parameter and return annotations.
The then more apparent differences between ports could / should then be documented in the respective page, or in the function's docstrings.

But one thing indeed remains the same; good documentation does not write itself, not even in the GPT era.

[tannewt]

We've been doing this quite a while in CircuitPython and it has worked well. We store Python stubs in the C files that implement the Python API (shared-bindings in CP land) by having them in //| prefixed comments. We then strip these lines out into .pyi files to make stubs and use AutoAPI to generate the rST that sphinx uses.

[Josverl]

When I started putting together the stubs a few years ago I did research on that approach as there were a few modxxxx.c files that had similar type comments, but iirc I could reproduce less than 1% of the python API.
But it's definitely an option to store the fragments in the .c files, to make it simpler to keep the implementation in sync with the python signatures.

In the stubs repro I also run tests to verify the stubs against known good code. Is there similar verification in CircuitPython, or is it mainly manual verification on changes?

[tannewt]

But it's definitely an option to store the fragments in the .c files, to make it simpler to keep the implementation in sync with the python signatures.

That's why we did it this way. It serves as good documentation too.

In the stubs repro I also run tests to verify the stubs against known good code. Is there similar verification in CircuitPython, or is it mainly manual verification on changes?

We don't have great verification after the fact. @elpekenin has been improving ours here: adafruit#9548 They found some mismatches between the native CircuitPython API implementation and the Blinka version that they are normalizing.

[Josverl]

In the micropython-stubs repo I have based QA or snippet tests based off the mypy / typeshed testing approach so that i can validate that changes in the stubs, do not break the typecheckers. Essentially this runs typecheckers in CI/CD against known good ( or bad) snippets of code.
The result is the snippet_score, that makes me feel good if it changes in the right direction.
https://micropython-stubs.readthedocs.io/en/main/qa_testing.html
and https://github.com/Josverl/micropython-stubs/tree/main/tests/quality_tests

wrtr to scope
I do not think that the above should be part of the micropython repo
I have been considering to extend this with sphinx doctest, but have not gotten there yet.
Sphinx based docstest]() that validate the stubs against the examples, rather than just try to run the code, could be a good addition for the code samples that cannot be actually run in CI/CD.

I think that may be a next step to consider though