Merge pull request #6901 from smokestacklightnin/ci/docs/try-mkdocs

Build docs with `mkdocs`

.github/workflows/cd-docs.yml

@@ -0,0 +1,45 @@
+name: deploy-docs
+on:
+  workflow_dispatch:
+  push:
+    branches:
+      - master
+permissions:
+  contents: write
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repo
+        uses: actions/checkout@v4
+
+      - name: Configure Git Credentials
+        run: |
+          git config user.name github-actions[bot]
+          git config user.email 41898282+github-actions[bot]@users.noreply.github.com
+
+      - name: Set up Python 3.9
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.9'
+          cache: 'pip'
+          cache-dependency-path: |
+            setup.py
+            tfx/dependencies.py
+
+      - name: Save time for cache for mkdocs
+        run: echo "cache_id=$(date --utc '+%V')" >> $GITHUB_ENV
+
+      - name: Caching
+        uses: actions/cache@v4
+        with:
+          key: mkdocs-material-${{ env.cache_id }}
+          path: .cache
+          restore-keys: |
+            mkdocs-material-
+
+      - name: Install Dependencies
+        run: pip install mkdocs mkdocs-material mkdocstrings[python] griffe-inherited-docstrings mkdocs-autorefs mkdocs-jupyter mkdocs-caption
+
+      - name: Deploy to GitHub Pages
+        run: mkdocs gh-deploy --force

.github/workflows/ci-test.yml

@@ -52,7 +52,7 @@ jobs:
         python -m pip install --upgrade pip wheel
         # TODO(b/232490018): Cython need to be installed separately to build pycocotools.
         python -m pip install Cython -c ./test_constraints.txt
-        pip install -c ./test_constraints.txt --extra-index-url https://pypi-nightly.tensorflow.org/simple --pre --editable .[all]
+        pip install -c ./test_constraints.txt --extra-index-url https://pypi-nightly.tensorflow.org/simple --pre .[all]
       env:
         TFX_DEPENDENCY_SELECTOR: ${{ matrix.dependency-selector }}
 

.pre-commit-config.yaml

@@ -28,7 +28,7 @@ repos:
         exclude: '\.svg$'
       - id: check-json
       - id: check-yaml
-        args: [--allow-multiple-documents]
+        args: [--allow-multiple-documents, --unsafe]
       - id: check-toml
 
   - repo: https://github.com/astral-sh/ruff-pre-commit

docs/api/v1/components.md

@@ -0,0 +1,3 @@
+# Components
+
+::: tfx.v1.components

docs/api/v1/dsl.md

@@ -0,0 +1,3 @@
+# DSL
+
+::: tfx.v1.dsl

docs/api/v1/extensions.md

@@ -0,0 +1,3 @@
+# Extension
+
+::: tfx.v1.extensions

docs/api/v1/orchestration.md

@@ -0,0 +1,3 @@
+# Orchestration
+
+::: tfx.v1.orchestration

docs/api/v1/proto.md

@@ -0,0 +1,5 @@
+# Proto
+
+::: tfx.v1.proto
+    options:
+      show_if_no_docstring: true

docs/api/v1/root.md

@@ -0,0 +1,17 @@
+# Modules
+
+[components][tfx.v1.components] module: TFX components module.
+
+[dsl][tfx.v1.dsl] module: TFX DSL module.
+
+[extensions][tfx.v1.extensions] module: TFX extensions module.
+
+[orchestration][tfx.v1.orchestration] module: TFX orchestration module.
+
+[proto][tfx.v1.proto] module: TFX proto module.
+
+[testing][tfx.v1.testing] module: Public testing modules for TFX.
+
+[types][tfx.v1.types] module: TFX types module.
+
+[utils][tfx.v1.utils] module: TFX utils module.

docs/api/v1/testing.md

@@ -0,0 +1,3 @@
+# Testing
+
+::: tfx.v1.testing

docs/api/v1/types.md

@@ -0,0 +1,3 @@
+# Types
+
+::: tfx.v1.types

docs/api/v1/utils.md

@@ -0,0 +1,3 @@
+# Utils
+
+::: tfx.v1.utils

docs/guide/beam.md

@@ -56,9 +56,9 @@ Please follow one of the paths in
 [Managing Python Pipeline Dependencies](https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/)
 to provide this using one of the following beam_pipeline_args:
 
-*   --setup_file
-*   --extra_package
-*   --requirements_file
+*   `--setup_file`
+*   `--extra_package`
+*   `--requirements_file`
 
 Notice: In any of above cases, please make sure that the same version of `tfx`
 is listed as a dependency.

docs/guide/build_local_pipeline.md

@@ -35,7 +35,7 @@ pip install tfx
 ```
 
 If you are new to TFX pipelines,
-[learn more about the core concepts for TFX pipelines](understanding_tfx_pipelines)
+[learn more about the core concepts for TFX pipelines](understanding_tfx_pipelines.md)
 before continuing.
 
 ## Build a pipeline using a template
@@ -51,24 +51,24 @@ it to meet your needs.
 
 1.  See list of the available TFX pipeline templates:
 
-    <pre class="devsite-click-to-copy devsite-terminal">
+    ```bash
     tfx template list
-    </pre>
+    ```
 
 1.  Select a template from the list
 
-    <pre class="devsite-click-to-copy devsite-terminal">
-    tfx template copy --model=<var>template</var> --pipeline_name=<var>pipeline-name</var> \
-    --destination_path=<var>destination-path</var>
-    </pre>
+    ```bash
+    tfx template copy --model=template --pipeline_name=pipeline-name \
+    --destination_path=destination-path
+    ```
 
     Replace the following:
 
-    *   <var>template</var>: The name of the template you want to copy.
-    *   <var>pipeline-name</var>: The name of the pipeline to create.
-    *   <var>destination-path</var>: The path to copy the template into.
+    *   `template`: The name of the template you want to copy.
+    *   `pipeline-name`: The name of the pipeline to create.
+    *   `destination-path`: The path to copy the template into.
 
-    Learn more about the [`tfx template copy` command](cli#copy).
+    Learn more about the [`tfx template copy` command](cli.md#copy).
 
 1.  A copy of the pipeline template has been created at the path you specified.
 
@@ -99,13 +99,13 @@ This section provides an overview of the scaffolding created by a template.
 
 1.  Run the following commands in your pipeline directory:
 
-    <pre class="devsite-click-to-copy devsite-terminal">
+    ```bash
     tfx pipeline create --pipeline_path local_runner.py
-    </pre>
+    ```
 
-    <pre class="devsite-click-to-copy devsite-terminal">
+    ```bash
     tfx run create --pipeline_name <var>pipeline_name</var>
-    </pre>
+    ```
 
     The command creates a pipeline run using `LocalDagRunner`, which adds the
     following directories to your pipeline:
@@ -157,8 +157,8 @@ template.
     implement a pipeline for tabular data using the TFX standard components. If
     you are moving an existing ML workflow into a pipeline, you may need to
     revise your code to make full use of
-    [TFX standard components](index#tfx_standard_components). You may also need
-    to create [custom components](understanding_custom_components) that
+    [TFX standard components](index.md#tfx_standard_components). You may also need
+    to create [custom components](understanding_custom_components.md) that
     implement features which are unique to your workflow or that are not yet
     supported by TFX standard components.
 
@@ -194,17 +194,17 @@ without using a template.
     functionality to help you implement a complete ML workflow. If you are
     moving an existing ML workflow into a pipeline, you may need to revise your
     code to make full use of TFX standard components. You may also need to
-    create [custom components](understanding_custom_components) that implement
+    create [custom components](understanding_custom_components.md) that implement
     features such as data augmentation.
 
     *   Learn more about
-        [standard TFX components](index#tfx_standard_components).
-    *   Learn more about [custom components](understanding_custom_components).
+        [standard TFX components](index.md#tfx_standard_components).
+    *   Learn more about [custom components](understanding_custom_components.md).
 
 1.  Create a script file to define your pipeline using the following example.
     This guide refers to this file as `my_pipeline.py`.
 
-    <pre class="devsite-click-to-copy prettyprint">
+    ```python
     import os
     from typing import Optional, Text, List
     from absl import logging
@@ -248,7 +248,7 @@ without using a template.
     if __name__ == '__main__':
       logging.set_verbosity(logging.INFO)
       run_pipeline()
-    </pre>
+    ```
 
     In the coming steps, you define your pipeline in `create_pipeline` and run
     your pipeline locally using the local runner.
@@ -277,7 +277,7 @@ without using a template.
     pipeline uses the `ExampleGen` standard component to load a CSV from a
     directory at `./data`.
 
-    <pre class="devsite-click-to-copy prettyprint">
+    ```python
     from tfx.components import CsvExampleGen
 
     DATA_PATH = os.path.join('.', 'data')
@@ -315,7 +315,7 @@ without using a template.
         )
 
       tfx.orchestration.LocalDagRunner().run(my_pipeline)
-    </pre>
+    ```
 
     `CsvExampleGen` creates serialized example records using the data in the CSV
     at the specified data path. By setting the `CsvExampleGen` component's
@@ -326,13 +326,13 @@ without using a template.
 
 1.  Use the following command to run your `my_pipeline.py` script.
 
-    <pre class="devsite-click-to-copy devsite-terminal">
+    ```bash
     python my_pipeline.py
-    </pre>
+    ```
 
     The result should be something like the following:
 
-    <pre>
+    ```
     INFO:absl:Component CsvExampleGen depends on [].
     INFO:absl:Component CsvExampleGen is scheduled.
     INFO:absl:Component CsvExampleGen is running.
@@ -347,6 +347,6 @@ without using a template.
     INFO:absl:Running publisher for CsvExampleGen
     INFO:absl:MetadataStore with DB connection initialized
     INFO:absl:Component CsvExampleGen is finished.
-    </pre>
+    ```
 
 1.  Continue to iteratively add components to your pipeline.

docs/guide/build_tfx_pipeline.md

@@ -1,49 +1,49 @@
 # Building TFX pipelines
 
 Note: For a conceptual view of TFX Pipelines, see
-[Understanding TFX Pipelines](understanding_tfx_pipelines).
+[Understanding TFX Pipelines](understanding_tfx_pipelines.md).
 
 Note: Want to build your first pipeline before you dive into the details? Get
 started
-[building a pipeline using a template](https://www.tensorflow.org/tfx/guide/build_local_pipeline#build_a_pipeline_using_a_template).
+[building a pipeline using a template](build_local_pipeline.md#build-a-pipeline-using-a-template).
 
 ## Using the `Pipeline` class
 
 TFX pipelines are defined using the
 [`Pipeline` class](https://github.com/tensorflow/tfx/blob/master/tfx/orchestration/pipeline.py){: .external }.
 The following example demonstrates how to use the `Pipeline` class.
 
-<pre class="devsite-click-to-copy prettyprint">
+```python
 pipeline.Pipeline(
-    pipeline_name=<var>pipeline-name</var>,
-    pipeline_root=<var>pipeline-root</var>,
-    components=<var>components</var>,
-    enable_cache=<var>enable-cache</var>,
-    metadata_connection_config=<var>metadata-connection-config</var>,
+    pipeline_name=pipeline-name,
+    pipeline_root=pipeline-root,
+    components=components,
+    enable_cache=enable-cache,
+    metadata_connection_config=metadata-connection-config,
 )
-</pre>
+```
 
 Replace the following:
 
-*   <var>pipeline-name</var>: The name of this pipeline. The pipeline name must
+*   `pipeline-name`: The name of this pipeline. The pipeline name must
     be unique.
 
     TFX uses the pipeline name when querying ML Metadata for component input
     artifacts. Reusing a pipeline name may result in unexpected behaviors.
 
-*   <var>pipeline-root</var>: The root path of this pipeline's outputs. The root
+*   `pipeline-root`: The root path of this pipeline's outputs. The root
     path must be the full path to a directory that your orchestrator has read
     and write access to. At runtime, TFX uses the pipeline root to generate
     output paths for component artifacts. This directory can be local, or on a
     supported distributed file system, such as Google Cloud Storage or HDFS.
 
-*   <var>components</var>: A list of component instances that make up this
+*   `components`: A list of component instances that make up this
     pipeline's workflow.
 
-*   <var>enable-cache</var>: (Optional.) A boolean value that indicates if this
+*   `enable-cache`: (Optional.) A boolean value that indicates if this
     pipeline uses caching to speed up pipeline execution.
 
-*   <var>metadata-connection-config</var>: (Optional.) A connection
+*   `metadata-connection-config`: (Optional.) A connection
     configuration for ML Metadata.
 
 ## Defining the component execution graph

docs/guide/bulkinferrer.md

@@ -2,7 +2,7 @@
 
 The BulkInferrer TFX component performs batch inference on unlabeled data. The
 generated
-InferenceResult([tensorflow_serving.apis.prediction_log_pb2.PredictionLog](https://github.com/tensorflow/serving/blob/master/tensorflow_serving/apis/prediction_log.proto))
+InferenceResult([`tensorflow_serving.apis.prediction_log_pb2.PredictionLog`](https://github.com/tensorflow/serving/blob/master/tensorflow_serving/apis/prediction_log.proto))
 contains the original features and the prediction results.
 
 BulkInferrer consumes:
@@ -11,7 +11,7 @@ BulkInferrer consumes:
     [SavedModel](https://www.tensorflow.org/guide/saved_model.md) format.
 *   Unlabelled tf.Examples that contain features.
 *   (Optional) Validation result from
-    [Evaluator](https://www.tensorflow.org/tfx/guide/evaluator.md) component.
+    [Evaluator](evaluator.md) component.
 
 BulkInferrer emits:
 
@@ -21,9 +21,9 @@ BulkInferrer emits:
 
 A BulkInferrer TFX component is used to perform batch inference on unlabeled
 tf.Examples. It is typically deployed after an
-[Evaluator](https://www.tensorflow.org/tfx/guide/evaluator.md) component to
+[Evaluator](evaluator.md) component to
 perform inference with a validated model, or after a
-[Trainer](https://www.tensorflow.org/tfx/guide/trainer.md) component to directly
+[Trainer](trainer.md) component to directly
 perform inference on exported model.
 
 It currently performs in-memory model inference and remote inference.
@@ -42,4 +42,4 @@ bulk_inferrer = BulkInferrer(
 ```
 
 More details are available in the
-[BulkInferrer API reference](https://www.tensorflow.org/tfx/api_docs/python/tfx/v1/components/BulkInferrer).
+[BulkInferrer API reference][tfx.v1.components.BulkInferrer].