diff --git a/.readme_generation/dev_guide.md b/.readme_generation/dev_guide.md new file mode 100644 index 00000000..c4a552c9 --- /dev/null +++ b/.readme_generation/dev_guide.md @@ -0,0 +1,157 @@ +# Monorepo Development + +## Overview + +This document aims to provide a thorough understanding of this monorepo setup. It covers +file layout, dependency management, tooling, common processes, and more. +Because there may be multiple monorepos in use by GHGA, the information is agnostic +towards the content therein. + +## Monorepo Structure + +The monorepo houses multiple microservice projects while maintaining isolation between +each service. Services are located in the [`services`](/services) folder in the root directory. +| Root-Level Directory | Description | +|-----------|-------------| +| [`/.github`](/.github) | Workflows to be executed on GitHub | +| [`/.readme_generation`](/.readme_generation) | Template files required to build the root-level README | +| [`/example_data`](/example_data) | Can be used to populate the application with data for test/dev | +| [`/lock`](/lock) | Dependency specifications and resulting lock files | +| [`/scripts`](/scripts) | Custom scripts for common tasks such as updating lock files or config docs | +| [`/services`](/services) | Application code for each of the microservices in the monorepo | + + +### Dependency Management + +Dependency management consists of defining 3rd party python package requirements and +pinning explicit versions of those requirements in what's known as a _lock file_. When +dependencies are installed for the project, they are installed using that lock file, +which is stored as [`requirements.txt`](/lock/requirements.txt) in `/lock`. +Dependencies are shared across all services in the monorepo, so it is not possible, for +example, to use hexkit v2 in one service and hexkit v3 in another. +There is a templating system in place that allows for cleaner control over dependencies. +Here's how it works: +1. In `/lock`, there are several `requirements-*` files: + - `requirements-dev-template.in`: Contains *uncapped* (no upper bound) dependencies + that are controlled by the Microservice Template Repository and used for development + in all GHGA microservice projects (including monorepos). + - `requirements-dev.in`: Contains any other dependencies required for development of + this repository specifically. + - `requirements-dev.txt`: Lock file containing production *and* development dependencies. + - `requirements.txt`: Lock file containing *only* production dependencies. + +2. `.pyproject_generation/pyproject_custom.toml` is used to define production-specific +python dependencies. When changes are made, `scripts/update_pyproject.py` is +run to update the `pyproject.toml` file at the root level. + +3. `scripts/update_lock.py --upgrade` is used to combine information from `requirements-dev-template.in`, +`requirements-dev.in`, and `pyproject.toml` to build the two lock files listed above. + + +### Monorepo Configuration + +Outside of the services, there are two primary points of configuration: `pyproject.toml` +and `.pre-commit-config.yaml`. Three, if you include the contents of `.devcontainer`. + +The root-level pyproject.toml file is updated not directly, but rather through the files +in the `.pyproject_generation` folder. Tooling configuration for ruff, mypy, pytest, etc. +are contained in `pyproject_template.toml`. When making changes, remember to run the +update script to ensure changes are reflected in the actual pyproject.toml file. + +Pre-commit is configured through the .pre-commit-config.yaml found in the root directory. +There are standard pre-commit checks as well as 3rd party checks from ruff and mypy. +The locally-sourced check at the top of the file is used to ensure that ruff, mypy, and +any other checks are kept up to date with their package versions listed in the lock file. + +### Tooling Scripts + +There are an assortment of scripts developed in-house to aid in common development work. +These scripts are all contained in `/scripts`. The titles should be self-explanatory. + +There is a keystroke-friendly command copied over with the devcontainer called +`update_service_files` that can be used to execute the scripts on one or all services. +- E.g.: `update_service_files openapi ifrs` will update the openapi docs for just the ifrs. +- Run `update_service_files` alone to see command help. + +### Code Quality Tools + +We previously used Black, flake8, iSort, and other tools to enforce rigid standards for +code quality, but these have been replaced with `ruff` for both linting and formatting. +Mypy supplies type-checking help. These tools are configured via the pyproject.toml file +and run with pre-commit (as well as manually). + +### CI/CD + +The CI/CD pipeline is largely carried out by GitHub actions and workflows. The monorepo +repository is concerned only with the workflows defined in `.github/workflows/`. +Here, each of the files represents a workflow that is executed on GitHub upon a configured +point, such as when new changes are pushed, or when a PR is opened. + +The defined workflows carry out static code analysis checks, execute tests, and push +new images to Docker Hub. To ensure that workflows are only triggered when necessary, the +[`get_affected_services`](/.github/workflows/get_affected_services.yaml) workflow is used +to examine git history and only run workflows for services affected by the current branch. + +### Docker + +Services are deployed in production via docker. In a polyrepo setup, each service maintains +its own production Dockerfile, but in this monorepo setup there is only one Dockerfile +for all services. When a release is made, the Dockerfile is used to build a docker image, +which is then pushed to Docker Hub. + +## Service Structure + +Services consist of the following high-level parts: +- `.readme_generation/`: Template components required to build the service-specific README. +- `scripts/`: Required only if the service uses FastAPI for a REST API. +- `src`: Application code stored here in a subdirectory labeled with the service abbreviation. +- `tests_`: E.g. `tests_pcs`. All service-specific tests are stored here. +- Config Files: `config_schema.json`, `dev_config.yaml`, `example_config.yaml` +- `openapi.yaml`: The OpenAPI specification (only required if using FastAPI). +- `pyproject.toml`: The service's metadata, used to install the package. +- `README.md`: Describes the service's purpose, configuration, and design. + +### Service Configuration + +Configuration is stored in `.yaml` format at the service level. I.e., there is one config +file for each directory under `services/` called `dev_config.yaml`. The config +is defined in application code using Pydantic, and the `scripts/update_config_docs.py` +script uses the code (a class called `Config`) and the dev_config.yaml file to compile +both `config_schema.json` and `example_config.yaml`. + +The dev_config.yaml for each service is loaded into the development environment when the +devcontainer is activated, but *only* if it is listed in `.devcontainer/docker-compose.yml` +under the `environment` section as _CONFIG_YAML. +- E.g.: `IRS_CONFIG_YAML: /workspace/services/irs/dev_config.yaml` + +### Testing + +Testing is performed with `pytest`, which is configured in the pyproject.toml at the repo +root. At the service level, tests are stored in the folder named `tests_`, +e.g. `tests_ifrs` for the ifrs project or `tests_pcs` for the pcs project. The tests +folder lives at the root of the service-specific directory, i.e. +`services//tests_`. + +Tests can be run for all services with the command `pytest`. For a specific service only, +add the service directory: `pytest services/ifrs`. + +## Versioning + +Service package versions are maintained in the service-specific pyproject.toml files. +The monorepo version number is updated via pyproject_custom.toml in .pyproject_generation. +For a given version change, only the highest-impact changes need to be considered. For +example, imagine two PRs are merged before a version number is bumped and a release made. +PR #1 is a low-impact change that fixes a small bug. No API changes are performed. +PR #2, however, modifies several configuration parameters and adds a REST API endpoint. +PR #1 is a patch-number change, and PR #2 is a major-number change. Therefore, when +updating the monorepo version number for a release with these two changes, the monorepo +major version number should be incremented, not the patch number. + +## Development Conventions + +Development in a monorepo setup requires adherence to conventions to minimize developer +blocking. All changes that span >1 service should be executed in a dedicated PR. This +includes lock file/dependency updates, widespread refactoring, library changes, etc. +Normally, branches and the resulting PR should be isolated to one service. This ensures +that developers do not inadvertently create merge conflicts. Coordinate monorepo-spanning +changes with other developers. diff --git a/.readme_generation/readme_template_monorepo.md b/.readme_generation/readme_template_monorepo.md index c0b00d84..d465d17e 100644 --- a/.readme_generation/readme_template_monorepo.md +++ b/.readme_generation/readme_template_monorepo.md @@ -38,6 +38,9 @@ The installation is performed automatically when you build the devcontainer. How if you update dependencies in the [`./pyproject.toml`](./pyproject.toml) or the [`./requirements-dev.txt`](./requirements-dev.txt), please run it again. +For more information on development with this monorepo, please see the +[Developer Guide](./.readme_generation/dev_guide.md). + ## License This repository is free to use and modify according to the diff --git a/README.md b/README.md index 474edba7..be8fdd71 100644 --- a/README.md +++ b/README.md @@ -41,6 +41,9 @@ The installation is performed automatically when you build the devcontainer. How if you update dependencies in the [`./pyproject.toml`](./pyproject.toml) or the [`./requirements-dev.txt`](./requirements-dev.txt), please run it again. +For more information on development with this monorepo, please see the +[Developer Guide](./.readme_generation/dev_guide.md). + ## License This repository is free to use and modify according to the