diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index d2a5223..a739c56 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -7,12 +7,20 @@ docs_dir: src nav: - Overview: index.md + - News: blog/index.md - Installation: installation.md - Tutorials: - Getting Started: tutorials/getting_started.md - Concepts: + - Projects: concepts/projects.md - CI: concepts/ci.md - Reference: + - Blueprints: reference/blueprint.md + - Deployments: reference/deployments.md + - Releases: + - Overview: reference/releases/index.md + - Docker: reference/releases/docker.md + - GitHub: reference/releases/github.md - Targets: reference/targets.md theme: @@ -45,4 +53,10 @@ markdown_extensions: - admonition - pymdownx.details - pymdownx.snippets - - pymdownx.superfences \ No newline at end of file + - pymdownx.superfences + +plugins: + - blog: + archive: false + categories: false + - tags \ No newline at end of file diff --git a/docs/src/blog/.authors.yml b/docs/src/blog/.authors.yml new file mode 100644 index 0000000..ef84dd1 --- /dev/null +++ b/docs/src/blog/.authors.yml @@ -0,0 +1,5 @@ +authors: + jmgilman: + name: Joshua Gilman + description: Maintainer + avatar: https://avatars.githubusercontent.com/u/2308444?v=4 \ No newline at end of file diff --git a/docs/src/blog/index.md b/docs/src/blog/index.md new file mode 100644 index 0000000..0e63028 --- /dev/null +++ b/docs/src/blog/index.md @@ -0,0 +1,2 @@ +# News + diff --git a/docs/src/blog/posts/001-whats-new-in-forge.md b/docs/src/blog/posts/001-whats-new-in-forge.md new file mode 100644 index 0000000..1a5d8d0 --- /dev/null +++ b/docs/src/blog/posts/001-whats-new-in-forge.md @@ -0,0 +1,147 @@ +--- +draft: false +date: 2024-10-25 +authors: + - jmgilman +--- + +# What's New in Forge - 10-25-2024 + +Check out what's new in Forge this week. + + + +## Releases + +The `publish` and `release` targets are no more! +They have been replaced with an entirely new system that will enable adding more release automation going forward. +Individual releases are now defined in a project's blueprint and Forge will automatically discover and execute them in the CI +pipeline. +Each release is run in parallel to maximize speed. + +The old targets will no longer automatically run in the CI. +You will need to configure new releases in your project's blueprint file to continue publishing/releasing. +The `publish` target has been replaced with the `docker` release type. +The `release` target has been replaced with the `github` release type. + +For example, you can continue to use the `publish` target in your `Earthfile` by configuring a `docker` release type: + +```cue +project: { + name: "myproject" + release: { + docker: { + on: { + merge: {} + tag: {} + } + config: { + tag: _ @forge(name="GIT_COMMIT_HASH") + } + target: "publish" + } + } +} +``` + +The above configuration will create a new docker release whenever a merge to the default branch occurs or when a new git tag is +created. +The published image will have its tag (`config.tag` above) automatically filled in with the git commit hash. +Finally, Forge will call the `publish` target in your `Earthfile` to generate the container image. + +To learn more about releases, please refer to the [reference documentation](../../reference/releases/index.md). + +## Deployment Templates + +A new command has been introduced to the CLI: `forge deploy template`. +This command can be used to generate the raw Kubernetes manifests (in YAML) that will be applied to the target Kubernetes cluster +during automatic deployments. +This is useful when troubleshooting why a deplyoment may be failing or acting in an unexpected way. +All generated manifests will be printed to `stdout` and can be redirected to a local file for easier viewing. + +The below example shows what it looks like to generate the raw manifests for the Foundry API server: + +```text +$ forge deploy template foundry/api +--- +# Instance: foundry-api +--- +apiVersion: v1 +kind: Service +metadata: + labels: + app.kubernetes.io/managed-by: timoni + app.kubernetes.io/name: foundry-api + app.kubernetes.io/version: 0.1.0 + name: foundry-api + namespace: default +spec: + ports: + - name: http + port: 80 + protocol: TCP + targetPort: http + selector: + app.kubernetes.io/name: foundry-api + type: ClusterIP +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + labels: + app.kubernetes.io/managed-by: timoni + app.kubernetes.io/name: foundry-api + app.kubernetes.io/version: 0.1.0 + name: foundry-api + namespace: default +spec: + replicas: 1 + selector: + matchLabels: + app.kubernetes.io/name: foundry-api + template: + metadata: + labels: + app.kubernetes.io/name: foundry-api + spec: + containers: + - image: 332405224602.dkr.ecr.eu-central-1.amazonaws.com/foundry-api:763fe7fd2bfdd39d630df9b5c5aa7e6588fc6eea + imagePullPolicy: IfNotPresent + livenessProbe: + httpGet: + path: / + port: http + initialDelaySeconds: 5 + periodSeconds: 5 + name: foundry-api + ports: + - containerPort: 8080 + name: http + protocol: TCP + readinessProbe: + httpGet: + path: / + port: http + initialDelaySeconds: 5 + periodSeconds: 5 +``` + +For more information, please refer to the [deployments documentation](../../reference/deployments.md#templating). + +## What's Next + +Work is currenetly being done to improve automatic deployments for projects. +Currently, Forge assumes a GitOps repository exists and will automatically generate and commit updated deployments to the configured +repository. +This makes setup complicated and introduces a mostly unecessary step in the deployment process. + +Instead, we are investigating having a GitOps operator (currently only Argo CD) point directly at a project's repository. +Since a blueprint file is self-contained, it's possible to generate Kubernetes manifests using only the information inside of it. +The first steps towards experimenting with this new solution was to create a +[custom management plugin](https://github.com/input-output-hk/catalyst-forge/tree/master/tools/argocd) capable of ingesting a +project and spitting out raw Kubernetes manifests. +With this in place, it should be possible to point Argo CD directly at a project folder and have it generate the necessary manifests +for deploying the project. +As this process matures, more documentation will be released with the updated deployment process. + +That's it for this week, thanks for tuning in! \ No newline at end of file diff --git a/docs/src/concepts/ci.md b/docs/src/concepts/ci.md index 70aa961..7093340 100644 --- a/docs/src/concepts/ci.md +++ b/docs/src/concepts/ci.md @@ -20,14 +20,14 @@ It then filters and orders these targets into discrete target groups using a lis The name and dependency order of these groups is hardcoded and does not often change. Each of these target groups can be considered _phases_ in the overall CI pipeline. -Each phase consists of the associated targets and each phase occur in dependency order. +Each phase consists of the associated targets and each phase occurs in dependency order. The name and order of these phases is hardcoded and does not often change. ### Execution -For each phase, the CI system spawns a series of parallel jobs that executes each individual target. -Each target execution is given its own unique job in GitHub Actions to allow easy identification of failing targets as well as -providing an isolated log stream for a single target. +For each phase, the CI system spawns a series of parallel jobs that executes the current target for all discovered projects. +Each project execution is given its own unique job in GitHub Actions to allow easy identification of failing targets as well as +providing an isolated log stream. If any target in the group fails, the entire group is considered to be failed, and CI execution stops. !!! hint @@ -37,13 +37,12 @@ If any target in the group fails, the entire group is considered to be failed, a This promotes a "full ownership" philosophy where developers are responsible for ensuring their changes keep CI passing. Projects are not required to define targets for each phase. -In some cases, a phase may only contain a subset of projects. +In some cases, a phase may only contain a subset of all projects in the repository. If a phase ends up with zero targets, the entire job is skipped. This allows repositories to define a small subset of targets initially and grow as project complexity increases. Some target executions are limited to running the associated Earthly target and then immediately finishing. Other targets have additional logic that is executed after the target finishes running. -For example, the `publish` target will automatically publish container images generated by the target to any configured registries. For more information on supported targets, please see the [reference documentation](../reference/targets.md). ## Extending diff --git a/docs/src/concepts/images/pipeline_dark.png b/docs/src/concepts/images/pipeline_dark.png index 577857c..821278a 100644 Binary files a/docs/src/concepts/images/pipeline_dark.png and b/docs/src/concepts/images/pipeline_dark.png differ diff --git a/docs/src/concepts/images/pipeline_light.png b/docs/src/concepts/images/pipeline_light.png index 957c4a7..491f6e0 100644 Binary files a/docs/src/concepts/images/pipeline_light.png and b/docs/src/concepts/images/pipeline_light.png differ diff --git a/docs/src/concepts/projects.md b/docs/src/concepts/projects.md new file mode 100644 index 0000000..5590af3 --- /dev/null +++ b/docs/src/concepts/projects.md @@ -0,0 +1,60 @@ +# Projects + +The primary component of Catalyst Forge is the _project_. +Forge is designed to interact with monorepos, where each repository contains one or more projects. +A project can be classified as a discrete deliverable within a repository: it has own its dedicated process for validating, +building, testing, and potentially deploying. + +## Organizing Projects + +!!! tip + While there's no hierarchical order enforced by Forge, it's against best practices to have projects live at the root of a + repository. + The only exception to this case is where the repository only has a single deliverable. + Since the global blueprint at the root of a repository is always unified with project blueprints, trying to configure a project + in the global blueprint will result in overlapping values and will cause the parsing process to fail. + +Catalyst Forge does not enforce projects live in any particular folder within the repository. +Developers are encouraged to organize the repository in whatever way makes sense for them. +The discovery mechanisms used by Forge will ensure that projects are found no matter where they live. + +In some cases, projects may have dependencies on each other. +For example, one project may have a dependency on one or more projects that provide language libraries. +Whether or not these are treated as separate projects is up to developers. +If a library is used by more than one project, or is consumed externally, it's recommended to treat it as a separate project. + +## Project Components + +Forge discovers projects within a repository using a specific set of rules. +Namely, a valid project is any folder within the repository that contains a blueprint (`blueprint.cue`). +This is the _only_ requirement for forge to classify that directory as a project. +While a project may consist of one or more _other_ files or directories, the blueprint should always exist at the root of the +project folder. + +Optionally, a project may also contain an `Earthfile` that contains definitions for the common targets used by the +[CI system](./ci.md). +The CI system automatically checks for the existence of this file after it discovers a project. +However, it's important to recognize the existence of an `Earthfile` _does not_ define a project according to Forge. + +## Blueprints + +A blueprint file contains the configuration for a project. +By convention, the blueprint file is named `blueprint.cue` and is placed at the root of the project folder. +A blueprint contains several options for configuring a project. +Please refer to the [reference documentation](../reference/blueprint.md) for more information. + +In addition to project blueprint files, a _global_ blueprint file can also be provided at the root of the repository. +This blueprint configures global options that impact every project in the repository. +The final configuration always consists of a unification of the project and global blueprints. + +## Tagging + +When tagging a project, it's recommended to use the following format: + +``` +/ +``` + +Various systems within Forge are configured to automatically detect and parse this tag structure. +For example, the `tag` event when configuring releases only triggers when a tag matching the current project name is found. +This structure also ensures that projects are versioned separately and are easy to identify when examining tags. \ No newline at end of file diff --git a/docs/src/index.md b/docs/src/index.md index e9ed220..18d7583 100644 --- a/docs/src/index.md +++ b/docs/src/index.md @@ -16,4 +16,5 @@ It is especially inspired by the principles and philosophies of ## Getting Started -If you're new to Catalyst Forge, the quickest way to get familiar with what it offers is through the getting started tutorial. \ No newline at end of file +If you're new to Catalyst Forge, the quickest way to get familiar with it is through the +[getting started tutorial](./tutorials/getting_started.md). \ No newline at end of file diff --git a/docs/src/reference/blueprint.md b/docs/src/reference/blueprint.md new file mode 100644 index 0000000..a250940 --- /dev/null +++ b/docs/src/reference/blueprint.md @@ -0,0 +1,181 @@ +# Blueprints + +A blueprint file is the native configuration file format for Catalyst Forge. +They may appear in various places in a repository and are always denoted by their filename: `blueprint.cue`. + +## Language + +Blueprints are written in the [CUE](https://cuelang.org/) language (denoted by the `.cue` extension). +They are parsed by the CUE Go API at runtime and must meet all language restrictions. +Blueprint files can be parsed using the `cue` CLI, however, Forge provides a limited number of custom attributes that will not +be recognized by the CLI (see the [section below](#custom-attributes)). + +Blueprint files must be _concrete_ at runtime. +Meaning, all fields must have _known_ values when Forge parses the blueprint file. +In cases where attributes are used to insert dynamic data, the data must be available at runtime, otherwise Forge will fail to +compile the blueprint. + +### Custom Attributes + +Forge currently provides two custom [attributes](https://cuelang.org/docs/reference/spec/#attributes) that can be used in blueprint +files. + +#### Env + +The `env` attribute can be used to pull values from an environment variable at runtime. +When Forge parses a blueprint file, it scans for this attribute and uses the fields to dynamically insert data from the environment. +If the specified environment variable is unset, a value will _not_ be set. + +Example usage: + +```cue +version: "1.0.0" +project: { + name: "foo" + ci: targets: { + docker: { + args: { + foo: string | *"bar" @env(name="FOO",type="string") + } + } + } +} +``` + +In the above example, the `foo` field will be set with to value present in the `FOO` environment variable. +The value in the `FOO` environment variable must be a valid string. +If the `FOO` environment variable is unset, the `foo` field will default to `bar`. + +#### Forge + +The `forge` attribute can be used to access runtime data collected and provided by Forge. +Not all runtime data is guaranteed to be present as some is contextual. +The below table documents all runtime data available in the latest version of Forge: + +| Name | Description | Type | Context | +| ----------------- | ----------------------------------------- | ------ | ----------------------------------- | +| `GIT_COMMIT_HASH` | The commit hash for the current commit | string | Always available | +| `GIT_TAG` | The full name of the current git tag | string | Available when a git tag is present | +| `GIT_TAG_VERSION` | The version suffix of the current git tag | string | Available when a git tag is present | + +Example usage: + +!!! note + When using the `forge` attribute, if the target field has a different type than the one specified in the above table, the + blueprint will fail to validate. + +```cue +version: "1.0.0" +project: { + name: "foo" + ci: targets: { + docker: { + args: { + version: string | *"dev" @forge(name="GIT_TAG_VERSION") + } + } + } +} +``` + +In the above example, the `version` field will set to the version suffix of the current git tag (e.g., `v1.0.0` for +`project/v1.0.0`). +If no tag is detected, the value will default to `dev`. + +### Schema + +The schema for blueprint files is defined in both Go and CUE. +The latest schema can be found +[here](https://github.com/input-output-hk/catalyst-forge/blob/master/lib/blueprint/schema/_embed/schema.cue){:target="_blank"}. +Alternatively, the Go code responsible for generating the schema can be explored +[here](https://godocs.io/github.com/input-output-hk/catalyst-forge/lib/blueprint/schema){:target="_blank"}. + +Note that the schema is enforced at runtime. +Improperly named fields, fields not specified in the schema, or incorrect types on fields will all cause runtime errors in all +Forge systems. + +### Versioning + +!!! note + + Blueprints are versioned using [semantic versioning](https://semver.org/). + However, only the major and minor sections are used. + The patch section is ignored. + +Every blueprint must specify a `version` field at the top-level of the file. +This informs Forge what version of the schema is being used. +The version is [hard-coded](https://github.com/input-output-hk/catalyst-forge/blob/master/lib/blueprint/schema/version.cue) in +the source code and is bumped whenever a change to the schema is made. + +All systems within the Forge ecosystem have a dependency on this schema. +For exampe, the version embedded into the Forge CLI can be found by running: + +```shell +$ forge version +forge version v0.1.0 linux/amd64 +config schema version 1.0.0 +``` + +When a Forge system is processing a blueprint file, it first checks the version specified in the file against the version it was +compiled with. +It then uses the following rules to determine if it is safe to proceed: + +- The major versions match +- The minor version of the embedded schema is _greater than or equal to_ the minor verion specified in the blueprint + +In the case where the major versions mismatch, the tool will refuse to parse the blueprint file. +In the case where the minor version is less than the one specified in the blueprint, a warning is emitted but the blueprint file +will still be parsed. + +## Types + +There are two types of blueprint files: _project_ and _global_. + +### Project + +A project blueprint file is responsible for defining the configuration of a project. +By convention, it is located at the root of the project directory in a repository, usually next to an `Earthfile`. +All projects _must_ be accompanied by a project blueprint file. + +A project blueprint is usually denoted by the existence of the `project` field: + +```cue +version: "1.0.0" +project: { + name: "project" +} +``` + +There are several different options for configuring a project. +Please refer to the [schema](#schema) for an exhaustive list. + +### Global + +A global blueprint file is located at the root of the git repository and defines repository-wide configuration options. +Every time Forge runs on a project, it also searches for and unifies the global blueprint with the local project blueprint. +This ensures that every execution uses the configuration options defined in both. + +A global blueprint is usually denoted by the existence of the `global` field: + +```cue +version: "1.0.0" +global: repo: { + name: "my-org/my-repo" +} +``` + +There are several different global options available. +Please refer to the [schema](#schema) for an exhaustive list. + +## Loading and Discovery + +When a project is loaded, the following occurs: + +1. A `blueprint.cue` is searched for at the given project path and parsed/loaded. +2. A recursive upward search is performed to find the root of the git repository. +3. Once the git root is found, a `blueprint.cue` is searched for and parsed/loaded. +4. Both blueprint files are unified and used as the final configuration value. + +It is an error for a `blueprint.cue` to exist outside of a git repository. +If a git root cannot be found, then the blueprint loading process will fail. +A global blueprint is optional and not required, although many features of Forge rely on the configuration options it provides. diff --git a/docs/src/reference/deployments.md b/docs/src/reference/deployments.md new file mode 100644 index 0000000..81a58db --- /dev/null +++ b/docs/src/reference/deployments.md @@ -0,0 +1,112 @@ +# Deployments + +Every project can be configured to be automatically deployed to Catalyst's development cluster. +This allows previewing the behavior of a project after merging changes. + +## Background + +Deployments are opinionated in that they are configured to work with a certain tech stack. +Specifically, application deployments are written using [Timoni](https://timoni.sh/), which uses CUE as the configuration language. +Projects will generate a [bundle file](https://timoni.sh/concepts/#bundle) from the blueprint that deploys one or more +[modules](https://timoni.sh/concepts/#module). +This bundle file is written to the configured GitOps repository where it's expected that +[Argo CD](https://argo-cd.readthedocs.io/en/stable/) will be responsible for reconciling it to a Kubernetes cluster. + +## How it Works + +When the CI pipeline runs, Forge will automatically scan and identify all projects that contain a `deployment` configuration block. +At the end of the pipeline, after all releases have been executed, all projects with deployments will be deployed. +Forge will use the `deployment` block to generate a Timoni bundle file. +The GitOps repository (configured in `global.deployment`) is then cloned locally and the bundle file is placed in specific location +within the repository. +The changes are then committed and pushed back to the repository. +This will trigger the GitOps operator (Argo CD) to consume the bundle file, generating and applying Kubernetes resources to the +configured cluster. + +## Configuration + +The deployment behavior can be specified using the `deployment` block in the project configuration. +For example: + +```cue +project: { + deployment: { + environment: "dev" + modules: main: { + container: "foundry-api-deployment" + version: "0.1.0" + values: { + environment: name: "dev" + server: image: { + tag: _ @forge(name="GIT_COMMIT_HASH") + } + } + } + } +} +``` + +The following section will break down this example configuration. + +### Environment + +The `environment` field specifies which environment to deploy the project to. +In the current version of Forge, `dev` is the only allowed value in this field (and is also the default value). + +### Modules + +Deployments can consist of one or more modules, of which one module is designated as the _main_ module. +All modules share the same configuration fields: + +| Name | Description | Type | Required | Default | +| ----------- | --------------------------------------------------- | ------ | -------- | --------------------------- | +| `container` | The name of the container holding the Timoni module | string | no | `[project_name]-deployment` | +| `namespace` | The kubernetes namespace to deploy to | string | no | `default` | +| `values` | The configuration values to pass to the module | Object | no | `{}` | +| `version` | The version of the container to use | string | yes | N/A | + +#### Main Module + +The _main_ module is the Timoni module that is responsible for deploying the primary service managed by the project. +For example, an API server's main module would configure all of the necessary Kubernetes resources required for running the server +(like a deployment and service). + +#### Support Modules + +Support modules are modules that provide supplementary resources that the project may require. +For example, a project may need access to a database. +A support module can be configured to point to a Timoni module that will ensure a database is setup for the project to use. + +In the current version of Forge, this field is not used, as no support modules exist at the time of this writing. + +### Values + +Most Timoni modules require values to be passed in order to configure how the module goes about deploying the project. +For example, most modules need to know the tag of the container image that should be deployed. +In the previous example, this was set to `@forge(name="GIT_COMMIT_HASH")` which means the tag will always be set to the current +git commit hash. +This is the standard approach as most projects are configured with a [docker release](./releases/docker.md) that is set to +publish images using the git commit hash. + +There is no enforced schema for the `values` field as it depends on the module being consumed. +Refer to the documentation for a specific module to determine what fields are available for configuration. + +## Templating + +!!! note + Most modules are published to a private AWS ECR instance. + Because of this, you must be authenticated with AWS before trying to generate a deployment template. + This necessarily means external contributors will not be able to use this feature. + +When the deployment step in the CI activates, Forge automatically compiles the raw Kubernetes manifests from the Timoni bundle and +passes it to Argo CD for reconciliation. +It's possible to generate these manifests yourself using the `forge deploy template` command: + +``` +forge deploy template +``` + +Note that you _must_ have the [Timoni CLI](https://timoni.sh/install/) installed locally for this command to work. +Forge will automatically generate the Timoni bundle and then call the Timoni CLI to convert it to its raw YAML counterpart. +The resulting manifests will then be printed to `stdout`. +This is useful for troubleshooting a deployment as it allows you to examine exactly what is getting deployed to Kubernetes. \ No newline at end of file diff --git a/docs/src/reference/releases/docker.md b/docs/src/reference/releases/docker.md new file mode 100644 index 0000000..303ea1b --- /dev/null +++ b/docs/src/reference/releases/docker.md @@ -0,0 +1,67 @@ +# Docker Release + +!!! note + The Earthly target specified for this release _must_ accept two arguments: `container` and `tag`. + These arguments must then be used when saving the image at the end of the target: + + ```earthly + docker: + ARG container="image_name" + ARG tag="latest" + + SAVE IMAGE ${container}:${tag} + ``` + + Without this, Forge will be unable to locat the image generated by the target. + +The `docker` release type publishes the image generated by the release target to configured container registries. + +## Config + +!!! note + If the release is configured to trigger on a `tag` event, then the `tag` field will automatically be set to the semantic version + included in the git tag (overriding any value configured in the blueprint). + For example, a tag for project `foo` might look like: `foo/v1.0.0`. + When the release runs, it will automatically tag images using `v1.0.0` instead of the value specified in the `tag` field. + +| Field | Description | Type | Required | Default | +| ----- | --------------------------------- | ------ | -------- | ------- | +| `tag` | The tag to publish the image with | string | yes | N/A | + +## How it Works + +The `docker` release calls the configured Earthly target and expects a container image to be generated (using `SAVE IMAGE`). +The configured target should accept two arguments: `container` and `tag`. +The target is called and passed values for these arguments which should be used when saving the image (see note above). + +Once the target successfully completes, the local docker daemon is searched to ensure the expected image was created. +At this point, if there is no currently triggered event, the release will stop. + +In the case where a release event is firing, the produced image will then be retagged using any registries configured in +`global.ci.registries` as well as the `tag` specified in the configuration. +Each uniquely tagged image created by this process is then pushed. +Note that Forge assumes appropriate authentication has already been configured for private registries. +Underneath the hood the `docker` CLI is used, so any supported authentication formats should work. + +## Mutli-platform images + +It's possible to produce multi-platform images by specifying the `platforms` field in the target configuration. +For example: + +```cue +project: { + ci: targets: docker: { + platforms: ["linux/amd64", "linux/arm64"] + } +} +``` + +This will cause the target to produce two images: one for the `linux/amd64` platform and the other for the `linux/arm64` platform. +The `docker` release type will automatically detect multi-platform builds and adjust its strategy accordingly. +It starts by validating that the expected images for each platform are present in the local docker daemon. +It then uses the same tag strategy to retag and push these images. +The only difference is tags have the platform appended to the end (e.g., `image:tag_linux_amd64`). + +Once all images have been pushed, a final multi-platform image is created that drops the suffix (e.g., `image:tag`) and contains +entries pointing to each of the previously pushed images. +Docker clients that natively support multi-platform images will be able to pull down the correct architecture from this image. \ No newline at end of file diff --git a/docs/src/reference/releases/github.md b/docs/src/reference/releases/github.md new file mode 100644 index 0000000..c07febd --- /dev/null +++ b/docs/src/reference/releases/github.md @@ -0,0 +1,58 @@ +# Github Release + +The `github` release type creates a new GitHub release and uploads any artifacts produced from the configured target as release +assets. + +## Config + +| Field | Description | Type | Required | Default | +| -------- | ---------------------------------------------- | ------ | -------- | ------- | +| `name` | The name to use for the release | string | yes | N/A | +| `prefix` | The prefix to use for naming assets | string | yes | N/A | +| `token` | The GitHub token to use for creating a release | secret | yes | N/A | + +## How it Works + +The `github` release calls the configured Earthly target and expects artifacts to be generated (using `SAVE ARTIFACT`). +Any number of artifacts may be generated and they do not need to be saved with a specific name. +After the target successfully completes, the release will validate that at least one artifact was produced by the target. +It is an error for the target to produce zero artifacts. +At this point, if there is no currently triggered event, the release will stop. + +In the case where a release event is firing, all artifacts are archived and compressed in a `.tar.gz` file. +The file name consists of the configured `prefix` plus the current platform (e.g., `prefix-linux-amd64.tar.gz`). +In the case where multiple platforms were specified in the target configuration, an archive is created and uploaded for each +platform. + +Finally, the release uses the GitHub API (using the provided `token`) to create a new release and upload the artifacts gathered in +the previous step. +The name of the release is determined by the `name` field. + +## Authentication + +The `github` release needs a valid GitHub token with write permissions to the target repository. +The default CI pipeline run by Forge will ensure the generated `GITHUB_TOKEN` has sufficient write permissions. +The `token` field is of the `secret` type and matches the format of secrets used elsewhere in blueprints. +To use the `GITHUB_TOKEN` present in the current context, configure the `token` field as seen below: + +```cue +project: { + release: { + github: { + on: tag: {} + config: { + token: { + provider: "env" + path: "GITHUB_TOKEN" + } + } + } + } +} +``` + +## Triggering + +It's recommended to always use the `tag` event type when configuring the release's `on` field. +This is because the release expects a git tag to exist in the current context so that it can properly configure the GitHub release. +If the release fails to find a tag it will stop execution and fail. \ No newline at end of file diff --git a/docs/src/reference/releases/index.md b/docs/src/reference/releases/index.md new file mode 100644 index 0000000..84929d3 --- /dev/null +++ b/docs/src/reference/releases/index.md @@ -0,0 +1,67 @@ +# Releases + +Releases are the primary mechanism that allow declaring what release automation should occur for projects. +They are defined per project and are aggregated and run in parallel during the CI run. +Releases are optional and are not required to be configured. + +There are a growing number of release types, each fulfilling a unique purpose. +Documentation for each release type can be found on their respective pages in this category. +The rest of this page will common the features shared by all release types. + +## Configuration + +All types share a common set of configuration properies. +These properties are further detailed below. + +### `on` + +The `on` field allows specifying the events that will trigger a release to run in the CI pipeline. +The field is a map of event names to their respective configurations. +The event name is static and must match one of the supported event types. + +The `on` field only specifies when a _full release_ should be created. +When none of the conditions in the `on` field are satisfied, most release types will perform a "dry run" of the release. +This usually consists of running the release target and validating it produces the expected output. +Doing so helps prevent merging a potentially broken release configuration. + +The supported events are documented below. + +#### `merge` event + +| Field | Description | Type | Default | +| -------- | ---------------------- | ------ | ---------------------------------------- | +| `branch` | The target branch name | string | The value of `global.repo.defaultBranch` | + +The `merge` event triggers when Forge detects that the current git branch matches the target branch specified in the configuration. + +```cue +on: { + merge: { + branch: "foo" + } +} +``` + +In the above example, the release will trigger when the CI is run for the `foo` branch. +This event type is normally left at its default value to trigger releases when commits are merged to the `main` or `master` +branches. | + +#### `tag` event + +The `tag` event triggers when Forge detects a git tag in the current context. +This could be because `HEAD` is associated with a tag or a tag is picked up from the GitHub Actions environment. + +### `target` + +The `target` field specifies which Earthly target should be executed for this release. +Most releases require an artifact of some sort to be produced. +For example, the `docker` release expects a container image and the `github` release expects a set of files to upload. +Refer to the individual release type documentation for details on what it expects. + +By default, the `target` option will use the release event type as the name. +For example, the `docker` release type will default to calling a target named `docker` in the `Earthfile`. + +### `config` + +The `config` field specifies configuration that is custom to the release type. +For more information on how to configure a release type, please refer to the associated documentation. \ No newline at end of file diff --git a/docs/src/reference/targets.md b/docs/src/reference/targets.md index 44c788a..7639200 100644 --- a/docs/src/reference/targets.md +++ b/docs/src/reference/targets.md @@ -108,68 +108,6 @@ test: RUN go test ./... ``` -## Publish - -The `publish` target is used to generate the container for a project. -It is an error to define the target and not generate an image and the CI system expects only a single image. -The contents of the image are project-specific, but it's often the artifacts generated by the project in a deployable form. -For example, an API server may copy the server binary into a vanilla image and automatically set it as the default entrypoint. - -The `publish` target must expose two arguments: `container` and `image`. -These arguments are given values at runtime by the CI system. -The arguments should be used when saving the final image name and tag (see the example below). -Omitting these arguments will cause the CI system to fail to find the saved image. - -After the `publish` target is ran for a project, the CI system automatically publishes the resulting container image to all -configured container registries. -In the case where multiple platforms are specified for a project, the CI system will first upload each platform-specific image to -all registries and then create a single multi-platform manifest that points to the images. -Each image is tagged based on the tagging strategy defined in the blueprint global settings. - -### Example - -```earthly -publish: - FROM debian:bookworm-slim - - ARG container - ARG tag - - COPY +build/program program - - ENTRYPOINT ["/program"] - SAVE IMAGE ${container}:${tag} -``` - -## Release - -The `release` target is used for creating -[GitHub Releases](https://docs.github.com/en/repositories/releasing-projects-on-github/about-releases) for a project. -The target must produce at least one artifact. -The contents of the artifact are specific to the project. -For example, the artifact may be the binary file for a compiled project or a directory of assets for a frontend project. - -After the `release` target is ran for a project, the CI system will check if there's a git tag present in the current run context. -If a tag is found it will be compared against the current project to ensure it matches. -In the case of a match, the CI system will then create a new GitHub release using the git tag as the name. -Finally, the CI system will archive and compress the artifact produced by the `release` target and attach it as an asset to the -release. - -In the case where multiple platforms are specified for a project, the system will archive and compress each artifact separately. -The generated archive name will have the associated platform added as a suffix in the file name. -All archives will then be attached to the generated release as assets. - -### Example - -```earthly -release: - FROM scratch - - COPY +build/program program - - SAVE ARTIFACT program -``` - ## Docs !!! warning diff --git a/docs/src/tutorials/getting_started.md b/docs/src/tutorials/getting_started.md index a36e2c2..698cf75 100644 --- a/docs/src/tutorials/getting_started.md +++ b/docs/src/tutorials/getting_started.md @@ -23,107 +23,26 @@ Prior to starting this tutorial, please ensure you have the following available 1. The latest version of the [forge CLI](https://github.com/input-output-hk/catalyst-forge/releases) 2. A recent version of [Earthly](https://earthly.dev/) installed and configured -1. (Optional) The [`uv` Python tool](https://github.com/astral-sh/uv) - -If you would like to use a differnet Python package manager, or just use vanilla tooling, then you will need to adapt some of the -below steps for those specific tools. -This tutorial is focused on introducing Catalyst Forge and `uv` is only being used to simplify the tutorial. ## Project Setup To begin, clone the [catalyst-forge-playground](https://github.com/input-output-hk/catalyst-forge-playground) locally. -Make a new folder under the `users` directory in the root of the repository using your GitHub username. -Inside of your newly created folder, initialize a new Python project: - -```shell -uv init -``` - -From the same folder, add the `click` package by running: - -```shell -uv add click -``` - -Finally, add `black`, `ruff`, and `pytest`: - -```shell -uv add black pytest ruff --optional dev -``` - -Since we are creating a CLI, we will need to modify the structure of the project: +Next, copy the `examples/python` directory to a new directory under `users` using your GitHub username: ```shell -$ mkdir -p src/hello -$ mv hello.py src/hello/hello.py -$ touch src/hello/__init__.py +cp -r examples/python users/myusername ``` -And then modify our `pyproject.toml` to include the following: - -```toml -[project.scripts] -hello = "hello.hello:cli" - -[build-system] -requires = ["hatchling"] -build-backend = "hatchling.build" - -[tool.hatch.build.targets.wheel] -packages = ["src/hello"] -``` - -## Create CLI - -Modify the `src/hello/hello.py` file with the following code: - -```python -import click - -def hello(name: str = "World") -> str: - return f"Hello, {name}!" - -@click.command() -@click.argument('name', required=False, default="World") -def cli(name): - click.echo(hello(name)) - -if __name__ == '__main__': - cli() -``` - -Now add a simple test in `tests/hello_test.py`: - -```python -from hello.hello import hello - - -def test_hello(): - assert hello("Alice") == "Hello, Alice!" - assert hello() == "Hello, World!" -``` - -Validate everything works: - -```shell -$ uv run hello test -Hello, test! - -$ uv run pytest . -=========================================================== test session starts =========================================================== -platform linux -- Python 3.12.5, pytest-8.3.3, pluggy-1.5.0 -rootdir: /var/home/josh/work/catalyst-forge-playground/users/jmgilman -configfile: pyproject.toml -collected 2 items - -hello_test.py .. [100%] - -============================================================ 2 passed in 0.01s ============================================================ -``` +The example consists of a simple Python CLI that takes a single argument and prints: `"Hello, !` to the screen. +The project was created using the [uv](https://docs.astral.sh/uv/) CLI. +The Python package has already been configured to install itself as a script named `hello`. ## Creating an Earthfile -Catalyst Forge uses [Earthly](https://earthly.dev/) underneath the hood for building applications. +Catalyst Forge uses [Earthly](https://earthly.dev/) underneath the hood for creating the CI pipeline. +It's recommended you take time to become familiar with Earthly and run through their onboarding documentation. +In this tutorial we will create various Earthly "targets" that will be responsible for performing specific steps in the pipeline. + To begin, we will create a simple `Earthfile` in our folder that will validate our code for us: ```earthly @@ -184,6 +103,25 @@ During CI runs, the `forge` CLI is used to execute all Earthly targets. This ensures more consistency between running the target locally and when running it in CI, as the CLI adds several additional features that are not natively present in Earthly. +### Building + +We will build our CLI as a Python wheel to make distributing/publishing it easier. +To do this, add a new `build` target to the `Earthfile`: + +```earthfile +build: + FROM +src + + RUN uv build --wheel + + SAVE ARTIFACT dist dist +``` + +Like the previous target, the `build` target is also called by CI. +This ensures that the `build` can run successfully before calling additional targets that might rely on it (it also caches it). +In the above case, `uv` will build our wheel and place it in the `dist/` folder for us. +We save the entire folder as an artifact for later targets to use. + ### Testing For running our tests, we will add a `test` target to the `Earthfile`: @@ -202,32 +140,19 @@ Like the `check` target, the `test` target is also called by CI. It's intended to be used for running all tests in a project (including unit and integration tests). Since we're using `pytest`, we only need to call it for our simple unit test to run. -### Building - -We will build our CLI as a Python wheel to make distributing/publishing it easier. -To do this, add a new `build` target to the `Earthfile`: - -```earthfile -build: - FROM +src - - RUN uv build --wheel +### Containerizing - SAVE ARTIFACT dist dist -``` +We will now create our first "release" target. +These targets are special in that they each serve different purposes but generally fall into the "release" category. +Unlike the other targets, release targets are not required to be a specific name, although it's common to use the same name as the +release type. -Like the previous targets, the `build` target is also called by CI. -This ensures that the `build` can run successfully before calling additional targets that might rely on it (it also caches it). -In the above case, `uv` will build our wheel and place it in the `dist/` folder for us. -We save the entire folder as an artifact for later targets to use. - -### Publishing - -We will now create the container that will be responsible for running our CLI. -To do this, add a new `publish` target to the `Earthfile`: +We will start by creating our `docker` release target. +As the name suggests, this release type is responsible for building the container that will be published to configured registries. +To do this, add a new `docker` target to the `Earthfile`: ```earthfile -publish: +docker: FROM python:3.12-slim ARG container=hello @@ -242,30 +167,32 @@ publish: SAVE IMAGE ${container}:${tag} ``` -The target copies the wheel from the `build` target and then globally installs it into the container. +This target copies the wheel from the `build` target and then globally installs it into the container. Since we've configured a script entry in our `pyproject.toml`, the CLI can be run by executing `hello`. The CI will automatically pass in the `container` and `tag` arguments in order to set the container name. -These arguments are not optional and must be included in all `publish` targets. +These arguments are not optional and must be included in all docker release targets. Let's now build our image and make sure it works: ```shell -$ forge run +publish +$ forge run +docker $ docker run hello:latest test Hello, test! ``` -Like the other targets, the CI system will automatically run the `publish` target in the pipeline. -Unlike the other targets, the `publish` target is special in that the CI system expects it to produce a container image. -During certain contexts, the CI system will automatically tag and publish the image for us. +Unlike the other targets, all release targets are run in parallel in a single step of the CI pipeline. +The release targets are generally run towards the end of the CI pipeline after all projects have been built and validated. +We will configure the specifics of this release target in a later step. ### Releasing -As a last step, we will create a `release` target that will expose our wheel from the `build` target: +The second release target, and our final target in the tutorial, will be the `github` target. +This release type is responsible for building and uploading artifacts from our project into a new GitHub release. +Like the previous release type, we will add a `github` target to the Earthfile: ```earthfile -release: +github: FROM scratch COPY +build/dist dist @@ -273,8 +200,10 @@ release: SAVE ARTIFACT dist/* hello.whl ``` -Like the `publish` target, the `release` target is special and the CI system expects it to produce at least one artifact. -During certain contexts, the CI system will publish a release using any artifacts produced by the target. +This release type expects artifacts to be produced by the target. +All artifacts are archived, compressed, and uploaded as assets when a new release is created. + +In this case, we output our Python wheel as an artifact, which is itself a sort of archive. You can validate this behavior by running the following: ```shell @@ -303,7 +232,7 @@ Add a new file in the root of the project folder called `blueprint.cue` with the ```cue version: "1.0.0" project: { - name: "hello-jmgilman" + name: "hello-jmgilman" // Replace "jmgilman" with your GitHub username } ``` @@ -316,6 +245,45 @@ The only required field in a blueprint file is the project name (shown as `hello This name should be unique across all repositories and is used to distinguish the project in several area (i.e. the container name). There are many more useful fields exposed in a blueprint file that can be explored later in the documentation. +### Configuring Releases + +The last step before pushing our code is to configure our releases in the blueprint file. + +```cue +version: "1.0.0" +project: { + name: "hello-jmgilman" + release: { + docker: { + on: { + merge: {} + tag: {} + } + config: { + tag: _ @forge(name="GIT_COMMIT_HASH") + } + } + github: { + on: tag: {} + config: { + name: string | *"dev" @forge(name="GIT_TAG") + prefix: project.name + token: { + provider: "env" + path: "GITHUB_TOKEN" + } + } + } + } +} +``` + +We've configured two releases: `docker` and `github`. +The "type" of the release is the same as the name. +Each release has an `on` field that specifies when the release is run and a `config` field that specifies type-specific options for +the release. +To learn more about releases, see the appropriate section in the documentation. + ## Testing Locally Prior to pushing our changes in a PR, let's test the full CI pipeline locally: @@ -358,28 +326,28 @@ After the merge, a new GitHub Workflow will run for this particular commit. Allow the workflow to run to completion before proceeding. Once the workflow has completed, you'll notice a new package created for the GitHub Container Registry. -The package will have the same name as the name as defined in `project.name` and will, by default, have a tag created that matches -the commit hash of the merge commit. -Whenever a new PR is merged into the default branch, all projects with a publish target will have their respective container images -pushed to configured registries using the commit hash as the tag. +The package will have the same name as the one defined in `project.name` and, as we configured, will have a tag created that +matches the commit hash of the merge commit. +Since we configured the `docker` release to run on `merge` events (which defaults to merges to the main branch), it automatically +ran and published our container image. For the final step, we will tag the repository with an application specific tag: ```shell -git tag -a "examples//v1.0.0" -m "My first version" +git tag -a "hello-jmgilman/v1.0.0" -m "My first version" git push origin master ``` -This creates a new tag using the traditional "mono repo" style tag. -The prefix is the path to the project in the repository and the suffix is the semantic version for that project. +The convention used above is important: `/`. +By default, the `tag` event used in our `github` release type only triggers when a tag matching the project name is pushed. +This ensures that projects are all versioned separately in the repository. After creating and pushing this tag, a new workflow is created and the CI pipeline runs again. Allow the new workflow to run to completion before proceeding. -Once the workflow is completed, you'll notice that a new release is created on GitHub with the name set to our git tag. -The release will contain a single artifact which is the Python wheel created by our `release` target. +Once the workflow is completed, you'll notice that a new release is created on GitHub with the name set to our git tag (this is +what we configured the name to earlier). +The release will contain a single artifact which is the Python wheel created by our `github` release. Notice that no other projects had a release added. -This is the default behavior: only projects which "match" the mono repo tag will create a new release and upload assets. -All other projects will skip this step. ## Conclusion diff --git a/docs/src/tutorials/images/gha.png b/docs/src/tutorials/images/gha.png index 44da829..4e5b2ed 100644 Binary files a/docs/src/tutorials/images/gha.png and b/docs/src/tutorials/images/gha.png differ