Convert pipelines to obsah

bin/forklift

@@ -0,0 +1,61 @@
+#!/usr/bin/env python3
+# PYTHON_ARGCOMPLETE_OK
+
+from typing import List
+from pathlib import Path
+
+import obsah
+
+
+class ApplicationConfig(obsah.ApplicationConfig):
+    """
+    A class describing the where to find various files
+    """
+
+    @staticmethod
+    def name() -> str:
+        """
+        Return the name as shown to the user in the ArgumentParser
+        """
+        return 'forklift'
+
+    @staticmethod
+    def target_name() -> str:
+        """
+        Return the name of the target in the playbook if the playbook takes a parameter.
+        """
+        return 'pipeline'
+
+    @staticmethod
+    def data_path() -> Path:
+        """
+        Returns the root of the project
+        """
+        return Path(__file__).absolute().parent.parent
+
+    @classmethod
+    def playbooks_path(cls: obsah.ApplicationConfig) -> str:
+        """
+        Return the data path. Houses playbooks and configs.
+        """
+        return (cls.data_path() / 'pipelines').as_posix()
+
+    @classmethod
+    def playbooks(cls: obsah.ApplicationConfig) -> List[obsah.Playbook]:
+        """
+        Return all playbooks in the playbook path.
+        """
+        paths = Path(cls.playbooks_path()).glob('*/*.yaml')
+        return sorted(obsah.Playbook(playbook_path.as_posix(), cls) for playbook_path in paths
+                      if playbook_path.stem == playbook_path.parent.name)
+
+    @classmethod
+    def inventory_path(cls) -> str:
+        """
+        Return the inventory path
+        """
+        return (cls.data_path() / 'inventories').as_posix()
+
+
+if __name__ == '__main__':
+    obsah.main(application_config=ApplicationConfig)

docs/testing.md

@@ -68,57 +68,63 @@ _Note_: Bats tests are not idempotent, so you may have to do some cleanup or ski
 
 ## Pipeline Testing
 
-Under `pipelines` are a series of playbooks designed around testing scenarios for various version of the Foreman and Katello stack. To run one:
+Under `pipelines` are a series of playbooks designed around testing scenarios for various version of the Foreman and Katello stack.
+Use `./bin/forklift --help` to find out which ones and their aguments.
 
-    ansible-playbook pipelines/<pipeline>.yml -e forklift_state=up -e <vars required by pipeline>
+```console
+$ ./bin/forklift --help
+usage: forklift [-h] action ...
 
-When you are finished with the test, you can tear down the associated infrastructure:
+positional arguments:
+  action      which action to execute
+    install   Run an install pipeline
+    upgrade   Run an upgrade pipeline
 
-    ansible-playbook pipelines/<pipeline>.yml -e forklift_state=destroy -e <vars required by pipeline>
-
-### Existing Pipelines
-
-* `install_pipeline` - Installs a Server and a Proxy VMs and runs the `foreman_testing` role to verify the setup.
-  Expects the `pipeline_os` variable to be set to a known OS (currently: centos7, debian10)
-  Expects the `pipeline_type` variable to be set to a known type (currently: foreman, katello, luna)
-  Expects the `pipeline_version` variable to be set to a known version (currently: 3.8, 3.9, 3.10, 3.11, nightly)
-* `upgrade_pipeline` - Installs a VM, upgrades it twice and runs the `foreman_testing` role to verify the final upgrade.
-  Expects the `pipeline_os` variable to be set to a known OS (currently: centos7, debian10)
-  Expects the `pipeline_type` variable to be set to a known type (currently: foreman, katello, luna)
-  Expects the `pipeline_version` variable to be set to a known version (currently: 3.8, 3.9, 3.10, 3.11, nightly).
+optional arguments:
+  -h, --help  show this help message and exit
+```
 
-#### Examples
+Individual pipelines also have help texts:
+```console
+$ ./bin/forklift install --help
+usage: forklift install [-h] [-v] [-e EXTRA_VARS] [--state FORKLIFT_STATE] [--os PIPELINE_OS] [--type PIPELINE_TYPE] [--version PIPELINE_VERSION]
+
+Run an install pipeline
+
+options:
+  -h, --help            show this help message and exit
+  -v, --verbose         verbose output
+  --state FORKLIFT_STATE
+                        Forklift state to ensure
+  --os PIPELINE_OS      Operating system to install, like centos8-stream, debian11 or ubuntu2004. Valid options depend on the pipeline
+  --type PIPELINE_TYPE  Type of pipeline, like foreman, katello or luna
+  --version PIPELINE_VERSION
+                        Version to install, like nightly, 3.7 or 4.9
+
+advanced arguments:
+  -e EXTRA_VARS, --extra-vars EXTRA_VARS
+                        set additional variables as key=value or YAML/JSON, if filename prepend with @
+```
 
-    ansible-playbook pipelines/install_pipeline.yml -e forklift_state=up -e pipeline_os=debian10 -e pipeline_type=foreman -e pipeline_version=nightly
-    ansible-playbook pipelines/upgrade_pipeline.yml -e forklift_state=up -e pipeline_os=centos7 -e pipeline_type=katello -e pipeline_version=3.10
+Pipelines typically have a state which defaults to `up`. Other valid values are `rebuild` and `destroy`. The latter one is useful to clean up which pipelines don't do by themselves.
 
-### Creating Pipelines
+For example to run a Foreman Nightly installation pipeline on Debian Bullseye:
 
-If you wish to add a new version of an existing pipeline (e.g. a new Katello release), you only have to add the corresponding vars files to `pipelines/vars/`.
+```console
+$ ./bin/forklift install --os debian11 --type foreman --version nightly
+... lots of output
+```
 
-For Katello 3.11, you'd be adding the following two files:
+When you're done, you can delete the boxes by adding `--state destroy`:
 
-`pipelines/vars/katello_3.11.yml`:
-```yaml
-forklift_name: pipeline-katello-3.11
-forklift_boxes:
-  pipeline-katello-3.11-centos7:
-    box: centos7
-    memory: 8192
-  pipeline-proxy-3.11-centos7:
-    box: centos7
-    memory: 3072
-katello_repositories_version: '3.11'
-katello_repositories_pulp_version: '2.19'
-foreman_repositories_version: '1.21'
-foreman_client_repositories_version: "{{ foreman_repositories_version }}"
+```console
+$ ./bin/forklift install --os debian11 --type foreman --version nightly --state destroy
 ```
 
-`pipelines/vars/katello_upgrade_3.11.yml`:
-```yaml
-katello_version_start: '3.9'
-katello_version_intermediate: '3.10'
-katello_version_final: '{{ katello_version }}'
+Similarly a Katello Nightly upgrade pipeline on CentOS 8 Stream:
+
+```console
+$ ./bin/forklift upgrade --os centos8-stream --type katello --version nightly
 ```
 
 ## Running Robottelo Tests

pipelines/install/install.yaml

@@ -0,0 +1,7 @@
+---
+- import_playbook: 01-boxes.yml
+- import_playbook: 02-setup.yml
+- import_playbook: 03-install_server.yml
+- import_playbook: 04-install_proxy.yml
+- import_playbook: 05-tests.yml
+- import_playbook: 06-smoker.yml

pipelines/install/metadata.obsah.yaml

@@ -0,0 +1,17 @@
+help: |
+  Run an install pipeline
+variables:
+  forklift_state:
+    parameter: --state
+    help: Forklift state to ensure
+    # TODO: choices: up/rebuild/destroy
+  pipeline_os:
+    parameter: --os
+    help: Operating system to install, like centos8-stream, debian11 or ubuntu2004. Valid options depend on the pipeline
+  pipeline_type:
+    parameter: --type
+    help: Type of pipeline, like foreman, katello or luna
+    # TODO: choices: foreman/katello/luna
+  pipeline_version:
+    parameter: --version
+    help: Version to install, like nightly, 3.7 or 4.9

pipelines/pulpcore/metadata.obsah.yaml

@@ -0,0 +1,14 @@
+help: |
+  Run a Pulpcore pipeline
+variables:
+  forklift_state:
+    parameter: --state
+    help: Forklift state to ensure
+    # TODO: up/rebuild/destroy
+  pipeline_os:
+    parameter: --os
+    help: Operating system to install, like centos7, debian10 or ubuntu1804
+  pipeline_type:
+    parameter: --type
+    help: Type of pipeline, like foreman, katello or luna
+    # TODO: choices: foreman/katello/luna

pipelines/pulpcore/pulpcore.yaml

@@ -0,0 +1,3 @@
+- import_playbook: 01-boxes.yml
+- import_playbook: 02-install.yml
+- import_playbook: 03-tests.yml

pipelines/upgrade/metadata.obsah.yaml

@@ -0,0 +1,18 @@
+help: |
+  Run an upgrade pipeline
+variables:
+  forklift_state:
+    parameter: --state
+    help: Forklift state to ensure
+    # TODO: up/rebuild/destroy
+  pipeline_os:
+    parameter: --os
+    help: Operating system to install, like centos7, debian10 or ubuntu1804
+  pipeline_type:
+    parameter: --type
+    help: Type of pipeline, like foreman, katello or luna
+    # TODO: choices: foreman/katello/luna
+  pipeline_version:
+    parameter: --version
+    help: Version to install, like nightly, 1.24 or 3.14
+

pipelines/upgrade/upgrade.yaml

@@ -0,0 +1,12 @@
+---
+- import_playbook: 01-boxes.yml
+- import_playbook: 02-setup.yml
+- import_playbook: 03-install_server.yml
+- import_playbook: 04-install_proxy.yml
+- import_playbook: 05-server_to_intermediate.yml
+- import_playbook: 06-proxy_to_intermediate.yml
+- import_playbook: 07-server_to_final.yml
+- import_playbook: 08-tests.yml
+- import_playbook: 09-proxy_to_final.yml
+- import_playbook: 10-tests.yml
+- import_playbook: 11-smoker.yml

pipelines/vars/install_base.yml

@@ -1,4 +1,5 @@
 pipeline_action: install
+forklift_directory: "{{ playbook_dir | dirname | dirname }}"
 forklift_name: "pipe-{{ pipeline_type }}-{{ pipeline_version }}-{{ pipeline_os }}"
 forklift_server_name: "pipe-{{ pipeline_type }}-server-{{ pipeline_version }}-{{ pipeline_os }}"
 forklift_proxy_name: "pipe-{{ pipeline_type }}-proxy-{{ pipeline_version }}-{{ pipeline_os }}"

pipelines/vars/upgrade_base.yml

@@ -1,4 +1,5 @@
 pipeline_action: upgrade
+forklift_directory: "{{ playbook_dir | dirname | dirname }}"
 forklift_name: "pipe-upgrade-{{ pipeline_type }}-{{ pipeline_version}}-{{ pipeline_os }}"
 forklift_server_name: "pipe-up-{{ pipeline_type }}-{{ pipeline_version}}-{{ pipeline_os }}"
 forklift_proxy_name: "pipe-up-{{ pipeline_type }}-proxy-{{ pipeline_version}}-{{ pipeline_os }}"

roles/forklift_versions/defaults/main.yml

@@ -1,2 +1,4 @@
-forklift_versions_file: "{{ lookup('env', 'PWD') }}/vagrant/config/versions.yaml"
+---
+forklift_directory: "{{ playbook_dir | dirname }}"
+forklift_versions_file: "{{ forklift_directory }}/vagrant/config/versions.yaml"
 upgrade: False