Improvements to Container CD

[tylertitsworth]

Description

Update CD to follow DevOps best practices.

How it works

Given an example(s) (AudioQnA), this workflow creates jobs that perform container CI on that example, and allow other event-based workflows to curate a matrix of these examples.

These are the inputs for the composites:

Build

# .github/workflows/composite/docker-build/action.yml
inputs:
  example_dir:
    description: Directory with docker-compose.yaml to build
    required: true
    type: string
  env_overrides:
    description: Bash Env Variable Overrides in `KEY=VAL && KEY2=VAL2` format
    required: false
    type: string
  registry:
    description: Container Registry URL
    required: false
    default: 'opea-project'
    type: string

Scan

# .github/workflows/composite/scan/action.yml
inputs:
  image-ref:
    description: 'image reference(for backward compatibility)'
    required: true
  output:
    description: 'writes results to a file with the specified file name'
    required: true

This is the inputs for the reusable workflow:

Container CI

# .github/workflows/container.ci.yaml
inputs:
  example_dir:
    required: true
    type: string
  scan:
    default: true
    required: false
    type: boolean
  test:
    default: true
    required: false
    type: boolean
  publish:
    default: false
    required: false
    type: boolean

These are the Events that this PR supports:

# .github/workflows/example-weekly-ci.yaml
on:
  # weekly
  schedule:
  - cron: "0 0 * * 0"
  # run all examples manually
  workflow_dispatch: null
  push:
    tags: ['**'] # whenever a tag is created/modified

# ./github/workflows/example-manual-ci.yaml
on:
  workflow_dispatch:
    inputs:
      examples:
        default: 'example1, example2'
        description: 'List of comma-separated examples to test'
        required: true
        type: string
      scan:
        default: true
        description: 'Scan all examples with Trivy'
        required: false
        type: boolean
      test:
        default: true
        description: 'Test Examples'
        required: false
        type: boolean
      publish:
        default: false
        description: 'Publish Images to Dockerhub'
        required: false
        type: boolean

Changes

• Add StepSecurity network traffic monitoring during workflows via harden-runner
• Tests labels are determined based on the test workflow filename:

# .github/workflows/container-ci.yaml
- name: Get Tests
  id: test-matrix
  run: |
    echo "matrix=$(find ${{ inputs.example_dir }}/tests -type f -name 'test_*.sh' -print | \
    sed -n 's|.*/\(test_[^_]*_on_\([^\.]*\)\).sh|\{"test": "\1", "hardware_label": "\2"}|p' | \
    jq -sc .)" >> $GITHUB_OUTPUT
  # ex: [{"test":"test_audioqna_on_xeon","hardware_label":"xeon"},{"test":"test_audioqna_on_gaudi","hardware_label":"gaudi"}]
...
runs-on: ${{ matrix.tests.hardware_label }}
...
- name: Run Test
  run: bash ${{ matrix.tests.test }}.sh
  env:
    REGISTRY: ${{ secrets.REGISTRY }}

• Simplify developer workflow:

# Before
docker build -t container1
docker build -t container2
docker build -t container3
...
# After
docker compose build

• Test only the latest version of that example's container
  • We pull the last known good version of the comps, and test the changes made in the example only.
  • Modifying the test file or docker-compose.yaml file for a given test can allow for testing specific versions of comp containers. No need to modify CI.
• status-check flag in pull_request events that verifies if any job in example-ci.yaml failed.

Issues

n/a

Type of change

List the type of change like below. Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds new functionality)
• Breaking change (fix or feature that would break existing design and interface)

Dependencies

n/a

Tests

I don't have the runners required to do the validation portion, however when previously testing with ubuntu-latest I was able to launch the test script from the correct directory on that runner before it ran out of space.
https://github.com/tylertitsworth/GenAIExamples/actions/runs/9896085740?pr=3

[claynerobison]

All I see is the minor doc change. other than that, seems like an elegant architecture improvement.

[chensuyue]

How does this new structure combined with current compose and k8s functionality test? Our test machine are not able to access an internal docker registry, so we have 2 local registry for CI test. We need to sync about the design.

[ashahba]

Thanks @tylertitsworth for the PR.
My change requests are mostly minor except for introducing dependency on ai-containers/.github@main which needs to be reviewed by the community.

[ashahba]

there may be a few more file types and extensions:
*.dockerfile, *Dockerfile*, *.cjs, *.svelte and etc

[tylertitsworth]

How does this new structure combined with current compose and k8s functionality test? Our test machine are not able to access an internal docker registry, so we have 2 local registry for CI test. We need to sync about the design.

@chensuyue the current compose structure could be merged with this design. It's all about having a consistent file system and making sure your dependencies are in subfolders so you can accurately track changes to individual containers.

Regarding the local CI, we can change the runner labels and just make sure to use the correct secrets.

Regarding k8s, I believe we previously spoke about how to best test helm charts. We have a composite action for this. However I see a lot of manifests so it might be better to utilize the baremetal test runner mode to call your existing test scripts to validate those manifests. If I were smarter I would've made a branch rather than a fork so I could actually test these things, it's hard to integrate GitHub CI without maintainer access.

[tylertitsworth]

My change requests are mostly minor except for introducing dependency on ai-containers/.github@main which needs to be reviewed by the community.

Thanks @ashahba, it's common practice to re-use GitHub actions from other locations. I hope that the community will accept another intel repository as a trusted and maintained source similar to how they trust GitHub, docker, helm, pre-commit.ci, and other open source tools already in use.

If the fear of a dependency failing is important. These actions can be pinned at a specific SHA to avoid failures like I did with many other actions introduced by this workflow.

[chensuyue]

How to use a common private registry on IDC machine? Now we have 2 local registry separately for xeon and gaudi cluster.

[tylertitsworth]

If you can use something like harbor that can be accessed by both clusters or a cloud registry that would be ideal.

If your runners have pre-authenticated access to these local registries, then we could remove the login commands/secrets and use environment variables that are present in the runner environment.