[Improvement] Add a new data source, github-tagged-images-file, to automate retrieving images from a image-list file of a GitHub release

[PhanLe1010]

Current issue:

Longhorn would like to automate adding/mirroring images. However, the currently available list of data sources don't fit the use-case of Longhorn:

1. github-releases data source: This one finds new GitHub release tag and adds the images defined in the images field in the config.json file. For example:

   {
     "vsphere-cpi": {
       "images": [
         "gcr.io/cloud-provider-vsphere/cpi/release/manager"
       ],
       "versionSource": "github-releases:kubernetes/cloud-provider-vsphere",
       "versionConstraint": ">1.21.0"
     }
   }

   This config.json file instructs the script to find GitHub release tags in the repo kubernetes/cloud-provider-vsphere. Then only add the gcr.io/cloud-provider-vsphere/cpi/release/manager image with the found tags to the images-list. This doesn't fit the use-case of Longhorn because the list of Longhorn images are not fixed. We added/removed images between the releases. Therefore, it would require manual works to modify the "images" fied of the config.json oftenly
2. github-latest-release data source: this data source has same limitation as the github-releases. Additionaly, Longhorn maintain multiple minor releases so a smaller version (e.g., v1.4.5) might be released after the current latest version (e.g., v1.5.3). This data source will not sync and add the smaller version (e.g., v1.4.5)
3. registry data source: Longhorn doesn't maintain a registry. Not applicable
4. helm-latest, helm-oci, and helm-directory data sources. With these data sources, the workflow attempts to run helm template and extract the images from the workload (deployment/daemonset/pod) output of helm template. This approach doesn't work for Longhorn because not all Longhorn images appears in the output of Helm template (the images of Longhorn system managed components)

Proposal

Add a new data source to automate retrieving images from file which contains the list of images of a GitHub release, github-releases-images-file. This will look up GitHub releases, excluding pre-releases, and find the list of images inside a specified file of the release. This can be used if your project maintains a list of images in a file, e.g., https://github.com/longhorn/longhorn/blob/master/deploy/longhorn-images.txt

An example of configuration for github-releases-images-file could be:

{
  "longhorn": {
    "versionSource": "github-releases-images-file:longhorn/longhorn",
    "imagesFilePath": "deploy/longhorn-images.txt",
    "versionConstraint": ">=1.4.0"
  }
}

With the new github-releases-images-file data source, the above config.json instructs the GitHub workflow to:

1. Look up GitHub releases at the repo longhorn/longhorn, excluding pre-releases
2. Only consider the releases which are >=1.4.0
3. For each release, download the image list at deploy/longhorn-images.txt and add the newly found images to the images-list.

[PhanLe1010]

cc @superseb

[brandond]

I like this idea, should it perhaps be named github-release-list-artifact, to indicate that the list comes from a release artifact?

[PhanLe1010]

Thanks @brandond for reviewing. Currently, Longhorn doesn't include images list in release's artifact. It stores the images list in a file in the source code of each release. Therefore, I think the datasource name github-releases-images-file might be a better fit. In the future, if some users have the images list in file in release's artifact, we can easily modify the code to for this use-case and the name github-releases-images-file still fits this use-case. WDYT?

[brandond]

Ah that's interesting. Most Rancher projects make the airgap image list a release artifact. This was a requirement for centralized image security scanning. Could LH consider doing the same?

[PhanLe1010]

This was a requirement for centralized image security scanning. Could LH consider doing the same?

I am not very sure about how Longhorn does image security scanning so not sure if this is something Longhorn would like to do. cc @innobead could you provide some thoughts?

Regarding to the data source naming, do you agree that github-releases-images-file sounds ok for both use-case (download from source code and download from release's artifact)? @brandond

[PhanLe1010]

ping @brandond @superseb Could we continue the review?

Longhorn 1.6.0 is about to release and we need this one soon

[brandond]

I still don't think that "release" is the correct term to use here. It is not pulling content from GitHub Release Artifacts. It is is getting files from at git repo (or a GitHub repo, specifically) at a specific tag. Could we update things it to reflect that?

[PhanLe1010]

Discussed with @brandond and we agree to change the name to github-tagged-images-file.
PR updated!

[brandond]

nits on the docs

[PhanLe1010]

Hi @brandond all checks have passed and you have approved the PR. Is it ok to merge now?