[Improvement] Add a new data source, github-tagged-images-file, to automate retrieving images from a image-list file of a GitHub release

README.md

@@ -37,6 +37,7 @@ There is also a scheduled workflow called [Retrieve image tags](https://github.c
 
 - `github-releases`: This will use GitHub releases as source, excluding pre-releases. This can be used if you need to keep all tags from the configured images in sync with GitHub releases
 - `github-latest-release`: This will use the release on GitHub marked as `Latest release`. This can be used if you only want one release to be added that is marked as latest.
+- `github-tagged-images-file`: This will look up GitHub git repository tags, and find the list of images inside a specified file. The tag must have an associated release, with the pre-release flag unset. This can be used if your project maintains a list of images in a file, e.g., https://github.com/longhorn/longhorn/blob/master/deploy/longhorn-images.txt
 - `registry`: This will use the registry of the first image and look up available tags.
 - `helm-latest:helm-repo-fqdn`: This will add the helm-repo-fqdn, and use the latest version of configured Helm chart(s) (`helmCharts`) configured to extract the images. It uses `helm template` and `helm show values` to extract images. You can specify one ore more iterations of `helm template` by specifying one ore more `values` configurations to make sure all required images are extracted. If you want to block certain images from being extracted, you can use `imageDenylist` in the configuration. See example below.
 - `helm-oci`: This is the same as `helm-latest`, except you don't need to provide a repository but it will use the charts directly from the provided `helmCharts` (which should be formatted as `oci://hostname/chart`).
@@ -49,6 +50,9 @@ The current filters for tags are:
 - `latest`: Sorts the found tags numerically and returns only the latest tag
 - `latest_entry`: Returns the last found (newest) tag only (can be used when tags are not semver/cannot be sorted numerically)
 
+`github-tagged-images-file` specific options:
+- `imagesFilePath`: the path to the list of images inside a GitHub git repository
+
 Helm specific options:
 
 - `imageDenylist`: An array of images that will not be added (in case the image matching finds images that shouldn't be added as the automation only accounts for adding tags to existing images, not adding new images as they need to be approved first)
@@ -59,7 +63,7 @@ Helm specific options:
 
 See example configuration for `github-releases`, `github-latest-release` and `registry`:
 
-```
+```json
 {
   "vsphere-cpi": {
     "images": [
@@ -117,9 +121,20 @@ See example configuration for `github-releases`, `github-latest-release` and `re
 }
 ```
 
+See example configuration for `github-tagged-images-file`:
+```json
+{
+  "longhorn": {
+    "versionSource": "github-tagged-images-file:longhorn/longhorn",
+    "imagesFilePath": "deploy/longhorn-images.txt",
+    "versionConstraint": ">=1.4.0"
+  }
+}
+```
+
 See example configuration for `helm-latest:helm-repo-fqdn`:
 
-```
+```json
 {
   "cilium": {
     "versionSource": "helm-latest:https://helm.cilium.io",

retrieve-image-tags/config.json

@@ -86,6 +86,11 @@
     ],
     "versionSource": "github-latest-release:kubernetes-csi/node-driver-registrar"
   },
+  "longhorn": {
+    "imagesFilePath": "deploy/longhorn-images.txt",
+    "versionSource": "github-tagged-images-file:longhorn/longhorn",
+    "versionConstraint": ">=1.4.0"
+  },
   "cpi-release-manager": {
     "images": [
       "gcr.io/cloud-provider-vsphere/cpi/release/manager"

retrieve-image-tags/retrieve-image-tags.py

@@ -314,6 +314,21 @@ def _extract_unique_images_from_helm_values(repo_name, chart, devel=False, versi
                         if not semver.match(release.tag_name.removeprefix('v'), constraint):
                             continue
                     found_releases.append(release.tag_name)
+        case 'github-tagged-images-file':
+            repo = g.get_repo(repoString[1])
+            releases = repo.get_releases()
+            if 'versionConstraint' in values:
+                constraint = values['versionConstraint']
+            for release in releases:
+                if not release.prerelease and not release.draft and semver.VersionInfo.isvalid(release.tag_name.removeprefix('v')):
+                    if 'versionFilter' in values:
+                        if not re.search(values['versionFilter'], release.tag_name):
+                            continue
+                    if 'versionConstraint' in values:
+                        # Checking constraint
+                        if not semver.match(release.tag_name.removeprefix('v'), constraint):
+                            continue
+                    found_releases.append(release.tag_name)
         case 'github-latest-release':
             repo = g.get_repo(repoString[1])
             # Get all the releases used as source for the tag
@@ -360,6 +375,24 @@ def _extract_unique_images_from_helm_values(repo_name, chart, devel=False, versi
                     if not _image_tag_already_exist(filetext, splitted_image[0], splitted_image[1]):
                         alldict[key]['full_images'].append(full_image)
             alldict[key]['full_images'].sort()
+        case 'github-tagged-images-file':
+            alldict[key]['full_images'] = []
+            repo = g.get_repo(repoString[1])
+            images_file_path = values['imagesFilePath']
+            for tag in found_releases:
+                file_content = repo.get_contents(images_file_path, ref=tag)
+                for full_image in file_content.decoded_content.decode('utf-8').split('\n'):
+                    full_image = full_image.strip()
+                    if not full_image:
+                        continue
+                    full_image = full_image.removeprefix('docker.io/')
+                    if full_image.startswith('rancher/'):
+                        continue
+                    splitted_image = full_image.split(':')
+                    if len(splitted_image) == 2:
+                        if not _image_tag_already_exist(filetext, splitted_image[0], splitted_image[1]):
+                            alldict[key]['full_images'].append(full_image)
+            alldict[key]['full_images'].sort()
         case _:
             alldict[key]['images'] = []
             alldict[key]['tags'] = []