Repo resource

Track changes in a repo project.

Source configuration

• url: Required. manifest repository location.

• revision: Optional. manifest branch or revision (use HEAD for default).

• name: Optional. initial manifest file. (use default.xml for default).

• private_key: Optional. Private key to use when pulling/pushing. Should be newline (\n) terminated. Note that keys requiring a passphrase are un-supported. Example:

  private_key: |
    -----BEGIN RSA PRIVATE KEY-----
    MIIEowIBAAKCAQEAtCS10/f7W7lkQaSgD/mVeaSOvSF9ql4hf/zfMwfVGgHWjj+W
    <Lots more text>
    DWiJL+OFeg9kawcUL6hQ8JeXPhlImG6RTUffma9+iGQyyBMCGd1l
    -----END RSA PRIVATE KEY-----

• depth: Optional. shallow clone with a history truncated to the specified number of commits. Defaults to full git clone for each project.

• jobs: Optional. number of jobs to run in parallel (default: 0; based on number of CPU cores) Reduce this if you observe network errors.

• rewrite: Optional. Any URL that starts with this value will be rewritten with the give value. Similar to git url insteadOf option. Example: rewrite http(s):// to git://

  rewrite:
    "http://": "git://"
    "https://": "git://"

• check_jobs: for check step only: number of jobs to run in parallel (default: jobs*2, 2 if jobs is undefined).

Example

Resource configuration for a public project using repo (Android)

resource_types:
- name: repo
  type: registry-image
  source:
    repository: mkorpershoek/repo-resource
    tag: v1.0.0

resources:
- name: aosp
  type: repo
  check_every: 2h # default of 1min seems too frequent
  source:
    url: https://android.googlesource.com/platform/manifest
    branch: main
    name: default.xml
    depth: 1 # use shallow clone for faster syncing
    jobs: 4  # run with -j4
    check_jobs: 6 # do 6 git ls-remote operations in parallel (check only)

Behavior

check: Check for new versions in each project under a manifest

Repo init and git ls-remote are called for a given manifest. After that we update the input manifest with the revisions retrieved from git ls-remote. The whole list of projects is returned as a "manifest version".

in: Syncs the repository, for a given manifest version

Repo syncs the repo to the destination, and locks it down to a given manifest version. It will return the same given ref as version.

out: No-op

Out is not implemented.

Contributing/development

Rebuilding the docker image

make

Publish to docker hub

make push

Unit testing

Use make test for running the unit(integration) tests:

make test

If you wish to run a single test, just edit the Dockerfile:

diff --git a/Dockerfile b/Dockerfile
index f02c512..031008e 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -28,6 +28,6 @@ FROM repo-resource AS tests
 COPY repo_resource /root/repo_resource
 COPY development/ssh/ /root/development/ssh/
 WORKDIR /root/
-CMD python -m unittest
+CMD python -m unittest repo_resource.test_check.TestCheck.test_branch_defaults_to_HEAD

Local concourse instance and docker registry.

It's also possible to use a local docker registry (instead of docker hub) for development. This allows closer "end-to-end" testing on a development workstation.

This section assumes that you've installed docker and docker-compose as documented in https://concourse-ci.org/getting-started.html

1. Start by running a concourse development instance and install fly:

   curl -O https://concourse-ci.org/docker-compose.yml
   docker-compose up -d
   
   curl 'http://localhost:8080/api/v1/cli?arch=amd64&platform=linux' -o fly \
       && chmod +x ./fly && mv ./fly /usr/local/bin/
   
   fly -t tutorial login -c http://localhost:8080 -u test -p test
   

   See https://concourse-ci.org/quick-start.html#docker-compose-concourse for details

2. Start your local docker registry:

   # retrieve the concourse container's network name
   container_name=$(docker ps --filter 'ancestor=concourse/concourse' --format "{{.Names}}")
   network_name=$(docker container inspect $name -f "{{.HostConfig.NetworkMode}}" $container_name)
   docker run --network ${network_name} -d -p 5000:5000 --name registry registry:2
   

   See https://www.docker.com/blog/how-to-use-your-own-registry-2/ for more information

3. Rebuild and publish your docker image to the local registry:

   make dev-push
   

4. Get the IP address of your registry container. This is needed because the concourse/concourse container seems unable to lookup by hostname address:

   registry_container_id=$(docker ps --no-trunc --filter "name=registry" --format="{{.ID}}")
   docker inspect --format='{{range .NetworkSettings.Networks}}{{println .IPAddress}}{{end}}' ${registry_container_id}
   

   Note the IP address (172.20.0.4 in this example).

5. Publish a development job with fly using the development pipeline.yml

   fly -t tutorial set-pipeline -p development -c development/pipeline.yml
   fly -t tutorial unpause-pipeline -p development
   fly -t tutorial check-resource -r development/aosp-tools
   

   Note: eventually, you'll have to patch development/pipeline.yml with the IP address from step 4.