Improve CI Speed

[philippotto]

#7966 already implements some improvements. Additional ideas from here (quoting @daniel-wer):

• As you have already observed, the "Spin up environment" step takes the longest out of all steps (~3-4m). Looking into it, the time is spent downloading (Downloading: downloaded 13.01GiB, expanded to 14.54GiB, ratio 0.90, after 4m11.724810166s.). This is likely due to the docker layer caching (DLC) that is turned on. The caching should be worthwhile but one could still test to delete the docker layer cache and compare times without it (note that the DLC push at the end also takes 1-2m!). Also one could check whether the cache size inflated over time and whether a reset might shrink it back down again.
• It's a little hard to find information on that, but I think open source projects can make use of concurrently running jobs (the free plan includes 30 concurrent jobs). A fan out where the build steps are executed first and the frontend tests, backend linting, and end-to-end tests are then run in parallel should speed things up.
• For completeness sake (you have already prototyped this on another branch): The yarn cache restore and install takes rather long (50s + 30s). Using pnpm and/or installing without cache will/might improve the times.
• We should look into the other longer running steps
  • Can we speed up the frontend tests, for example by executing less tests serially? I have not found the option to print test times, but there's likely a way to identify the longest running tests.
  • The backend linting spends quite a bit of time compiling stuff, although there is a build step before. This might be expected.
  • The end-to-end tests are all ran serially. Maybe some groups can be executed in parallel? Also, the test database reset takes a considerable amount of time (multiple seconds) and is executed for each test file.

Also related: Skip some CI steps depending on diff to master #6523

[fm3]

Looks like the backend linter scapegoat runs clean first. It may help to do this step before the build task 🤔

Edit: tried that here https://app.circleci.com/pipelines/github/scalableminds/webknossos/18008/workflows/19b7c598-5689-4aca-a3be-4729ca70b9f1/jobs/29519 but did not help, the compile step compiled everything again 🤔

[philippotto]

If I'm reading this issue correctly, scapegoat needs to clean the target folder so the entire code can be linted because that happens during compilation 🤔 so, the code is compiled twice effectively. it probably would make sense to run scapegoat during the normal compilation. maybe this helps? scapegoat-scala/sbt-scapegoat#54
I'm not familiar with the sbt build process, so I can only guess..

[hotzenklotz]

Can we do a similar diff of the changes like in wk-libs? Or at least group frontend/backend steps? There is no need to build a WK docker image if I only do frontend changes and it crashes in formatting or unit testing.