Parallel evaluation

[danpalmer]

• Layer Linter version: latest
• Python version: 3.6
• Operating System: macOS

Description

Layer linter takes a while to run on a moderately sized codebase. This is understandable as there's a lot to do, but I noticed that it only maxes out a single logical core. Multi-threaded execution could provide a significant speed boost in the same way that it does for tools like flake8.

What I Did

Ran layer-lint with a single contract covering 400 packages with 14 layers. Time on my MBP is 67 seconds.

[seddonym]

Thanks Dan - I'll definitely consider this. I also have a feeling that the layer checking algorithm isn't optimal.

Out of interest, is your main pain point the duration of each test run per se, or slow feedback when editing/adjusting the layers.yml? If it's the second, probably an easier first step to making things quicker would be a way of caching/reusing the dependency graph.

[danpalmer]

Out of interest, is your main pain point the duration of each test run per se, or slow feedback when editing/adjusting the layers.yml? If it's the second, probably an easier first step to making things quicker would be a way of caching/reusing the dependency graph.

Right now it's the second, because I'm just testing this on my machine. If we put this into CI the second would be my concern as 1 minute of build time is quite a lot. We do parallelise parts of the build already, but a 1 minute (or likely 2+ mins with the final config) phase of the build is still quite long.

I would guess that parallelising might be straightforward? Parallelism per package per contract seems like the way to go? (Not to say it's not significant work, but I'd guess the structure lends itself well to parallelising).

[seddonym]

Thanks for these comments. I don't actually know what takes longer, building the graph or analysing the layers, so a bit of benchmarking would be the next step.

[seddonym]

Apologies, I closed this unintentionally. Reopening.