Fragment LC tests

[MattWellie]

I did a lot of complaining about the LC tests, here's something productive:

I don't like the LC tests

• they don't actually test anything
• it's just a chain of method calls
• they take forever

Improvement

• I've run the LC tests locally, and stashed the outputs (i.e. when running the densify stage, I've kept a copy of the resulting dense.mt)
• I've split up the LC mono-test into a separate test per function/stage, with each stage using data structures committed in the repo, instead of generated from source from gVCF each time
  • these objects are all derived from the gVCFs, which are already committed, and open source GIAB anyway, so I'm assuming this is fine
• Those data structures (VDS, MT, HT) are tons of separate files, so I've committed just a compressed version of the directories, which is unpacked into the temp directory of each test when it's going to be used as input (helper method to do this)
• This opens up much more flexibility to actually load up the results from each stage and do some assertion on the contents, not just that it exists

Benefits

• Instead of running tests in series, we can run tests in parallel, so I've added pytest-xdist into setup.py and requirements-dev.txt, and set the tests to run in parallel using pytest -n auto. This wasn't useful before because 99% of the runtime was this single test, but now it's useful.
• This repo now contains the input fixtures to run the tests - those fixtures are the output of prior stages. We can do fun things like generate a dense.mt from the stored VDS and compare it to the stored dense.mt, which should be derived from the same source. Bonus points for digging into the contents of the various MTs and HTs and making sure the content is actually legit before doing those comparisons 😬

Hiccups

• the size of the zipped VDS is > 500kb, so I've added an exclusion to the check-added-large-files pre-commit hook
• start_query_context() can only be called once, so I've wrapped it in a try: except so whichever of the parallel processes gets there first creates it, and the rest gracefully fail
• when running tests in parallel, the pytest tmp_path directory generated in test_seqr_loader_dry is too long to be valid in UNIX, so I've inserted a handrolled tmp_path which creates a unique folder name, then deletes it upon test completion.
  • the tmp_path must have been close to the size limit anyway - pytest's tmp_path includes a bunch of unique character strings. This is then used as the base for the Hail tmp_path which loads on even more unique strings. The breaking point here was that when running multiple tests in parallel, the pytest tmp_path also references the worker ID and python instance being used.

Other

• removed logging.info(f'Using VEP {vep_version}'). It's unrelated to anything, but if the seqr_loader test fails it prints the logging to terminal, and this is logged SO MANY times.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 26.42%. Comparing base (8240817) to head (18f7700).

Additional details and impacted files

@@             Coverage Diff             @@
##             main     #947       +/-   ##
===========================================
- Coverage   78.42%   26.42%   -52.00%     
===========================================
  Files          10        9        -1     
  Lines        1794     1680      -114     
===========================================
- Hits         1407      444      -963     
- Misses        387     1236      +849     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Alexander-Stuckey]

Do we actually need to create a cohort for all these downstream tests? It's very hard (for me) to work out what's going on with the whole query_command rubbish and what actually need to be passed to the run method.

[MattWellie]

This is a fair point... I don't think so. Ideally all the interactions with cohorts/sequencing groups should be done in the Stages, so the code is run during the runtime when the workflow is initiated. The changes you made recently to combiner implement that logic. that makes the interface with the methods way easier (run QC on these samples, instead of find some samples and run QC on them), and makes testing way simpler.

From scraping off additional bits of code from this test suite, I've found another instance of get_multicohort() being called inside the child/job logic in SampleQC. I'd refactor this to take that info as input, instead of to be determined at runtime.

If you went a step further, the logic under test here usually takes no more than 2/3 distinct entries from configuration, often taking a default value. Those could easily be made into arguments, then the stages don't even need a live configuration.

[Alexander-Stuckey]

Looks good to me