Add rename_embedded_ids.py script that updates @RG SM:{id} values

[jmarshall]

Add a fairly simpleminded script that reads CRAM files from a gs://… URL, applies samtools reheader to update the @RG SM:{id} headers as instructed, and writes the CRAM file and associated index file back to new gs://… URLs constructed from the old one by updating the {id} similarly.

Fortunately the samtools bug I ran into (samtools/samtools#1866) only affects CRAM v2.1 files, which were superseded by v3.0 in 2015 so we surely don't have any — so this streaming method will work after all. (It was just bad luck that I grabbed one when I was testing this locally!)

It's invoked as

Copies CRAM files and associated .crai index files to URLs based on new IDs,
also replacing SM:{id} values on the CRAM files' @RG header lines.
Each command-line argument is

    OLDURL[,OLDID],NEWID

If OLDID is not specified, a pattern matching /CPGdd...dd/ is used instead
to match CPG-style IDs in the URL and @RG SM fields.

This script should usually be run via analysis-runner. It requires an image
that contains samtools.

[jmarshall]

The changes to create_test_subset.py are somewhat draft and need testing. The rename_embedded_ids.py UI may need to become a little more general to handle all the scenarios we want it to work in.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 79.79%. Comparing base (f72850b) to head (06f8e1b).

Additional details and impacted files

@@           Coverage Diff           @@
##              dev     #504   +/-   ##
=======================================
  Coverage   79.79%   79.79%           
=======================================
  Files         169      169           
  Lines       14523    14523           
=======================================
  Hits        11589    11589           
  Misses       2934     2934           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[cassimons]

LGTM thanks.

[vivbak]

Looks good, John!

As we discussed, I think it would be good to call metamist here to make sure that our analysis entries don't go out of sync when this script is run (which could have downstream implications in pipeline runs).

Also it would be useful to add a comment on manual actions from a data management perspective (i.e. deleting the old cram files after this script is run).

[vivbak]

Oh also could you look at the linting and unit tests?

[jmarshall]

Turns out the unit test failure was a problem on dev that has since been fixed and has resolved itself with a rebase onto current dev.

Lint was whining about something I consider good style, sigh. But I'll probably refactor it in due course to shut the thing up.