staging --> master #1379

ekiernan · 2024-10-02T19:53:08Z

staging --> master

* pin all latest docker version * run with h5ad isntead o floom * run with h5ad isntead o floom * run with h5ad isntead o floom * run with h5ad isntead o floom * new docker * merge h5ad files * new path * redo h5ad merge * add verify task * add verify task * add verify task * no more loom * changing loom to h5ad * try updating all instances of warp tools * Updated pipeline_versions.txt with all pipeline version information * test out markdown * formatting * formatting again * formatting again * changelogs and add SingleNucleusSmartSeq2LoomOutput back in * Updated pipeline_versions.txt with all pipeline version information * changelogs * Updated pipeline_versions.txt with all pipeline version information --------- Co-authored-by: GitHub Action <action@github.com>

* Update Multiome, ATAC, and Optimus readmes

* updates to BGE pipeline from GVS tieout fix string * changelogs * Updated pipeline_versions.txt with all pipeline version information * updated JointGenotyping changelog * Update changelog docs Co-authored-by: ekiernan <55763654+ekiernan@users.noreply.github.com> --------- Co-authored-by: GitHub Action <action@github.com> Co-authored-by: Nikelle Petrillo <38223776+nikellepetrillo@users.noreply.github.com> Co-authored-by: ekiernan <55763654+ekiernan@users.noreply.github.com>

Added deprecation notices for deprecated pipelines

* pin all latest docker version * loom to h5ad to fix womtools check * Update ValidateSmartSeq2SingleNucleus.wdl

* Added parameter to avoid non-determinism in DRAGEN aligner

adding parameters to handle trailing slash

Added doublet scores to gex h5ads, created new nhash for ATAC

dev --> staging

github-actions · 2024-10-02T19:53:19Z

DO NOT SQUASH MERGE!!!

github-actions · 2024-10-02T19:53:29Z

🔍Version Validation Results:

basename: missing operand
Try 'basename --help' for more information.
Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
All WDLs and changelog files appear to be valid for this release.

github-actions · 2024-10-02T19:53:29Z

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/master':
All changelog files are valid for this release.

* Making sure temporary credentials or keys don't get pushed by accident * Adding a GHA that can authenticate using a service account in GCP * Plugging in the magical auth system to finally connect to Terra * Adding quotes in vars. Disabling PR trigger to avoid duplicated runs for now. * Fixing newline. Adding debug lines * Debugging why the access token is not being accepted * Adding time to token validity. Maybe it expired before getting used * Removing printings * retrigger * Retrigger * Trying out a different token * Setting back to access token * try setting scopes directly * Removing debug flag for cleaner output. * Adding getting submission list * Changing to warp tests * Switching back * Adding Firecloud API script * Trying out new py class * Changing variables to get this pipeline to run automatically * Plugging in actual values to test this. Increasing polling intervals * Fixing path mistake * Commenting out test code. Fixing error with memoryRetryMultiplier * Returning workflowID * Setting to unbuffered so we see printings from py * Adding functionality to handle multiple workflows in a single run. * Trying to parse the monitorSubmission properly. * Trying to parse the monitorSubmission properly. * Trying to parse the monitorSubmission properly. * Trying to parse the monitorSubmission properly. * It looks like python printings messes up bash inputs. Cleaning it up. * Creating a map of workflows and their statuses. * Creating a map of workflows and their statuses. * Adding summary printings * Parameterizing call_cache * Adding a summary in the event of a failure. * Better summary printing. Covering both success and failure. * Removing stderr redirection to see where things are failing * Finally found what was wrong. Token was expiring before finishing the polling step. Setting it back to the default, 1 hour. * Parsing the outputs dictionary properly. * One more try with the parsing of outputs. Adding an id to pipeline run step * Handling int to str in json * Better summary formatting * Better summary formatting * Some icons. * Adding note to make sure folks don't change the magic string * Adding proper triggers * Removing test trigger --------- Co-authored-by: Jack Warren <jack@jackwarren.info>

ekiernan · 2024-10-09T15:08:24Z

Rerunning Multiome with atac_nhash_id here: https://gotc-jenkins.dsp-techops.broadinstitute.org/job/warp-workflow-tests/85638/

Multiome looks good; similar chages as paired-tag and Optimus. The h5ad files differed by 3 UMIs, so dove into comparisons with the following:

>>> umis_1 = np.array(truth.X.sum(axis=1)).flatten()
>>> umis_2 = np.array(test.X.sum(axis=1)).flatten()
>>> umis_equal = np.array_equal(umis_ad1, umis_ad2)
>>> umis_equal = np.array_equal(umis_1, umis_2)
>>> umis_equal
True

ekiernan · 2024-10-09T17:59:53Z

Optimus tests:
8k pbmc stranded sample: X.sum differs by 2 UMIs, gene metrics flagged, but sums match when tested manually.

8k pbmc unstranded sample: Same errors we saw for stranded sample. Checked X.sums manually for cells and genes alone and the sums matched.

Example:

>>> umis_ad1 = np.array(test.X.sum(axis=0)).flatten()
>>> umis_ad2 = np.array(truth.X.sum(axis=0)).flatten()
>>> umis_equal = np.array_equal(umis_ad1, umis_ad2)

Paired-tag tests:
10k pbmc example looks good, has the same changes as the optimus examples above. Library metrics look good.
10k_PBMC_BC011_gex.h5ad also has same changes. Library metrics look good.

jessicaway · 2024-10-09T18:13:37Z

@broadinstitute/warp-admins (just testing)

ekiernan · 2024-10-09T19:12:23Z

Joint genotyping:
dev_bge failed. see gs://broad-gotc-dev-cromwell-execution/TestJointGenotyping/e6d98c0e-740b-428c-8167-4a155570faed/

added atac_nhash_id to multiome sci test json

dev --> staging

github-actions · 2024-10-09T20:26:45Z

DO NOT SQUASH MERGE!!!

github-actions · 2024-10-09T20:26:52Z

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/master':
All changelog files are valid for this release.

github-actions · 2024-10-09T20:26:57Z

🔍Version Validation Results:

basename: missing operand
Try 'basename --help' for more information.
Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
All WDLs and changelog files appear to be valid for this release.

ekiernan · 2024-10-09T20:28:42Z

retest this please

ekiernan · 2024-10-10T14:04:51Z

retest this please

ekiernan · 2024-10-11T13:25:23Z

Smart-tests rerun here: https://gotc-jenkins.dsp-techops.broadinstitute.org/job/warp-smart-tests/8522/

As expected, Multiome failed with similar failures that we saw before in library metrics and h5ad because we added doublet info.

Multiome truth update is here:https://gotc-jenkins.dsp-techops.broadinstitute.org/job/warp-workflow-tests/85725/parameters/

All other tests now pass because we updated truth.

nikellepetrillo and others added 10 commits September 12, 2024 08:09

WARP Doc Site Updates (#1371)

e1891ab

* Update Multiome, ATAC, and Optimus readmes

Lk pd 2744 (#1372)

1268eb6

Added deprecation notices for deprecated pipelines

Np fix womtools validation (#1374)

58285d0

* pin all latest docker version * loom to h5ad to fix womtools check * Update ValidateSmartSeq2SingleNucleus.wdl

added --preserve-map-align-order (#1367)

1bd7527

* Added parameter to avoid non-determinism in DRAGEN aligner

Update .dockstore.yml with TestIllumina (#1375)

b433b53

Lk doc updates 20240925 (#1376)

0c6697b

adding parameters to handle trailing slash

Lk pd 2736 doublets (#1362)

30e4994

Added doublet scores to gex h5ads, created new nhash for ATAC

Merge pull request #1378 from broadinstitute/develop

e485b0a

dev --> staging

ekiernan added 2 commits October 9, 2024 16:12

adding multiome sci test atac_nhash_id (#1383)

ec91e51

added atac_nhash_id to multiome sci test json

Merge pull request #1384 from broadinstitute/develop

3ff3c74

dev --> staging

rsc3 approved these changes Oct 11, 2024

View reviewed changes

jessicaway approved these changes Oct 11, 2024

View reviewed changes

ekiernan merged commit 3091fbd into master Oct 11, 2024
7 of 8 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

staging --> master #1379

staging --> master #1379

ekiernan commented Oct 2, 2024

github-actions bot commented Oct 2, 2024

github-actions bot commented Oct 2, 2024

github-actions bot commented Oct 2, 2024

ekiernan commented Oct 9, 2024 •

edited

Loading

ekiernan commented Oct 9, 2024 •

edited

Loading

jessicaway commented Oct 9, 2024

ekiernan commented Oct 9, 2024

github-actions bot commented Oct 9, 2024

github-actions bot commented Oct 9, 2024

github-actions bot commented Oct 9, 2024

ekiernan commented Oct 9, 2024

ekiernan commented Oct 10, 2024

ekiernan commented Oct 11, 2024

staging --> master #1379

staging --> master #1379

Conversation

ekiernan commented Oct 2, 2024

github-actions bot commented Oct 2, 2024

github-actions bot commented Oct 2, 2024

🔍Version Validation Results:

github-actions bot commented Oct 2, 2024

🔍Changelog Validation Results:

ekiernan commented Oct 9, 2024 • edited Loading

ekiernan commented Oct 9, 2024 • edited Loading

jessicaway commented Oct 9, 2024

ekiernan commented Oct 9, 2024

github-actions bot commented Oct 9, 2024

github-actions bot commented Oct 9, 2024

🔍Changelog Validation Results:

github-actions bot commented Oct 9, 2024

🔍Version Validation Results:

ekiernan commented Oct 9, 2024

ekiernan commented Oct 10, 2024

ekiernan commented Oct 11, 2024

ekiernan commented Oct 9, 2024 •

edited

Loading

ekiernan commented Oct 9, 2024 •

edited

Loading