Grohmm overhaul #165

edmundmiller · 2024-09-26T19:44:23Z

Closes #127

PR checklist

github-actions · 2024-09-26T19:47:01Z

`nf-core pipelines lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit e7b2513

+| ✅ 216 tests passed       |+
#| ❔   4 tests were ignored |#
!| ❗   4 tests had warnings |!

❗ Test warnings:

pipeline_todos - TODO string in nextflow.config: Optionally, you can add a pipeline-specific nf-core config at https://github.com/nf-core/configs
pipeline_todos - TODO string in README.md: Describe the minimum required steps to execute the pipeline, e.g. how to prepare samplesheets.
pipeline_todos - TODO string in base.config: Check the defaults for all processes
pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline

❔ Tests ignored:

files_unchanged - File ignored due to lint config: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_unchanged - File ignored due to lint config: .github/workflows/linting.yml
files_unchanged - File ignored due to lint config: assets/email_template.html
actions_ci - actions_ci

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-nascent_logo_light.png
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/images/nf-core-nascent_logo_light.png
files_exist - File found: docs/images/nf-core-nascent_logo_dark.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes.config
files_exist - File found: conf/igenomes_ignored.config
files_exist - File found: .github/workflows/awstest.yml
files_exist - File found: .github/workflows/awsfulltest.yml
files_exist - File found: modules.json
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: docs/images/nf-core-nascent_logo.png
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/NfcoreTemplate.groovy
files_exist - File not found check: lib/Utils.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: lib/WorkflowMain.groovy
files_exist - File not found check: lib/WorkflowNascent.groovy
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: Singularity
files_exist - File not found check: lib/nfcore_external_java_deps.jar
files_exist - File not found check: .travis.yml
nextflow_config - Found nf-schema plugin
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: validation.help.enabled
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable found: validation.help.beforeText
nextflow_config - Config variable found: validation.help.afterText
nextflow_config - Config variable found: validation.help.command
nextflow_config - Config variable found: validation.summary.beforeText
nextflow_config - Config variable found: validation.summary.afterText
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config variable (correctly) not found: params.max_cpus
nextflow_config - Config variable (correctly) not found: params.max_memory
nextflow_config - Config variable (correctly) not found: params.max_time
nextflow_config - Config variable (correctly) not found: params.validationFailUnrecognisedParams
nextflow_config - Config variable (correctly) not found: params.validationLenientMode
nextflow_config - Config variable (correctly) not found: params.validationSchemaIgnoreParams
nextflow_config - Config variable (correctly) not found: params.validationShowHiddenParams
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: 2.3.0dev
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
nextflow_config - nextflow.config contains configuration profile test
nextflow_config - Config default value correct: params.aligner= bwa
nextflow_config - Config default value correct: params.skip_grohmm= false
nextflow_config - Config default value correct: params.grohmm_min_uts= 5
nextflow_config - Config default value correct: params.grohmm_max_uts= 45
nextflow_config - Config default value correct: params.grohmm_min_ltprobb= -100
nextflow_config - Config default value correct: params.grohmm_max_ltprobb= -400
nextflow_config - Config default value correct: params.igenomes_base= s3://ngi-igenomes/igenomes/
nextflow_config - Config default value correct: params.human_pangenomics_base= https://s3-us-west-2.amazonaws.com/human-pangenomics
nextflow_config - Config default value correct: params.custom_config_version= master
nextflow_config - Config default value correct: params.custom_config_base= https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Config default value correct: params.publish_dir_mode= copy
nextflow_config - Config default value correct: params.max_multiqc_email_size= 25.MB
nextflow_config - Config default value correct: params.validate_params= true
nextflow_config - Config default value correct: params.pipelines_testdata_base_path= https://raw.githubusercontent.com/nf-core/test-datasets/
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/CONTRIBUTING.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - assets/nf-core-nascent_logo_light.png matches the template
files_unchanged - docs/images/nf-core-nascent_logo_light.png matches the template
files_unchanged - docs/images/nf-core-nascent_logo_dark.png matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - .gitignore matches the template
files_unchanged - .prettierignore matches the template
actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml does not use -profile test
readme - README Nextflow minimum version badge matched config. Badge: 24.04.2, Config: 24.04.2
readme - README Zenodo placeholder was replaced with DOI.
plugin_includes - No wrong validation plugin imports have been found
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (0 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: release-announcements.yml
actions_schema_validation - Workflow validation passed: awsfulltest.yml
actions_schema_validation - Workflow validation passed: template_version_comment.yml
actions_schema_validation - Workflow validation passed: download_pipeline.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: awstest.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - assets/multiqc_config.yml found and not ignored.
multiqc_config - assets/multiqc_config.yml contains report_section_order
multiqc_config - assets/multiqc_config.yml contains export_plots
multiqc_config - assets/multiqc_config.yml contains report_comment
multiqc_config - assets/multiqc_config.yml follows the ordering scheme of the minimally required plugins.
multiqc_config - assets/multiqc_config.yml contains a matching 'report_comment'.
multiqc_config - assets/multiqc_config.yml contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'
base_config - conf/base.config found and not ignored.
modules_config - conf/modules.config found and not ignored.
modules_config - FASTQC found in conf/modules.config and Nextflow scripts.
modules_config - MULTIQC found in conf/modules.config and Nextflow scripts.
modules_config - GFFREAD found in conf/modules.config and Nextflow scripts.
modules_config - FASTP found in conf/modules.config and Nextflow scripts.
modules_config - BWA_MEM found in conf/modules.config and Nextflow scripts.
modules_config - BWAMEM2_MEM found in conf/modules.config and Nextflow scripts.
modules_config - BOWTIE2_ALIGN found in conf/modules.config and Nextflow scripts.
modules_config - DRAGMAP_ALIGN found in conf/modules.config and Nextflow scripts.
modules_config - STAR_ALIGN found in conf/modules.config and Nextflow scripts.
modules_config - PRESEQ_ found in conf/modules.config and Nextflow scripts.
modules_config - BBMAP_PILEUP found in conf/modules.config and Nextflow scripts.
modules_config - BEDTOOLS_GENOMECOV_PLUS found in conf/modules.config and Nextflow scripts.
modules_config - BEDTOOLS_GENOMECOV_MINUS found in conf/modules.config and Nextflow scripts.
modules_config - DEEPTOOLS_BAMCOVERAGE_PLUS found in conf/modules.config and Nextflow scripts.
modules_config - DEEPTOOLS_BAMCOVERAGE_MINUS found in conf/modules.config and Nextflow scripts.
modules_config - DREG_PREP found in conf/modules.config and Nextflow scripts.
modules_config - HOMER_ found in conf/modules.config and Nextflow scripts.
modules_config - HOMER_FINDPEAKS found in conf/modules.config and Nextflow scripts.
modules_config - HOMER_MAKETAGDIRECTORY found in conf/modules.config and Nextflow scripts.
modules_config - HOMER_MAKEUCSCFILE found in conf/modules.config and Nextflow scripts.
modules_config - PINTS_CALLER found in conf/modules.config and Nextflow scripts.
modules_config - GROHMM_ found in conf/modules.config and Nextflow scripts.
modules_config - BEDTOOLS_SORT found in conf/modules.config and Nextflow scripts.
modules_config - BEDTOOLS_MERGE found in conf/modules.config and Nextflow scripts.
modules_config - BEDTOOLS_INTERSECT_FILTER found in conf/modules.config and Nextflow scripts.
modules_config - BEDTOOLS_INTERSECT found in conf/modules.config and Nextflow scripts.
modules_config - SUBREAD_FEATURECOUNTS_GENE found in conf/modules.config and Nextflow scripts.
modules_config - SUBREAD_FEATURECOUNTS_PREDICTED found in conf/modules.config and Nextflow scripts.
nfcore_yml - Repository type in .nf-core.yml is valid: pipeline
nfcore_yml - nf-core version in .nf-core.yml is set to the latest version: 3.0.2

Run details

nf-core/tools version 3.0.2
Run at 2024-10-20 02:53:44

Apparently we weren't doing this. It's in the guidelines and in Shayne's scripts

Now it handles when there are no broken transcripts(ie with a test dataset)

Didn't know that was a thing!

Since these are are split into seperate jobs now, they're not "long"(longest I've seen one take is ~143 minutes). Adding arrays later because there's so many of them and they're the same

Didn't know that was a thing!

With the help of cursor

refactor: Clean up code more

Scrap all of the CHM13 attempts I think it's the gtf conversion stuff that's causing the memory overload. But I can get them all to load locally. So I'm going to try to use gff3 files, if they're supplied, otherwise fall back to gtf.

edmundmiller changed the base branch from master to dev September 26, 2024 19:45

edmundmiller self-assigned this Sep 26, 2024

nf-core deleted a comment from github-actions bot Sep 26, 2024

edmundmiller force-pushed the grohmm-overhaul branch 3 times, most recently from 75040bc to 0ca9dce Compare September 30, 2024 12:36

edmundmiller force-pushed the grohmm-overhaul branch 2 times, most recently from abc2eea to 308911a Compare October 9, 2024 17:54

edmundmiller marked this pull request as ready for review October 15, 2024 13:34

edmundmiller force-pushed the grohmm-overhaul branch from 41b5e55 to 505251b Compare October 15, 2024 13:45

edmundmiller force-pushed the dev branch from c63bfba to a9ae47a Compare October 15, 2024 17:25

edmundmiller force-pushed the grohmm-overhaul branch from 505251b to dcea11c Compare October 15, 2024 17:51

edmundmiller added 17 commits October 16, 2024 14:05

style: Run styler

f41ed1c

test: Write grohmm tests

9e75201

fix(grohmm): bam => bams

e2ea287

fix(grohmm): Link up tuning files and samples

67472af

refactor(grohmm): Add notes of brainstorming

485651f

feat: Add grohmm min and max tuning params

220ef34

fix: of => fromList

215ee98

refactor(grohmm): Use parameter tuning split

e47d2c6

fix(grohmm): Use windowAnalysis

8487770

Apparently we weren't doing this. It's in the guidelines and in Shayne's scripts

refactor: Setup transcript calling

3cb5736

build: Build a seperate grohmm conda package

1ce7790

chore: Add working

2668e95

build(grohmm): Add Seqera containers

428dca0

test(grohmm): Add chr7 gtf

a0d5f4a

test: Setup tests for grohmm

980fd9b

chore(grohmm): Add example tuning evals from tutorial

35f1e45

fix(grohmm): Try removing any genes from "random" Chromosome

ff66d45

fix(grohmm): Update groHMM with fix

ffd7482

Now it handles when there are no broken transcripts(ie with a test dataset)

edmundmiller force-pushed the grohmm-overhaul branch from 7f3dd19 to f206870 Compare October 16, 2024 19:05

edmundmiller added 17 commits October 17, 2024 20:11

Add consensus bed as output for testing

f264b18

test(grohmm): Update all the snapshots

a24248c

refactor(grohmm): Use each input

3811a91

Didn't know that was a thing!

feat(grohmm): Add Native MultiQC support

b460df0

fix(grohmm): Update labels for parametertuning

5b8e3b1

Since these are are split into seperate jobs now, they're not "long"(longest I've seen one take is ~143 minutes). Adding arrays later because there's so many of them and they're the same

chore: Add a note about the subworkflow functionality

2e3f182

chore: oras => https

2c77f02

refactor(grohmm): Use each input

9c9fdf2

Didn't know that was a thing!

test(grohmm): Write a failing test

c135b97

With the help of cursor

fix(grohmm): Update to work with CHM13

b62cabf

refactor: Clean up code more

refactor: Replace jpg with png

f9ec2bf

test: Remove skip_tuning test

1e26e22

test: Bump snapshot

f5ea409

fix(multiqc): Update groHMM plot fn

bc7ecc1

Update CHANGELOG

2abdf46

docs: Update groHMM docs

68f53bb

fix(grohmm): Try calling no more than 10 cores

83c870a

edmundmiller force-pushed the grohmm-overhaul branch from ff38d91 to ea5bba6 Compare October 18, 2024 01:39

edmundmiller added 6 commits October 18, 2024 10:09

chore: Add a custom makeConsensusAnnotations function

42d1996

Try creating a custom makeConsensusAnnotations and setup test

58913d1

Get custom running

e06478f

Start over

495cd58

refactor: Split into chunks

fceb582

More tests

c5d7b3b

edmundmiller force-pushed the grohmm-overhaul branch 2 times, most recently from d27b481 to b6a93ca Compare October 20, 2024 00:25

fix(grohmm): Support gxf

e7b2513

Scrap all of the CHM13 attempts I think it's the gtf conversion stuff that's causing the memory overload. But I can get them all to load locally. So I'm going to try to use gff3 files, if they're supplied, otherwise fall back to gtf.

edmundmiller force-pushed the grohmm-overhaul branch from b6a93ca to e7b2513 Compare October 20, 2024 02:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Grohmm overhaul #165

Grohmm overhaul #165

edmundmiller commented Sep 26, 2024 •

edited

Loading

github-actions bot commented Sep 26, 2024 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

Grohmm overhaul #165

Are you sure you want to change the base?

Grohmm overhaul #165

Conversation

edmundmiller commented Sep 26, 2024 • edited Loading

PR checklist

github-actions bot commented Sep 26, 2024 • edited Loading

nf-core pipelines lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

edmundmiller commented Sep 26, 2024 •

edited

Loading

github-actions bot commented Sep 26, 2024 •

edited

Loading

`nf-core pipelines lint` overall result: Passed ✅ ⚠️