Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ashlar #3551

Merged
merged 135 commits into from
Aug 15, 2023
Merged

Ashlar #3551

Show file tree
Hide file tree
Changes from 129 commits
Commits
Show all changes
135 commits
Select commit Hold shift + click to select a range
92ef528
working on ashlar nextflow module outside nf-core template. Input an…
RobJY Nov 1, 2022
715a1c4
added default output filename.
RobJY Nov 1, 2022
10f2054
changed output parameter to match ashlar param name and changed outpu…
RobJY Nov 1, 2022
91b8e3a
added optional input parameter channel
RobJY Nov 1, 2022
e09d8ad
added most input paramters. working code
RobJY Nov 1, 2022
84269e4
clean-up
RobJY Nov 1, 2022
724412e
added output-channels parameter
RobJY Nov 1, 2022
62e16c6
added support for multiple input files
RobJY Nov 2, 2022
ccdc22d
added ffp input parameter
RobJY Nov 2, 2022
b4178e7
bug fix: should not have had multiple channels
RobJY Nov 2, 2022
11ee7b8
most of meta.yml filled out. still need the case for *.tif output
RobJY Nov 3, 2022
7e20a5d
ah, also need to add defaults and optional args to meta.yml
RobJY Nov 3, 2022
c85d04d
minimal set of input parameters added to meta.yml for testing; need t…
RobJY Nov 3, 2022
2a8ea15
runs simple test with hard-coded paths
RobJY Nov 14, 2022
68e4189
simple test works with input parameter and no hard-coded paths
RobJY Nov 14, 2022
6ad6999
added ashlar main module file
RobJY Nov 14, 2022
54c9029
added test image path to test workflow so you can run with default te…
RobJY Nov 15, 2022
cb26c76
added ext.args
RobJY Nov 15, 2022
828aba3
added meta map
RobJY Nov 16, 2022
63e7ebc
passed test with wildcarded input files
RobJY Nov 16, 2022
4970bd3
added test config file
RobJY Nov 16, 2022
deb9bb5
added functionality to handle optional arguments
RobJY Nov 16, 2022
c47d3b6
added meta map args
RobJY Nov 17, 2022
d555aee
supposed to output meta map if you input one
RobJY Nov 17, 2022
c8ac4fb
clean-up
RobJY Nov 17, 2022
31acde9
changed test input to a channel
RobJY Nov 21, 2022
288f590
not running, but close I think
RobJY Nov 22, 2022
6843268
added some missing files
RobJY Nov 22, 2022
69a807c
basic samplesheet working
RobJY Nov 28, 2022
eaa39c3
clean-up post testing
RobJY Nov 29, 2022
b57595a
enabled multiple input images
RobJY Nov 30, 2022
125281f
bug fix: sheet meta arg not an it var
RobJY Nov 30, 2022
9f98637
clean-up
RobJY Nov 30, 2022
9c17dff
added samplesheet validation
RobJY Dec 1, 2022
1f45791
samplesheet output to subdirs with sample name; needs clean-up
RobJY Dec 3, 2022
9bc2269
cleaned up meta.yml and passed test
RobJY Dec 9, 2022
f6bedd0
clean-up; passed tests
RobJY Dec 9, 2022
afb2c27
moved publishDir to test config as suggested by lint
RobJY Dec 9, 2022
2329c99
removed initOptions; lint says deprecated; tests pass
RobJY Dec 9, 2022
307d803
multi-file row of test_1 failing, but others work
RobJY Jan 4, 2023
2d5091c
passes 3 tests with stable md5 sums
RobJY Jan 9, 2023
bbfa613
clean-up and changed back to uuid block zero out instead of while loo…
RobJY Jan 10, 2023
edc4998
removed test publishDir for sample sheets
RobJY Jan 10, 2023
12ad5df
clean-up
RobJY Jan 10, 2023
eea6c46
added Dockerfile
RobJY Jan 10, 2023
ba68c8e
clean-up
RobJY Jan 11, 2023
9996fa7
updated to correct DOI
RobJY Jan 13, 2023
c513dbb
added zero_uuid.nf
RobJY Jan 13, 2023
71b4be6
removed test sample sheet
RobJY Jan 13, 2023
aff97c0
changed input type to *.tif to match output type
RobJY Jan 13, 2023
356f478
removed old test ashlar nextflow module
RobJY Jan 13, 2023
c809ad4
Update .nf-core.yml
RobJY Jan 13, 2023
14fb0dc
restored formatting in pytest_modules.yml
RobJY Jan 13, 2023
b111ed2
added more accurate input pattern
RobJY Jan 17, 2023
b3fbd52
works with new test images
RobJY Mar 14, 2023
a5165c6
added test.yml
RobJY Mar 14, 2023
5487b2c
removed unsupported output test and updated input and output patterns…
RobJY Mar 15, 2023
95160c5
refactored zero_uuid.nf to use input parameter offset; passed test
RobJY Mar 17, 2023
2d9411c
added tile size test as discussed in PR
RobJY Mar 17, 2023
79a1ef4
bug fix: off by one error for second test offset
RobJY Mar 17, 2023
2c133d6
clean-up
RobJY Mar 20, 2023
0828127
simplified zero_uuid.nf
RobJY Mar 21, 2023
46451f7
Merge branch 'nf-core:master' into ashlar
RobJY Mar 30, 2023
c908aea
changed to built-in test files
RobJY Mar 31, 2023
5c3ce93
removed conda command so runs locally
RobJY Mar 31, 2023
f1a99f2
added commented code that causes break
RobJY Mar 31, 2023
b26f50a
works in linux with conda so uncommented that section
RobJY Mar 31, 2023
f567546
added containers and tested all 3 ways of running; all working
RobJY Apr 3, 2023
77d4e0f
ran prettier
RobJY May 16, 2023
f693887
updated docker container URL
RobJY May 16, 2023
f50061d
testing container URL with hash
RobJY May 16, 2023
b133416
switched container to docker.io
RobJY May 16, 2023
3fb1383
fixed meta.yaml lint errors
RobJY May 16, 2023
fa2e25d
trying CI again with both container hashes
RobJY May 17, 2023
f8c84ca
trying with default container registry
RobJY May 17, 2023
efe10c8
bug fix: removed extra char output in version.yml
RobJY May 17, 2023
854865b
changed container URL syntax and it works locally; testing with CI
RobJY May 17, 2023
9709eb9
added conda again to test with CI
RobJY May 17, 2023
5725872
trying container URL without quay.io again
RobJY May 18, 2023
8a7dae8
Merge branch 'master' into ashlar
RobJY May 18, 2023
1bad2c1
testing URL with quay.io again
RobJY May 18, 2023
a8e7105
added singularity; passes local tests
RobJY May 25, 2023
fb3833c
Merge branch 'master' into ashlar
RobJY Jun 6, 2023
14f1034
Merge branch 'master' into ashlar
RobJY Jun 20, 2023
f73e168
cleaned-up unused files
RobJY Jun 21, 2023
cd4a506
removed meta args
RobJY Jun 21, 2023
2e3bd97
removed unnecessary Channel.toList from test main
RobJY Jun 21, 2023
27a0c1d
restored tile test using withName technique for param
RobJY Jun 21, 2023
19ec447
simplified input arg name
RobJY Jun 21, 2023
a2a5c96
updated module output path and emit
RobJY Jun 21, 2023
352d579
updated module usage command
RobJY Jun 21, 2023
ff0ef63
updated licence in meta
RobJY Jun 21, 2023
97c5beb
cleaned-up meta.yml
RobJY Jun 21, 2023
d73023b
Merge branch 'master' into ashlar
RobJY Jun 22, 2023
c8bdccd
Update modules/nf-core/ashlar/main.nf
RobJY Jun 22, 2023
c1f36ae
Update modules/nf-core/ashlar/main.nf
RobJY Jun 22, 2023
9b018f2
Update modules/nf-core/ashlar/main.nf
RobJY Jun 22, 2023
cc5c668
changed dfp and ffp to optional input channels and tested with no inp…
RobJY Jun 22, 2023
1e03f60
added dfp ffp test with hard-coded files
RobJY Jun 23, 2023
fca179d
Merge branch 'master' into ashlar
SPPearce Jun 23, 2023
3a5e027
added validation for dfp and ffp
RobJY Jun 26, 2023
b746eea
changed validation for dfp/ffp and added test
RobJY Jun 28, 2023
00077da
Merge branch 'master' into ashlar
RobJY Jul 5, 2023
7d8c759
Merge branch 'master' into ashlar
RobJY Jul 14, 2023
e9d0b19
now pulling dfp/ffp from test_data repo
RobJY Jul 24, 2023
aa88e25
Merge branch 'master' into ashlar
RobJY Jul 25, 2023
2be482d
Merge branch 'master' into ashlar
RobJY Jul 26, 2023
06e9913
Merge branch 'master' into ashlar
RobJY Jul 31, 2023
6e192d7
test_data formatting change
RobJY Jul 31, 2023
78baab7
Merge branch 'master' into ashlar
RobJY Aug 1, 2023
a89bfaf
Merge branch 'master' into ashlar
RobJY Aug 2, 2023
784057d
Merge branch 'master' into ashlar
RobJY Aug 3, 2023
e660f93
Update modules/nf-core/ashlar/main.nf
RobJY Aug 3, 2023
6d7fa19
corrected dfp/ffp error message
RobJY Aug 3, 2023
059c947
Update modules/nf-core/ashlar/meta.yml
RobJY Aug 3, 2023
3b26a84
Update modules/nf-core/ashlar/meta.yml
RobJY Aug 3, 2023
65f2e98
finished input name change
RobJY Aug 3, 2023
0e5e56b
Apply suggestions from code review
RobJY Aug 3, 2023
8bec89d
Update tests/modules/nf-core/ashlar/main.nf
RobJY Aug 3, 2023
50d4a6c
broken change for discussion
RobJY Aug 4, 2023
4521bb8
Update tests/modules/nf-core/ashlar/main.nf
RobJY Aug 4, 2023
056a240
fixed zero_uuid output bug
RobJY Aug 4, 2023
dbbfeb3
Merge branch 'master' into ashlar
RobJY Aug 7, 2023
d05796a
Merge branch 'master' into ashlar
RobJY Aug 8, 2023
57a8dc4
Merge branch 'master' into ashlar
RobJY Aug 9, 2023
146534a
Merge branch 'master' into ashlar
RobJY Aug 10, 2023
76ee81e
Merge branch 'master' into ashlar
RobJY Aug 10, 2023
b92c085
Use sed to remove UUID inside main module
SPPearce Aug 11, 2023
440f140
Fixed sed
SPPearce Aug 11, 2023
10bf81e
Update modules/nf-core/ashlar/main.nf
SPPearce Aug 11, 2023
d5d60ff
Merge branch 'master' into ashlar
RobJY Aug 11, 2023
7fa6f95
proposed change to sed command
RobJY Aug 11, 2023
7988e55
adding modified test.yml file
RobJY Aug 11, 2023
4aa7498
reverted to in-place sed
RobJY Aug 14, 2023
926caa7
Merge branch 'master' into ashlar
RobJY Aug 15, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 53 additions & 0 deletions modules/nf-core/ashlar/main.nf
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
process ASHLAR {
tag '$meta.id'
label 'process_single'

conda "bioconda::ashlar=1.17.0"
container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
'https://depot.galaxyproject.org/singularity/ashlar:1.17.0--pyh5e36f6f_0' :
'biocontainers/ashlar:1.17.0--pyh5e36f6f_0' }"

input:
tuple val(meta), path(images)
path(opt_dfp)
path(opt_ffp)

output:
tuple val(meta), path("*.ome.tif"), emit: tif
path "versions.yml" , emit: versions

when:
task.ext.when == null || task.ext.when

script:
def args = task.ext.args ?: ''
def prefix = task.ext.prefix ?: "${meta.id}"
def dfp = opt_dfp ? "--dfp $opt_dfp" : ""
def ffp = opt_ffp ? "--ffp $opt_ffp" : ""
def num_files = images instanceof List ? images.size() : 1
def opt_dfp_size = opt_dfp instanceof List ? opt_dfp.size() : 1
def opt_ffp_size = opt_ffp instanceof List ? opt_ffp.size() : 1
def dfp_validated = (opt_dfp_size == 0 || opt_dfp_size == 1 || opt_dfp_size == num_files) ? true : false
def ffp_validated = (opt_ffp_size == 0 || opt_ffp_size == 1 || opt_ffp_size == num_files) ? true : false
SPPearce marked this conversation as resolved.
Show resolved Hide resolved

if ( !dfp_validated ) { error "Please input only zero, one, or N dfp files, where N is the number of input images" }
if ( !ffp_validated ) { error "Please input only zero, one, or N ffp files, where N is the number of input images" }

"""

ashlar \\
-o ${prefix}.uuid.ome.tif \\
$images \\
$args \\
$dfp \\
$ffp

sed -E 's/UUID="urn:uuid:[[:xdigit:]]{8}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{12}"/UUID="urn:uuid:00000000-0000-0000-0000-000000000000"/g' ${prefix}.uuid.ome.tif > ${prefix}.ome.tif
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's so ugly I love it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you want to just remove the UUID instead of replacing it with zeros? Does that break stuff?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have no idea whether any downstream tool may care about it or not.
I considered just finding and replacing everything after the urn:uuid, but didn't want to accidentally match anything else.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@SPPearce Can you comment on why you changed from sed -i (in-place) to operating on a copy? These images can be hundreds of GBs so we try to avoid making copies where possible.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The -i flag didn't seem to be working on the CI for some reason. I definitely agree that would be better.

One thought is that the metadata is at the very end of the tif from what I could see, so there might be some clever sed (or similar) magic that could be used to start from the end of a file and only make one change.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, the test.yml was broken in your commit that failed CI. The file tests were changed from md5sum to contains with the placeholder TODO message. That's since been fixed so we're going to try again with sed -i.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It now passes all tests.
Thank you for figuring that out @jmuhlich!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this an acceptable change @SPPearce?

rm ${prefix}.uuid.ome.tif

cat <<-END_VERSIONS > versions.yml
"${task.process}":
ashlar: \$(ashlar --version 2>&1 | sed 's/^.*ashlar //; s/Using.*\$//' )
END_VERSIONS
"""
}
42 changes: 42 additions & 0 deletions modules/nf-core/ashlar/meta.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
name: "ashlar"
description: Alignment by Simultaneous Harmonization of Layer/Adjacency Registration
keywords:
- image_processing
- alignment
- registration
tools:
- "ashlar":
description: "Alignment by Simultaneous Harmonization of Layer/Adjacency Registration"
homepage: "https://labsyspharm.github.io/ashlar/"
documentation: "https://labsyspharm.github.io/ashlar/"
doi: "10.1093/bioinformatics/btac544"
licence: ["MIT"]

input:
- meta:
type: map
description: |
Groovy Map containing sample information
e.g. [ id:'test' ]
- images:
type: file
description: Overlapping tile image data in formats from most commercial microscopes
pattern: "*.{ome.tiff,ome.tif,rcpnl,btf,nd2,tiff,tif,czi}"
output:
- meta:
type: map
description: |
Groovy Map containing sample information
e.g. [ id:'test' ]
- tif:
type: file
description: A pyramidal, tiled OME-TIFF file created from input images.
pattern: "*.ome.tif"
- versions:
type: file
description: File containing software versions
pattern: "versions.yml"

authors:
- "@RobJY"
- "@jmuhlich"
4 changes: 4 additions & 0 deletions tests/config/pytest_modules.yml
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,10 @@ ascat:
- modules/nf-core/ascat/**
- tests/modules/nf-core/ascat/**

ashlar:
- modules/nf-core/ashlar/**
- tests/modules/nf-core/ashlar/**

assemblyscan:
- modules/nf-core/assemblyscan/**
- tests/modules/nf-core/assemblyscan/**
Expand Down
2 changes: 2 additions & 0 deletions tests/config/test_data.config
Original file line number Diff line number Diff line change
Expand Up @@ -671,6 +671,8 @@ params {
cycif_tonsil_cycle1 = "${params.test_data_base}/data/imaging/ome-tiff/cycif-tonsil-cycle1.ome.tif"
cycif_tonsil_cycle2 = "${params.test_data_base}/data/imaging/ome-tiff/cycif-tonsil-cycle2.ome.tif"
cycif_tonsil_cycle3 = "${params.test_data_base}/data/imaging/ome-tiff/cycif-tonsil-cycle3.ome.tif"
cycif_tonsil_dfp = "${params.test_data_base}/data/imaging/ome-tiff/cycif-tonsil-dfp.ome.tif"
cycif_tonsil_ffp = "${params.test_data_base}/data/imaging/ome-tiff/cycif-tonsil-ffp.ome.tif"
}
'registration' {
markers = "${params.test_data_base}/data/imaging/registration/markers.csv"
Expand Down
90 changes: 90 additions & 0 deletions tests/modules/nf-core/ashlar/main.nf
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
#!/usr/bin/env nextflow

nextflow.enable.dsl = 2

include { ASHLAR } from '../../../../modules/nf-core/ashlar/main.nf'
include { ASHLAR as ASHLAR_TILE } from '../../../../modules/nf-core/ashlar/main.nf'

workflow test_ashlar_1_file {

input_list = [ [ id:'test_all' ],
[file(params.test_data['imaging']
['ome-tiff']
['cycif_tonsil_cycle1'], checkIfExists: true)] ]

ASHLAR ( input_list, [], [] )

}

workflow test_ashlar_all_files {

input_list = [ [ id:'test_all' ],
[file(params.test_data['imaging']
['ome-tiff']
['cycif_tonsil_cycle1'], checkIfExists: true),
file(params.test_data['imaging']
['ome-tiff']
['cycif_tonsil_cycle2'], checkIfExists: true),
file(params.test_data['imaging']
['ome-tiff']
['cycif_tonsil_cycle3'], checkIfExists: true)] ]

ASHLAR ( input_list, [], [] )

}

workflow test_ashlar_all_files_tile_size {

input_list = [ [ id:'test_all' ],
[file(params.test_data['imaging']
['ome-tiff']
['cycif_tonsil_cycle1'], checkIfExists: true),
file(params.test_data['imaging']
['ome-tiff']
['cycif_tonsil_cycle2'], checkIfExists: true),
file(params.test_data['imaging']
['ome-tiff']
['cycif_tonsil_cycle3'], checkIfExists: true)] ]

ASHLAR_TILE ( input_list, [], [] )

}

workflow test_ashlar_all_files_dfp_ffp {

input_list = [ [ id:'test_all' ],
[file(params.test_data['imaging']
['ome-tiff']
['cycif_tonsil_cycle1'], checkIfExists: true),
file(params.test_data['imaging']
['ome-tiff']
['cycif_tonsil_cycle2'], checkIfExists: true),
file(params.test_data['imaging']
['ome-tiff']
['cycif_tonsil_cycle3'], checkIfExists: true)] ]

ASHLAR ( input_list, [file(params.test_data['imaging']
['ome-tiff']
['cycif_tonsil_dfp'], checkIfExists: true)],
[file(params.test_data['imaging']
['ome-tiff']
['cycif_tonsil_ffp'], checkIfExists: true)] )

}

workflow test_ashlar_1_file_dfp_ffp {

input_list = [ [ id:'test_all' ],
[file(params.test_data['imaging']
['ome-tiff']
['cycif_tonsil_cycle1'], checkIfExists: true)] ]

ASHLAR ( input_list, [file(params.test_data['imaging']
['ome-tiff']
['cycif_tonsil_dfp'], checkIfExists: true)],
[file(params.test_data['imaging']
['ome-tiff']
['cycif_tonsil_ffp'], checkIfExists: true)] )

}

9 changes: 9 additions & 0 deletions tests/modules/nf-core/ashlar/nextflow.config
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
process {

publishDir = { "${params.outdir}/${task.process.tokenize(':')[-1].tokenize('_')[0].toLowerCase()}" }

withName: ASHLAR_TILE {
ext.args = "--tile-size 512"
}

}
44 changes: 44 additions & 0 deletions tests/modules/nf-core/ashlar/test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
- name: ashlar test_ashlar_1_file
command: nextflow run ./tests/modules/nf-core/ashlar -entry test_ashlar_1_file -c ./tests/config/nextflow.config -c ./tests/modules/nf-core/ashlar/nextflow.config
tags:
- ashlar
files:
- path: output/ashlar/test_all.ome.tif
md5sum: 796fa3687d4ff1389567c22eafb1e4cf
- path: output/ashlar/versions.yml

- name: ashlar test_ashlar_all_files
command: nextflow run ./tests/modules/nf-core/ashlar -entry test_ashlar_all_files -c ./tests/config/nextflow.config -c ./tests/modules/nf-core/ashlar/nextflow.config
tags:
- ashlar
files:
- path: output/ashlar/test_all.ome.tif
md5sum: 576bc81fa8d678547e87d25c403ed3a4
- path: output/ashlar/versions.yml

- name: ashlar test_ashlar_all_files_tile_size
command: nextflow run ./tests/modules/nf-core/ashlar -entry test_ashlar_all_files_tile_size -c ./tests/config/nextflow.config -c ./tests/modules/nf-core/ashlar/nextflow.config
tags:
- ashlar
files:
- path: output/ashlar/test_all.ome.tif
md5sum: fb34bb112d0c95698afa76095e12085f
- path: output/ashlar/versions.yml

- name: ashlar test_ashlar_all_files_dfp_ffp
command: nextflow run ./tests/modules/nf-core/ashlar -entry test_ashlar_all_files_dfp_ffp -c ./tests/config/nextflow.config -c ./tests/modules/nf-core/ashlar/nextflow.config
tags:
- ashlar
files:
- path: output/ashlar/test_all.ome.tif
md5sum: 576bc81fa8d678547e87d25c403ed3a4
- path: output/ashlar/versions.yml

- name: ashlar test_ashlar_1_file_dfp_ffp
command: nextflow run ./tests/modules/nf-core/ashlar -entry test_ashlar_1_file_dfp_ffp -c ./tests/config/nextflow.config -c ./tests/modules/nf-core/ashlar/nextflow.config
tags:
- ashlar
files:
- path: output/ashlar/test_all.ome.tif
md5sum: 796fa3687d4ff1389567c22eafb1e4cf
- path: output/ashlar/versions.yml
Loading