-
Notifications
You must be signed in to change notification settings - Fork 15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
TransPi on HPC failing to pull singularity image #53
Comments
Hi Chris, not an expert with the singularity stuff here but it seems as if the image is not being pulled because of some naming problem. We run TransPi with conda and tend not to get conflicts except for one step. Do you face conflicts creating all environments or only one? Sergio |
Hi Sergio, Thanks for the fast response. When running the same command in a fresh environment with -profile conda, it halts with the following: Error executing process > 'evigene (SRR8641368_R)' Caused by:
|
Hi Chris, I checked the code. Specifically, the following line in the error message means that a different docker image is being used in this particular step. Why exactly busco is not included in the transpi image is something I cannot answer because I am not using docker/singularity and cannot test in our hpc anything related to these systems.
Now, I think you can solve the issue by adding the following line to your
the variable
The problem right now is that A working block of code, say Transabyss, looks like:
You see the structure is the same but the TPcontainer is called. This works before I hope this solves your issue. Please report back to the community and feel free to close the issue if solved. Note however, that other issues may arise from using that particular image, I hope not because it is used if the oneContainer variable is false i.e., you are not using one container with all the stuff but getting multiple images for each process in the pipeline. cheers Sergio |
Hi there,
I've been trying to use TransPi (love the concept and the benchmarking in the paper looks great!) to re-assemble some older public datasets. Running on an interactive slurm session of an HPC which has singularity installed:
(TransPi) chril1@hpc-gpu-002:~/TransPiDBs$ ./nextflow run ../TransPi/TransPi.nf --all --maxReadLen 150 --k 25,35,55,75,85 --reads '/home/chril1/NERC_PhyloCourse/transcriptome_data/Flagellophora_sp/SRR8641368_R[1,2].fastq.gz' --outdir Results_Flagellophora -profile singularity,TransPiContainer
N E X T F L O W ~ version 21.04.1
Launching
../TransPi/TransPi.nf
[furious_einstein] - revision: 5a26a6e85dTransPi - Transcriptome Analysis Pipeline v1.3.0-rc
TransPi.nf Directory: /gpfs/nhmfsa/bulk/share/data/mbl/share/workspaces/users/chril1/TransPi/TransPi.nf
Launch Directory: /gpfs/nhmfsa/bulk/share/data/mbl/share/workspaces/users/chril1/TransPiDBs
Results Directory: Results_Flagellophora
Work Directory: /gpfs/nhmfsa/bulk/share/data/mbl/share/workspaces/users/chril1/TransPiDBs/work
TransPi DBs: /home/chril1/TransPiDBs
Uniprot DB: /home/chril1/TransPiDBs/DBs/uniprot_db/uniprot_metazoa_33208.fasta
Busco DB: /home/chril1/TransPiDBs/DBs/busco_db/metazoa_odb10
Reads Directory: /home/chril1/NERC_PhyloCourse/transcriptome_data/Flagellophora_sp/SRR8641368_R[1,2].fastq.gz
Read Length: 150
Kmers: 25,35,55,75,85
executor > local (2)
[6c/876909] process > fasqc (SRR8641368_R) [ 0%] 0 of 1
executor > local (3)
[6c/876909] process > fasqc (SRR8641368_R) [ 0%] 0 of 1
executor > local (3)
[6c/876909] process > fasqc (SRR8641368_R) [ 0%] 0 of 1
executor > local (3)
[6c/876909] process > fasqc (SRR8641368_R) [ 0%] 0 of 1
executor > local (3)
[6c/876909] process > fasqc (SRR8641368_R) [ 0%] 0 of 1
executor > local (3)
[6c/876909] process > fasqc (SRR8641368_R) [100%] 1 of 1 ✔
executor > local (3)
[6c/876909] process > fasqc (SRR8641368_R) [100%] 1 of 1 ✔
executor > local (4)
executor > local (5)
executor > local (6)
executor > local (6)
executor > local (6)
executor > local (6)
executor > local (6)
executor > local (6)
[6c/876909] process > fasqc (SRR8641368_R) [100%] 1 of 1 ✔
executor > local (8)
[6c/876909] process > fasqc (SRR8641368_R) [100%] 1 of 1 ✔
executor > local (9)
[6c/876909] process > fasqc (SRR8641368_R) [100%] 1 of 1 ✔
executor > local (9)
[6c/876909] process > fasqc (SRR8641368_R) [100%] 1 of 1 ✔
executor > local (10)
executor > local (10)
executor > local (10)
executor > local (10)
executor > local (11)
executor > local (11)
executor > local (11)
executor > local (11)
executor > local (11)
[6c/876909] process > fasqc (SRR8641368_R) [100%] 1 of 1 ✔
executor > local (13)
[6c/876909] process > fasqc (SRR8641368_R) [100%] 1 of 1 ✔
executor > local (13)
[6c/876909] process > fasqc (SRR8641368_R) [100%] 1 of 1 ✔
[38/2daf75] process > fastp (SRR8641368_R) [100%] 1 of 1 ✔
[09/b36296] process > fastp_stats (SRR8641368_R) [100%] 1 of 1 ✔
[a5/77f26f] process > skip_rrna_removal (SRR8641368_R) [100%] 1 of 1 ✔
[94/71880f] process > normalize_reads (SRR8641368_R) [100%] 1 of 1 ✔
[de/71f78c] process > trinity_assembly (SRR8641368_R) [100%] 1 of 1 ✔
[de/bc6216] process > soap_assembly (SRR8641368_R) [100%] 1 of 1 ✔
[25/457622] process > velvet_oases_assembly (SRR8641368_R) [100%] 1 of 1 ✔
[84/17fe6c] process > rna_spades_assembly (SRR8641368_R) [100%] 1 of 1 ✔
[2e/672733] process > transabyss_assembly (SRR8641368_R) [100%] 1 of 1 ✔
[80/f0f8e6] process > evigene (SRR8641368_R) [ 0%] 0 of 1
[- ] process > rna_quast -
[- ] process > mapping_evigene -
[- ] process > busco4 -
[ca/050e8b] process > mapping_trinity (SRR8641368_R) [ 0%] 0 of 1
[- ] process > summary_evigene_individual -
[- ] process > busco4_tri -
[47/a485e7] process > skip_busco_dist (SRR8641368_R) [100%] 1 of 1, cached: 1 ✔
executor > local (13)
[6c/876909] process > fasqc (SRR8641368_R) [100%] 1 of 1 ✔
[38/2daf75] process > fastp (SRR8641368_R) [100%] 1 of 1 ✔
[09/b36296] process > fastp_stats (SRR8641368_R) [100%] 1 of 1 ✔
[a5/77f26f] process > skip_rrna_removal (SRR8641368_R) [100%] 1 of 1 ✔
[94/71880f] process > normalize_reads (SRR8641368_R) [100%] 1 of 1 ✔
[de/71f78c] process > trinity_assembly (SRR8641368_R) [100%] 1 of 1 ✔
[de/bc6216] process > soap_assembly (SRR8641368_R) [100%] 1 of 1 ✔
[25/457622] process > velvet_oases_assembly (SRR8641368_R) [100%] 1 of 1 ✔
[84/17fe6c] process > rna_spades_assembly (SRR8641368_R) [100%] 1 of 1 ✔
[2e/672733] process > transabyss_assembly (SRR8641368_R) [100%] 1 of 1 ✔
[80/f0f8e6] process > evigene (SRR8641368_R) [ 0%] 0 of 1
[- ] process > rna_quast -
[- ] process > mapping_evigene -
[- ] process > busco4 -
[ca/050e8b] process > mapping_trinity (SRR8641368_R) [ 0%] 0 of 1
[- ] process > summary_evigene_individual -
[- ] process > busco4_tri -
[47/a485e7] process > skip_busco_dist (SRR8641368_R) [100%] 1 of 1, cached: 1 ✔
[- ] process > summary_busco4_individual -
[- ] process > get_busco4_comparison -
[- ] process > transdecoder_longorf -
[- ] process > transdecoder_diamond -
[- ] process > transdecoder_hmmer -
[- ] process > transdecoder_predict -
[- ] process > swiss_diamond_trinotate -
[- ] process > custom_diamond_trinotate -
[- ] process > hmmer_trinotate -
[- ] process > skip_signalP -
[- ] process > skip_tmhmm -
[- ] process > skip_rnammer -
[- ] process > trinotate -
[- ] process > get_GO_comparison -
[- ] process > summary_custom_uniprot -
[- ] process > skip_kegg -
[- ] process > get_transcript_dist -
[- ] process > summary_transdecoder_individual -
[- ] process > summary_trinotate_individual -
[- ] process > get_report -
[98/934751] process > get_run_info [100%] 1 of 1 ✔
Pulling Singularity image docker://null [cache /gpfs/nhmfsa/bulk/share/data/mbl/share/workspaces/users/chril1/TransPiDBs/singularityCache/null.img]
Error executing process > 'busco4_tri (SRR8641368_R)'
Caused by:
Failed to pull singularity image
command: singularity pull --name null.img.pulling.1664887974429 docker://null > /dev/null
status : 255
message:
FATAL: While making image from oci registry: error fetching image to cache: failed to get checksum for docker://null: Error reading manifest latest in docker.io/library/null: errors:
denied: requested access to the resource is denied
unauthorized: authentication required
It seems that it gets all the way through building the four assemblies then crashes when trying to pull a singularity image for BUSCO evaluation. I confess I'm not quite sure how to debug this one - do you have any ideas? I've also tried -profile conda and am getting environment creation conflict errors out of that, and so seem to have reached an impasse.
Thanks very much for your attention,
Chris L
The text was updated successfully, but these errors were encountered: