drop demo not working

[gevro]

Hi, The command 'drop demo' in the latest conda version of drop is giving this error:

resource/dna_vcf/ALL.chr21.phase3_shapeit2_mvncall_integrated_v5a.20130502.genotypes.exome.vcf.gz.tbi
resource/dna_vcf/demo_chr21.vcf.gz.tbi
resource/dna_vcf/demo_chr21.vcf.gz
resource/dna_vcf/ALL.chr21.phase3_shapeit2_mvncall_integrated_v5a.20130502.genotypes.exome.vcf.gz
resource/external_geneCounts.tsv.gz
resource/hpo_genes.tsv.gz
Traceback (most recent call last):
File "/gpfs/home/evrong01/evronylab/bin/drop-conda/bin/drop", line 10, in
sys.exit(main())
File "/gpfs/share/apps/python/cpu/3.6.5/lib/python3.6/site-packages/click/core.py", line 829, in call
return self.main(*args, **kwargs)
File "/gpfs/share/apps/python/cpu/3.6.5/lib/python3.6/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/gpfs/share/apps/python/cpu/3.6.5/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
return process_result(sub_ctx.command.invoke(sub_ctx))
File "/gpfs/share/apps/python/cpu/3.6.5/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/gpfs/share/apps/python/cpu/3.6.5/lib/python3.6/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/gpfs/home/evrong01/evronylab/bin/drop-conda/lib/python3.6/site-packages/drop/cli.py", line 87, in demo
dict[key] = str(pathlib.Path(dict_[key]).resolve())
File "/gpfs/share/apps/python/cpu/3.6.5/lib/python3.6/site-packages/pathlib-1.0.1-py3.6.egg/pathlib.py", line 1034, in resolve

File "/gpfs/share/apps/python/cpu/3.6.5/lib/python3.6/site-packages/pathlib-1.0.1-py3.6.egg/pathlib.py", line 320, in resolve
if newpath in seen:
File "/gpfs/share/apps/python/cpu/3.6.5/lib/python3.6/site-packages/pathlib-1.0.1-py3.6.egg/pathlib.py", line 305, in _resolve
accessor = path._accessor
File "/gpfs/share/apps/python/cpu/3.6.5/lib/python3.6/site-packages/pathlib-1.0.1-py3.6.egg/pathlib.py", line 398, in readlink
lstat = _wrap_strfunc(os.lstat)
FileNotFoundError: [Errno 2] No such file or directory: '/gpfs/scratch/evrong01/dropdemo/Output'

[mumichae]

Thanks for pointing that out. It should be fixed now, if you redownload your demo.

[gevro]

I tried reinstalling drop conda and running 'drop demo' in a new directory, but I'm getting the same error.

[mumichae]

I couldn't reproduce the error. Which config files does your Data/ directory contain?

[gevro]

I have these files in the Data directory:

-rwxrwx--- 1 evrong01 evrong01 47M Nov 19 2019 chr21.fa
-rw-rw---- 1 evrong01 evrong01 23 Jun 1 03:19 chr21.fa.fai
-rw-rw---- 1 evrong01 evrong01 1.4K Jul 6 23:18 config_old.yaml
-rw-rw---- 1 evrong01 evrong01 1.3K Sep 10 21:53 config_relative_wb1.8.yaml
-rw-rw---- 1 evrong01 evrong01 1.3K Sep 10 21:55 config_relative.yaml
drwxrwx--- 2 evrong01 evrong01 4.0K Jul 8 16:35 dna_vcf
-rw-rw---- 1 evrong01 evrong01 4.8K Jul 23 00:19 external_geneCounts.tsv.gz
-rwxrwx--- 1 evrong01 evrong01 348 Jul 29 02:33 fix_sample_anno.py
-rwxrwx--- 1 evrong01 evrong01 15M Nov 15 2019 gencode_annotation_trunc.gtf
-rw-rw---- 1 evrong01 evrong01 7.9M Mar 9 2020 hpo_genes.tsv.gz
-rwxrwx--- 1 evrong01 evrong01 227K Oct 14 2019 qc_vcf_1000G.vcf.gz
-rwxrwx--- 1 evrong01 evrong01 136K Oct 14 2019 qc_vcf_1000G.vcf.gz.tbi
-rw-rw---- 1 evrong01 evrong01 862 Apr 12 23:07 readme.md
drwxrwx--- 2 evrong01 evrong01 4.0K Jul 8 16:35 rna_bam
-rw-rw---- 1 evrong01 evrong01 6.1K Jul 29 01:00 sample_annotation_external.tsv
-rwxrwx--- 1 evrong01 evrong01 3.5K Jul 31 15:35 sample_annotation_relative.tsv
-rw-rw---- 1 evrong01 evrong01 5.1K Sep 12 01:17 sample_annotation.tsv

[mumichae]

Ok, the files seem to be fine.
Assuming that you're getting exactly the same error message as above, there seems to be an issue with one files in your config.yaml. This seems strange though, as I could reproduce the error before, when one of the files in Data/ broke the code. Now that I fixed it, you shouldn't get that error anymore. Can you please double-check if you're getting the same error message or a different one?

[mumichae]

Or maybe some of the files can't be read. Could you double-check to see if you have proper permissions to the files in the config?

[gevro]

Here is the config.yaml made by 'drop demo'. And below that is the error.
I think the 'drop demo' code is not making the correct directories. If I manually make the Output and html directories, then it doesn't give the errors anymore.

dropdemo]$ cat config.yaml
projectTitle: Detection of RNA Outlier Pipeline
htmlOutputPath: Output/html
indexWithFolderName: true
fileRegex: .*.R

root: Output
sampleAnnotation: Data/sample_annotation.tsv
geneAnnotation:
v29: Data/gencode_annotation_trunc.gtf
genomeAssembly: hg19
scanBamParam: null
exportCounts:
geneAnnotations:
- v29
excludeGroups:
- mae

aberrantExpression:
groups:
- outrider
fpkmCutoff: 1
implementation: autoencoder
padjCutoff: 1
zScoreCutoff: 0
maxTestedDimensionProportion: 3
dassie:
tssWindow: 500
pasWindow: 1000

aberrantSplicing:
groups:
- fraser
recount: true
longRead: false
keepNonStandardChrs: true
filter: false
minExpressionInOneSample: 20
minDeltaPsi: 0.05
implementation: PCA
padjCutoff: 1
zScoreCutoff: 0
deltaPsiCutoff : 0.05
maxTestedDimensionProportion: 6

mae:
groups:
- mae
genome: Data/chr21.fa
gatkIgnoreHeaderCheck: true
padjCutoff: .05
allelicRatioCutoff: 0.8
addAF: false
maxAF: .001
maxVarFreqCohort: .04
# VCF-BAM matching
qcVcf: Data/qc_vcf_1000G.vcf.gz
qcGroups:
- mae

tools:
gatkCmd: gatk
bcftoolsCmd: bcftools
samtoolsCmd: samtools

Here is the error again. It is the same error:
Traceback (most recent call last):
File "/gpfs/home/evrong01/evronylab/bin/drop-conda/bin/drop", line 10, in
sys.exit(main())
File "/gpfs/share/apps/python/cpu/3.6.5/lib/python3.6/site-packages/click/core.py", line 829, in call
return self.main(*args, **kwargs)
File "/gpfs/share/apps/python/cpu/3.6.5/lib/python3.6/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/gpfs/share/apps/python/cpu/3.6.5/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
return process_result(sub_ctx.command.invoke(sub_ctx))
File "/gpfs/share/apps/python/cpu/3.6.5/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/gpfs/share/apps/python/cpu/3.6.5/lib/python3.6/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/gpfs/home/evrong01/evronylab/bin/drop-conda/lib/python3.6/site-packages/drop/cli.py", line 87, in demo
dict[key] = str(pathlib.Path(dict_[key]).resolve())
File "/gpfs/share/apps/python/cpu/3.6.5/lib/python3.6/site-packages/pathlib-1.0.1-py3.6.egg/pathlib.py", line 1034, in resolve

File "/gpfs/share/apps/python/cpu/3.6.5/lib/python3.6/site-packages/pathlib-1.0.1-py3.6.egg/pathlib.py", line 320, in resolve
if newpath in seen:
File "/gpfs/share/apps/python/cpu/3.6.5/lib/python3.6/site-packages/pathlib-1.0.1-py3.6.egg/pathlib.py", line 305, in _resolve
accessor = path._accessor
File "/gpfs/share/apps/python/cpu/3.6.5/lib/python3.6/site-packages/pathlib-1.0.1-py3.6.egg/pathlib.py", line 398, in readlink
lstat = _wrap_strfunc(os.lstat)
FileNotFoundError: [Errno 2] No such file or directory: '/gpfs/scratch/evrong01/dropdemo/Output'

[mumichae]

As the error occurs when converting relative paths to absolute paths using pathlib.Path.resolve, the FileNotFoundError would only occur under python < 3.6 if the Output directory doesn't exists. It seems you are using python 3.6.5, which could potentially cause the issue.

By default, when I install the environment I get python 3.6.11. You might not be using your conda environment but a different instance of python. If you are using the conda environment, try updating python and see if it solves the problem.

[gevro]

I just checked and the drop conda environment python is 3.6.11.

Is it possible your fix yesterday did not get pushed to the conda repository?
Yesterday, I reinstalled like this:
'conda install -c conda-forge -c bioconda drop'

[mumichae]

The fix I made was not in the code base but rather in the downloadable demo input. It was fine, as the correct files are present in Data/. This is a different issue.

[mumichae]

I just noticed that the python prefixes were different from one of your other git issues, so I was assuming you changed which python you were using

[mumichae]

Ah no, I see that they are the same.Still puzzled where this error comes from

[mumichae]

When creating a new project, you will have to fill in the file paths by hand anyway, and this problem is specific to the convenience function of drop demo. The strange thing is that this part of the code did not cause any errors before and I cannot reproduce it. So it is very difficult for me to find the cause.

I suggest, in order to get it running, you should fill in the absolute paths to in the config.yaml by hand. Everything else should work as before.

[gevro]

Ok, I manually created the Output and Output/html directories. Then I ran drop demo, and now it creates the project.

Then I ran snakemake -n but I get a new error:
(/gpfs/home/evrong01/bin/drop-conda) [evrong01@bigpurple-ln3 dropdemo]$ snakemake -n
check for missing R packages
TypeError in line 7 of /gpfs/scratch/evrong01/dropdemo/Snakefile:
expected str, bytes or os.PathLike object, not PosixPath
File "/gpfs/scratch/evrong01/dropdemo/Snakefile", line 7, in
File "/gpfs/home/evrong01/bin/drop-conda/lib/python3.6/site-packages/drop/setupDrop.py", line 10, in setupDrop
File "/gpfs/home/evrong01/bin/drop-conda/lib/python3.6/site-packages/drop/setupDrop.py", line 27, in installRPackages
File "/gpfs/home/evrong01/bin/drop-conda/lib/python3.6/subprocess.py", line 423, in run
File "/gpfs/home/evrong01/bin/drop-conda/lib/python3.6/subprocess.py", line 729, in init
File "/gpfs/home/evrong01/bin/drop-conda/lib/python3.6/subprocess.py", line 1295, in _execute_child

[gevro]

This is the config.yaml now in the demo directory. Seems that the demo itself is broken.

projectTitle: Detection of RNA Outlier Pipeline
htmlOutputPath: /gpfs/scratch/evrong01/dropdemo/Output/html
indexWithFolderName: true
fileRegex: .*.R
root: /gpfs/scratch/evrong01/dropdemo/Output
sampleAnnotation: /gpfs/scratch/evrong01/dropdemo/Data/sample_annotation.tsv
geneAnnotation:
v29: /gpfs/scratch/evrong01/dropdemo/Data/gencode_annotation_trunc.gtf
genomeAssembly: hg19
scanBamParam: null
exportCounts:
geneAnnotations:

• v29
  excludeGroups:
• mae
  aberrantExpression:
  groups:
• outrider
  fpkmCutoff: 1
  implementation: autoencoder
  padjCutoff: 1
  zScoreCutoff: 0
  maxTestedDimensionProportion: 3
  dassie:
  tssWindow: 500
  pasWindow: 1000
  aberrantSplicing:
  groups:
• fraser
  recount: true
  longRead: false
  keepNonStandardChrs: true
  filter: false
  minExpressionInOneSample: 20
  minDeltaPsi: 0.05
  implementation: PCA
  padjCutoff: 1
  zScoreCutoff: 0
  deltaPsiCutoff: 0.05
  maxTestedDimensionProportion: 6
  mae:
  groups:
• mae
  genome: /gpfs/scratch/evrong01/dropdemo/Data/chr21.fa
  gatkIgnoreHeaderCheck: true
  padjCutoff: 0.05
  allelicRatioCutoff: 0.8
  addAF: false
  maxAF: 0.001
  maxVarFreqCohort: 0.04
  qcVcf: /gpfs/scratch/evrong01/dropdemo/Data/qc_vcf_1000G.vcf.gz
  qcGroups:
• mae
  tools:
  gatkCmd: gatk
  bcftoolsCmd: bcftools
  samtoolsCmd: samtools

[mumichae]

there is an error when drop tried to install r packages. However, you should already have all the necessary R packages in your environment. Please make sure you're using the correct R path, libPath of the env

[gevro]

I'm using the correct libPath, but it is still not working.

.libPaths()
[1] "/gpfs/data/bin/drop-conda/lib/R/library"

[gevro]

Also, note this problem will happen to everyone, because conda by default sets the .libPaths() to first have the local R library, not the conda R library. To fix this, you need to add a script in conda activate.d to set $R_LIBS to the conda R library.
conda-forge/r-base-feedstock#37

Regardless, the drop demo error is still happening.

[mumichae]

I can't reproduce this error on a virtual machine (travis) or in a docker image (continuumio/miniconda3). There's probably something specific to your system that causes issues with linking conda to its R. I'd suggest checking out other resources to troubleshoot this problem.