Error in checkForAndCreateDir

[pkrithivasan]

Hi,

I'm running DROP v1.4.0 on a set of 10 samples, with only the AberrantSplicing and AberrantExpression modules. The AberrantExpression module completes successfully, but I get the following error with the AberrantSplicing module:

[Thu Jun 27 09:57:03 2024]
rule AberrantSplicing_pipeline_Counting_03_filter_expression_FraseR_R:
input: Output/processed_data/aberrant_splicing/datasets/savedObjects/raw-local-fraser/jaccard.h5, Scripts/AberrantSplicing/pipeline/Counting/03_filter_expression_FraseR.R
output: Output/processed_data/aberrant_splicing/datasets/savedObjects/fraser/fds-object.RDS, Output/processed_data/aberrant_splicing/datasets/savedObjects/fraser/filter_FRASER2.done
log: /mnt/disks/cromwell_root/.drop/tmp/AS/fraser/03_filter.Rds
jobid: 38
reason: Missing output files: Output/processed_data/aberrant_splicing/datasets/savedObjects/fraser/filter_FRASER2.done; Input files updated by another job: Output/processed_data/aberrant_splicing/datasets/savedObjects/raw-local-fraser/jaccard.h5
wildcards: dataset=fraser
threads: 3
resources: tmpdir=/mnt/disks/cromwell_root/tmp.8ef94c54

Load packages
Loading assay: rawCountsJ
Loading assay: psi5
Loading assay: psi3
Loading assay: rawOtherCounts_psi5
Loading assay: rawOtherCounts_psi3
Loading assay: rawCountsJnonsplit
Loading assay: jaccard
Loading assay: rawOtherCounts_jaccard
Loading assay: delta_jaccard
Loading assay: delta_psi5
Loading assay: delta_psi3
Loading assay: rawCountsSS
Loading assay: theta
Loading assay: rawOtherCounts_theta
Loading assay: delta_theta
symLink fraser dir
Thu Jun 27 09:57:24 2024: Filtering out introns with low read support ...
Thu Jun 27 09:57:49 2024: Filtering out non-variable introns ...
Thu Jun 27 09:57:53 2024: Filtering done!
Error in checkForAndCreateDir(fds, outDir) :
Can not create workding directory: Output/processed_data/aberrant_splicing/datasets//savedObjects/raw-fraser
Calls: saveFraserDataSet -> checkForAndCreateDir
In addition: Warning message:
In dir.create(dir, recursive = TRUE) :
'Output/processed_data/aberrant_splicing/datasets//savedObjects/raw-fraser' already exists
Execution halted
[Thu Jun 27 09:57:54 2024]
Error in rule AberrantSplicing_pipeline_Counting_03_filter_expression_FraseR_R:
jobid: 38
input: Output/processed_data/aberrant_splicing/datasets/savedObjects/raw-local-fraser/jaccard.h5, Scripts/AberrantSplicing/pipeline/Counting/03_filter_expression_FraseR.R
output: Output/processed_data/aberrant_splicing/datasets/savedObjects/fraser/fds-object.RDS, Output/processed_data/aberrant_splicing/datasets/savedObjects/fraser/filter_FRASER2.done
log: /mnt/disks/cromwell_root/.drop/tmp/AS/fraser/03_filter.Rds (check log file(s) for error details)

RuleException:
CalledProcessError in file /mnt/disks/cromwell_root/tmp.8ef94c54/tmprhayda0h, line 183:
Command 'set -euo pipefail; Rscript --vanilla /mnt/disks/cromwell_root/.snakemake/scripts/tmpcb4w4ev7.03_filter_expression_FraseR.R' returned non-zero exit status 1.
File "/mnt/disks/cromwell_root/tmp.8ef94c54/tmprhayda0h", line 183, in __rule_AberrantSplicing_pipeline_Counting_03_filter_expression_FraseR_R
File "/root/miniconda3/envs/DROP_1.4.0/lib/python3.12/concurrent/futures/thread.py", line 58, in run
Select jobs to execute...

This is what I have in my config file:

projectTitle: Detection of RNA Outlier Pipeline - VAL1-10, AS, AE
htmlOutputPath: Output/html
indexWithFolderName: true
root: Output
sampleAnnotation: sample_annotation_VAL1-10.tsv
geneAnnotation:
v44: gencode.v44.annotation.gtf
genomeAssembly: hg38
genome:
gatk: GRCh38_full_analysis_set_plus_decoy_hla.fa
hpoFile: null
random_seed: false
exportCounts:
geneAnnotations:

• v44

aberrantExpression:
run: true
groups:

• outrider
  fpkmCutoff: 1
  implementation: autoencoder
  padjCutoff: 0.05
  zScoreCutoff: 0
  genesToTest: null
  maxTestedDimensionProportion: 3
  yieldSize: 2000000

aberrantSplicing:
run: true
groups:

• fraser
  recount: true
  longRead: false
  keepNonStandardChrs: false
  filter: true
  minExpressionInOneSample: 20
  quantileMinExpression: 10
  minDeltaPsi: 0.05
  implementation: PCA
  padjCutoff: 0.1
  maxTestedDimensionProportion: 6
  genesToTest: null
  FRASER_version: FRASER2
  deltaPsiCutoff : 0.1
  quantileForFiltering: 0.75

And this is the snakemake command: snakemake --cores 3 -k

Please let me know if there's something that needs to be updated in the config file. Thanks!

[vyepez88]

Hi, hard to say, the error returned non-zero exit status 1 usually appears when there's not enough memory allocated. Can you please check that? Also, you may want to change recount: false, otherwise your samples get recounted every time you execute the aberrantSplicing rule. Finally, add the --rerun-triggers mtime parameter every time you run the aberrantSplicing module

[pkrithivasan]

Hi, thanks for the prompt response. I believe I am providing sufficient memory for the run - I'm running DROP on a machine with 256GB memory and 30cores. I re-ran the same set of 10 samples with recount: false and --re-run-triggers mtime, and still get the same error.
I manually ran the 03_filter_expression_FraseR.R script and ran into the same error. This specific line throws the error:
devNull <- saveFraserDataSet(fds,dir = workingDir)

[ischeller]

Hi, can you check what the output of dir.exists(workingDir) is for your case? It seems there is an issue in recognizing that the directory already exists for some reason.
Also, what version of FRASER are you using?

[pkrithivasan]

Hi,
This is the FRASER version: FRASER_1.99.4
and the directory does exist:
dir.exists("Output/processed_data/aberrant_splicing/datasets/") [1] TRUE

[pkrithivasan]

Also, I should add that when I run the same module on a single sample with external counts from 10 other samples, this AberrantSplicing_pipeline_Counting_03_filter_expression_FraseR_R script runs without error.

[Jakob37]

I believe I am running into the same error (DROP version 1.3.3). I have managed to run through the splicing module previously when using external counts. In this run I only have a set of ~100 non-external samples. The AE module went through without external counts.

Looking in htop while running, it looks like there is plenty of RAM available so I don't think this is a memory issue.

A bit up in the R script I see that it is symlinking if no external counts are present. Can it be related to this? Seems saveFraserDataSet in the FRASER package somehow collides with this.

# Add external data if provided by dataset
if(length(exCountIDs) > 0){
  ...
} else {
    message("symLink fraser dir")
    file.symlink(paste0(workingDir, "savedObjects/","raw-local-", dataset),
                 paste0(workingDir, "savedObjects/","raw-", dataset))
    
    fds@colData$isExternal <- as.factor(FALSE)
    workingDir(fds) <- workingDir
    name(fds) <- paste0("raw-", dataset)
}

The singularity container I am using to run DROP uses FRASER_1.99.3.

Here is my stack trace:

Fri Aug 16 12:18:54 2024: Filtering out introns with low read support ...
Fri Aug 16 12:20:38 2024: Filtering out non-variable introns ...
Fri Aug 16 12:21:10 2024: Filtering done!
Error in checkForAndCreateDir(fds, outDir) :
  Can not create workding directory: output/processed_data/aberrant_splicing/datasets//savedObjects/raw-fraser
Calls: saveFraserDataSet -> checkForAndCreateDir
In addition: Warning message:
In dir.create(dir, recursive = TRUE) :
  'output/processed_data/aberrant_splicing/datasets//savedObjects/raw-fraser' already exists
Execution halted
[Fri Aug 16 12:21:11 2024]
Error in rule AberrantSplicing_pipeline_Counting_03_filter_expression_FraseR_R:
    jobid: 3
    input: output/processed_data/aberrant_splicing/datasets/savedObjects/raw-local-fraser/jaccard.h5, Scripts/AberrantSplicing/pipeline/Counting/03_filter_expression_FraseR.R
    output: output/processed_data/aberrant_splicing/datasets/savedObjects/fraser/fds-object.RDS, output/processed_data/aberrant_splicing/datasets/savedObjects/fraser/filter_FRASER2.done
    log: <work>/.drop/tmp/AS/fraser/03_filter.Rds (check log file(s) for error details)

RuleException:
CalledProcessError in file /tmp/tmpuq_thm5b, line 89:
Command 'set -euo pipefail;  Rscript --vanilla <work>/.snakemake/scripts/tmporwzmnov.03_filter_expression_FraseR.R' returned non-zero exit status 1.
  File "/tmp/tmpuq_thm5b", line 89, in __rule_AberrantSplicing_pipeline_Counting_03_filter_expression_FraseR_R
  File "/opt/conda/lib/python3.11/concurrent/futures/thread.py", line 58, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2024-08-16T121828.769496.snakemake.log

[Jakob37]

Any thoughts on this one? This is the one step blocking us at the moment from evaluating the DROP output in our data.

[AtaJadidAhari]

Hi @Jakob37,

I cannot reproduce this issue and haven't seen it before. My guess is that if you remove the directory (output/processed_data/aberrant_splicing/datasets/savedObjects/raw-fraser) and re-run the pipeline, it will run successfully.

[Jakob37]

Hi @Jakob37,

I cannot reproduce this issue and haven't seen it before. My guess is that if you remove the directory (output/processed_data/aberrant_splicing/datasets/savedObjects/raw-fraser) and re-run the pipeline, it will run successfully.

Hello @AtaJadidAhari and thank you for taking the time to look into this.

I digged into this a bit more. I could indeed reproduce the error, and also tried removing the soft-link as you mentioned, which did not solve it.

The issue seems to be that the soft link to the raw-local-fraser folder is soft-linked using the wrong working directory. This yields an invalid link as such:

<workdir>/output/processed_data/aberrant_splicing/datasets/savedObjects $ ls -l
lrwxrwxrwx 1 jakob cmd-bnf 78 Aug 22 20:31 raw-fraser -> output/processed_data/aberrant_splicing/datasets/savedObjects/raw-local-fraser
drwxr-xr-x 3 jakob cmd-bnf 19 Aug 22 20:31 raw-local-fraser

This causes the crash as such:

• saveFraserDataset in saveHDF5Objects.R is called, which in turn calls checkForAndCreateDir in helper-functions.R
• Here, the dir is now a symlink pointing to an empty location.
• dir.exists(dir) will yield FALSE on the broken symlink, but will fail to create the dir with a warning.
• Finally the "Can not create workding directory" message is triggered from the code below.

checkForAndCreateDir <- function(object, dir){
    verbose <- 0
    if(is(object, "FraserDataSet")){
        verbose <- verbose(object)
        if(missing(dir)){
            dir <- workingDir(object)
        }
    }
    if(!dir.exists(dir)){
        if(verbose > 1){
            message(date(), ": The given working directory '", 
                    dir, "' does not exists. We will create it.")
        }
        dir.create(dir, recursive=TRUE)
    }
    if(!dir.exists(dir)){
        stop("Can not create workding directory: ", dir)
    }
    return(TRUE)
}

I tried hard-coding in the absolute path for the work dir into the DROP-script, and then the step successfully went through.

Unsure why it sometimes works then, as it works for you. Might be a version issue? (Edit: I see now that the author of the issue has the same issue with the latest version). Or something with running it through SLURM. I am using a container running DROP 1.3.3 (running this through the nextflow pipeline Tomte).

[Jakob37]

OK, after some more digging I think I have figured out the issue.

It seems the current code will only work if provided an absolute path. If given a relative path, it will crash here.

The working dir is calculated as such:

#'   - workingDir: '`sm cfg.getProcessedDataDir() + "/aberrant_splicing/datasets/"`'

The cfg.getProcessedDataDir() value is assign as such:

self.processedDataDir = self.root / "processed_data"

I suspect self.root comes from the config.yaml file.

At the moment (as generated by the Nextflow pipeline I am running), it is provided as such, i.e. with a relative path:

root: output

I think we could fix this in the Tomte pipeline, such that this isn't an issue.

It would be helpful though with an early error for relative paths if not allowed.

[vyepez88]

Thanks @Jakob37 for digging out! We'll now see on our side how to prevent this.