ln: failed to create symbolic link 'OPTIONAL_FILE': File exists

[weishwu]

Operating System

Other Linux (please specify below)

Other Linux

RedHat 7.9 (Maipo)

Workflow Version

v2.9.0-gd9e8593

Workflow Execution

Command line

EPI2ME Version

No response

CLI command run

module load singularity
smp_id="10272-CV-1_A-WGS"

pjdir=/nfs/turbo/lsa-bis/DNAmet/analysis/wgbs/ont_UW/
nf_conf=/nfs/turbo/lsa-bis/DNAmet/analysis/scripts/dorado_basecall_scripts/nextflow_resource.cfg

source /nfs/mm-isilon/bioinfcore/ActiveProjects/weishwu/common/Anaconda3/bin/activate nextflow

bcdir=${pjdir}/analysis/aln/

mkdir -p ${bcdir}
mkdir -p ${bcdir}/${smp_id}

cd ${bcdir}/${smp_id}
export NXF_SINGULARITY_CACHEDIR=/nfs/turbo/lsa-bis/envs/epi2melabs_sifs/

(nextflow run epi2me-labs/wf-basecalling -revision prerelease \
--input /nfs/turbo/lsa-bis/DNAmet/analysis/wgbs/ont_UW/raw_data/10272-CV-1_A-WGS_pod5/ \
--output_bam \
--out_dir ${bcdir}/${smp_id} \
--sample_name ${smp_id} \
--basecaller_cfg dna_r10.4.1_e8.2_400bps_sup@v4.3.0 \
--remora_cfg dna_r10.4.1_e8.2_400bps_sup@v4.3.0_5mCG_5hmCG@v1 \
--dorado_ext pod5 \
--qscore_filter 8 \
--basecaller_basemod_threads 8 \
--ubam_sort_threads 8 \
--merge_threads 4 \
--stats_threads 4 \
-profile singularity \
-c ${nf_conf} ) 2>&1|tee > ${pjdir}/analysis/logs/basecall_aln_sup.${smp_id}.log

Workflow Execution - CLI Execution Profile

singularity

What happened?

The previous run with the same command-line (only with a different sample name) worked out without errors. This run failed after running for 6 days. I guess the main basecalling was completed but some final wrapping-up failed. Is there a way to retrieve the basecall results in one of the output folders?

The log file is attached. The main error message is:

Command error:
  ln: failed to create symbolic link 'OPTIONAL_FILE': File exists

basecall_aln_sup.10272-CV-1_A-WGS.log

Relevant log output

[e3/3b0931] process > output_stream (5)          [ 11%] 1 of 9, failed: 1
ERROR ~ Error executing process > 'output_stream (5)'

Caused by:
  Process `output_stream (5)` terminated with an error exit status (1)

Command executed:

  echo "Writing output files."

Command exit status:
  1

Command output:
  (empty)

Command error:
  ln: failed to create symbolic link 'OPTIONAL_FILE': File exists

Work dir:
  /nfs/turbo/lsa-bis/DNAmet/analysis/wgbs/ont_UW/analysis/aln/10272-CV-1_A-WGS/work/e3/3b093189d518b317c2c0a148ec49ce

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

 -- Check '.nextflow.log' file for details
WARN: Killing running tasks (8)

Application activity log entry

No response

Were you able to successfully run the latest version of the workflow with the demo data?

yes

Other demo data information

No response

[jmcamza]

Hi, were you able to solve this issue?

[weishwu]

@jmcamza I'm rerunning it with exactly the same command-line. And in parallel I'm running a direct basecalling job with Dorado. Do you have any idea what could be the cause of the error?

[jmcamza]

@weishwu I had the same issue and it was solved when I deleted the work folder that is automatically created to store temporary files. After that, the command was run successfully. However, I’m not sure about the root cause, hope this information helps.

[weishwu]

@jmcamza Thanks. My rerun worked out smoothly. I agree that it may be because I didn't delete the previous folder. The problematic run was the first time I ran two basecalling jobs simultaneously, so not sure if that could contribute to it ( I did separate them into two folders though).

[weishwu]

Sorry to reopen this.
It seems that this error is triggered every time I run multiple basecalling jobs simultaneously. Right now I can only run the jobs sequentially which is taking too much time with 40 samples. Please help.
Update: now I keep getting this error even if I run a single job.

@jmcamza Is this what occurred to you also?

[SamStudio8]

Hi @weishwu, apologies for missing this. I've opened a ticket on our side to investigate this and will keep you updated. In the mean time, does using a separate work directory for each workflow instance solve your problem? You can specify a working directory to nextflow run with -w.

[hackkr]

Hi @SamStudio8,

I'm experiencing the same issue when I try to run the workflow, including when I set the working dir with -w. Repeatedly running the demo analysis script with an existing work/ directory works doesn't result in a similar error, so I'm not sure why we are encountering this issue with real data. I wasn't able to analyze the demo data without specifying a reference file, something I and @weishwu were attempting.

my script:

#!/bin/bash

#SBATCH -J basecaller_gpu_test
#SBATCH -p gpu
#SBATCH -A r00910
#SBATCH -o filename_%j.txt
#SBATCH -e filename_%j.err
#SBATCH --mail-type=ALL
#SBATCH --mail-user=rhackley@iu.edu
#SBATCH --gpus=1
#SBATCH --cpus-per-gpu=12
#SBATCH --mem=32G
#SBATCH --time=1:00:00

#Load modules
module load cudatoolkit/12.2 python/gpu/3.10.10 openssl java dorado/0.5.3 singularity/3.7.2

srun /N/slate/rhackley/nextflow run epi2me-labs/wf-basecalling \
    -w /N/slate/rhackley/01_poreC_PB/fast5_pass/sup-wk \
    -profile singularity \
    --input /N/slate/rhackley/01_poreC_PB/fast5_pass \
    --dorado_ext fast5 \
    --out_dir /N/slate/rhackley/01_poreC_PB/fast5_pass/sup \
    --output_bam True \
    --output_pod5 True \
    --sample_name poreC_PB \
    --basecaller_cfg dna_r9.4.1_e8_sup@v3.6

my error output:

ERROR ~ Error executing process > 'output_stream (8)'
Caused by:
  Process `output_stream (8)` terminated with an error exit status (1)
Command executed:
  echo "Writing output files."
Command exit status:
  1
Command output:
  (empty)
Command error:
  ln: failed to create symbolic link 'OPTIONAL_FILE': File exists

demo data w/o reference:

srun /N/slate/rhackley/nextflow run epi2me-labs/wf-basecalling \
    -profile singularity \
    --input /N/slate/rhackley/00_wf-basecalling-demo/input \
    --dorado_ext pod5 \
    --out_dir /N/slate/rhackley/00_wf-basecalling-demo/basecall_test2 \
    --basecaller_cfg dna_r10.4.1_e8.2_400bps_hac@v4.1.0

N E X T F L O W  ~  version 23.10.1
Launching `https://github.com/epi2me-labs/wf-basecalling` [romantic_bhabha] DSL2 - revision: 2e9fce3763 [master]
WARN: NEXTFLOW RECURSION IS A PREVIEW FEATURE - SYNTAX AND FUNCTIONALITY CAN CHANGE IN FUTURE RELEASES

||||||||||   _____ ____ ___ ____  __  __ _____      _       _
||||||||||  | ____|  _ \_ _|___ \|  \/  | ____|    | | __ _| |__  ___
|||||       |  _| | |_) | |  __) | |\/| |  _| _____| |/ _` | '_ \/ __|
|||||       | |___|  __/| | / __/| |  | | |__|_____| | (_| | |_) \__ \
||||||||||  |_____|_|  |___|_____|_|  |_|_____|    |_|\__,_|_.__/|___/
||||||||||  wf-basecalling v1.1.7-g2e9fce3
--------------------------------------------------------------------------------
Core Nextflow options
  revision       : master
  runName        : romantic_bhabha
  containerEngine: singularity
  container      : [withLabel:wf_basecalling:ontresearch/dorado:shac28cd94f2303b0493a4b16ca86e711852c2b8525, withLabel:wf_common:ontresearch/wf-common:sha1c5febff9f75143710826498b093d9769a5edbb9]
  launchDir      : /N/slate/rhackley/00_wf-basecalling-demo
  workDir        : /N/slate/rhackley/00_wf-basecalling-demo/work
  projectDir     : /N/u/rhackley/Quartz/.nextflow/assets/epi2me-labs/wf-basecalling
  userName       : rhackley
  profile        : singularity
  configFiles    : /N/u/rhackley/Quartz/.nextflow/assets/epi2me-labs/wf-basecalling/nextflow.config, /N/slate/rhackley/00_wf-basecalling-demo/nextflow.config

Input Options
  input          : /N/slate/rhackley/00_wf-basecalling-demo/input
  ref            : wf-basecalling-demo/GCA_000001405.15_GRCh38_no_alt_analysis_set.fasta

Output Options
  out_dir        : /N/slate/rhackley/00_wf-basecalling-demo/basecall_test2

Basecalling options
  basecaller_cfg : dna_r10.4.1_e8.2_400bps_hac@v4.1.0
  remora_cfg     : dna_r10.4.1_e8.2_400bps_hac@v4.1.0_5mCG_5hmCG@v2

!! Only displaying parameters that differ from the pipeline defaults !!
--------------------------------------------------------------------------------
If you use epi2me-labs/wf-basecalling for your analysis please cite:

* The nf-core framework
  https://doi.org/10.1038/s41587-020-0439-x

--------------------------------------------------------------------------------
This is epi2me-labs/wf-basecalling v1.1.7-g2e9fce3.
--------------------------------------------------------------------------------
ERROR ~ No such file or directory: /N/slate/rhackley/00_wf-basecalling-demo/wf-basecalling-demo/GCA_000001405.15_GRCh38_no_alt_analysis_set.fasta
 -- Check '.nextflow.log' file for details
ERROR ~ No such file or directory: /N/slate/rhackley/00_wf-basecalling-demo/wf-basecalling-demo/GCA_000001405.15_GRCh38_no_alt_analysis_set.fasta
 -- Check '.nextflow.log' file for details