Differences in results when processing runs in parallel vs all in one go

[Munchic]

Hello Vadim,

I have ~100 related DIA runs that I would like to process separately in the first pass (for parallel compute to speed up time) and then combine together for cross-run analysis. There seems to be considerable differences in IDs when running the whole pipeline sequentially vs in parallel. How can I make the parallel version yield similar results to sequential?

As a small example, for sequential version, all 4 runs were processed together. For parallel version, these 4 runs were processed separately, then combined. I also tried sequential version with individual rt and mass accuracy settings. All of these are run on Linux.

I would like to know if there is something incorrect in the way I set this up and how to make the parallel version have the same (or very similar) results? Thank you so much for your help!

For sequential processing, the command was:

/usr/diann/1.8.1/diann-1.8.1 $file_flag data/$dia_files_cmd --lib $lib --threads 64 --verbose 1 \
  --out "out/report.tsv" --qvalue 0.01 --matrices --temp "out" --out-lib "out/spect_lib.tsv" $gen_spec_lib $predictor --prosit --fasta $db $fasta_search \
  --min-fr-mz 300 --max-fr-mz 1400 $met_excision --cut K*,R* --missed-cleavages 1 \
  --min-pep-len 7 --max-pep-len 30 --min-pr-mz 300 --max-pr-mz 1400 --min-pr-charge 1 --max-pr-charge 4 \
  $unimod4 --var-mods 1 $unimod35 $unimod1 $unimod21 --monitor-mod UniMod:1 \
  --use-quant --reanalyse --smart-profiling --peak-center --no-ifs-removal --global-norm

For parallel processing, the commands were:

# Pass 1 (applied each of the 4 files), "data" here contains only 1 file since it's a separate cluster node. $lib is the spectral library generated from FASTA (the same as the one that the sequential processing has)

diann-1.8.1 --dir "data" --lib $lib --threads 32 --verbose 1 \
  --out "out/report.tsv" --qvalue 0.01 --matrices --temp "data" --prosit  \
  --min-fr-mz 300 --max-fr-mz 1400 $met_excision --cut K*,R* --missed-cleavages 1 \
  --min-pep-len 7 --max-pep-len 30 --min-pr-mz 300 --max-pr-mz 1400 --min-pr-charge 1 --max-pr-charge 4 \
  $unimod4 --var-mods 1 $unimod35 $unimod1 $unimod21 --monitor-mod UniMod:1 \
  --smart-profiling --peak-center --no-ifs-removal --global-norm

# Pass 2 (applied to all 4 files), "data" here contains all 4 .quant files and 4 .mzML files from pass 1

diann-1.8.1 --dir "data" --lib $lib --threads 32 --verbose 1 \
  --out "out/report.tsv" --qvalue 0.01 --matrices --temp "data" --prosit \
  --min-fr-mz 300 --max-fr-mz 1400 $met_excision --cut K*,R* --missed-cleavages 1 \
  --min-pep-len 7 --max-pep-len 30 --min-pr-mz 300 --max-pr-mz 1400 --min-pr-charge 1 --max-pr-charge 4 \
  $unimod4 --var-mods 1 $unimod35 $unimod1 $unimod21 --monitor-mod UniMod:1 \
  --use-quant --reanalyse --smart-profiling --peak-center --no-ifs-removal --global-norm

When looking at stripped sequences, we observe some differences across all runs (~5% unique to either parallel or separate versions):
pr_strip_seq_comparisons.zip

When looking at modified sequences, we observe even larger differences (10-15% unique to either versions):
pr_mod_seq_comparisons.zip

Log for sequential processing:
sequential_log.txt
sequential_unrelated_log.txt (individual rt and mass acc)

Output: sequential_out_matrices.zip

Logs for parallel processing:
Pass 1:
parallel_pass1_file1_log.txt
parallel_pass1_file2_log.txt
parallel_pass1_file3_log.txt
parallel_pass1_file4_log.txt

Pass 2:
parallel_pass2_all_log.txt

Output: parallel_out_matrices.zip

[vdemichev]

Please see https://github.com/vdemichev/DiaNN#output, 'Flexible reanalysis': mass accuracies & scan window must be fixed if you'd like to reuse the .quant files. Please also see the docs on the MBR, if you'd like to use it.