SegFaults on Linux DiaNN v. 1.8

[MatteoLacki]

Hello Vadim,

We have observed some seg-faults (not entirely determinisitic) while testing the linux version of version 1.8.
Basically, after installing the binary, I am running a bash scripts like this:

DATAFOLDER=/home/matteo/Projects/DiaNN/ramdisk
DATAFOLDERS="--f $DATAFOLDER/HYE_A_200ng_100min_dia_Slot1-46_1_5000.d --f $DATAFOLDER/HYE_A_200ng_100min_dia_Slot1-46_1_5001.d --f $DATAFOLDER/HYE_A_200ng_100min_dia_Slot1-46_1_5002.d --f $DATAFOLDER/HYE_A_200ng_100min_dia_Slot1-46_1_5000.d --f $DATAFOLDER/HYE_B_200ng_100min_dia_Slot1-47_1_5005.d --f $DATAFOLDER/HYE_B_200ng_100min_dia_Slot1-47_1_5005.d"
WD=/home/matteo/Projects/DiaNN
FASTAPATH=$WD/fastas/hye_30867_conts_172_2020-3-2_2-0-11.fasta
DIANN=/usr/diann/1.8/diann-1.8
RESULTS=$WD/RESULTS_RAMDISK

ARGS="--verbose 1 --qvalue 0.01 --matrices --gen-spec-lib --predictor --fasta $FASTAPATH --fasta-search --min-fr-mz 200 --max-fr-mz 1800 --met-excision --cut K*,R*,!*P --missed-cleavages 1 --min-pep-len 7 --max-pep-len 30 --min-pr-mz 300 --max-pr-mz 1800 --min-pr-charge 1 --max-pr-charge 4 --unimod4 --reanalyse --smart-profiling --pg-level 2 --species-genes"

for THREADS in 16 32 64 96 128
do
RES=$RESULTS/threads_$THREADS
mkdir -p $RES
$DIANN $DATAFOLDERS $ARGS --threads $THREADS --out $RES/demo.report.tsv --out-lib $RES/demo_lib.tsv
done

For 32 threads (in this case only) it resulted in

DIA-NN 1.8 (Data-Independent Acquisition by Neural Networks)                                                                                                                                                                                                   
Compiled on Jun 28 2021 10:59:57                                                                                                                                                                                                                               
Current date and time: Wed Jul  7 21:49:44 2021                                                                                                                                                                                                                
Logical CPU cores: 128                                                                                                                                                                                                                                         
Output will be filtered at 0.01 FDR                                                                                                                                                                                                                            
Precursor/protein x samples expression level matrices will be saved along with the main report                                                                                                                                                                 
A spectral library will be generated                                                                                                                                                                                                                           
Deep learning will be used to generate a new in silico spectral library from peptides provided                                                                                                                                                                 
Library-free search enabled                                                                                                                                                                                                                                    
Min fragment m/z set to 200                                                                                                                                                                                                                                    
Max fragment m/z set to 1800                                                                                                                                                                                                                                   
N-terminal methionine excision enabled                                                                                                                                                                                                                         
In silico digest will involve cuts at K*,R*                                                                                                                                                                                                                    
But excluding cuts at *P                                                                                                                                                                                                                                       
Maximum number of missed cleavages set to 1                                                                                                                                                                                                                    
Min peptide length set to 7                                                                                                                                                                                                                                    
Max peptide length set to 30                                                                                                                                                                                                                                   
Min precursor m/z set to 300                                                                                                                                                                                                                                   
Max precursor m/z set to 1800                                                                                                                                                                                                                                  
Min precursor charge set to 1                                                                                                                                                                                                                                  
Max precursor charge set to 4                                                                                                                                                                                                                                  
Cysteine carbamidomethylation enabled as a fixed modification                                                                                                                                                                                                  
A spectral library will be created from the DIA runs and used to reanalyse them; .quant files will only be saved to disk during the first step                                                                                                                 
When generating a spectral library, in silico predicted spectra will be retained if deemed more reliable than experimental ones                                                                                                                                
Implicit protein grouping: genes; this determines which peptides are considered 'proteotypic' and thus affects protein FDR calculation                                                                                                                         
Species suffix will be added to gene annotation; this affects proteotypicity definition                                                                                                                                                                        
Thread number set to 32                                                                                                                                                                                                                                        
DIA-NN will optimise the mass accuracy automatically using the first run in the experiment. This is useful primarily for quick initial analyses, when it is not yet known which mass accuracy setting works best for a particular acquisition scheme.          
                                                                                                                                                                                                                                                               
6 files will be processed                                                                                                                                                                                                                                      
[0:00] Loading FASTA /home/matteo/Projects/DiaNN/fastas/hye_30867_conts_172_2020-3-2_2-0-11.fasta                                                                                                                                                              
[0:03] Processing FASTA                                                                                                                                                                                                                                        
[0:08] Assembling elution groups                                                                                                                                                                                                                               
[0:13] 4581738 precursors generated                                                                                                                                                                                                                            
[0:13] Gene names missing for some isoforms                                                                                                                                                                                                                    
[0:13] Library contains 30883 proteins, and 30497 genes                                                                                                                                                                                                        
[0:13] Encoding peptides for spectra and RTs prediction                                                                                                                                                                                                        
[0:19] Predicting spectra and IMs                                                                                                                                                                                                                              
Segmentation fault (core dumped) 

You might have noticed that I kept data on a ram-disk: don't worry about getting out of RAM: we have 256GB of it, so that should not play a role.
Also without ramdisk I have seen a seg-fault, while running 128 threads.
What should I do to help you eliminate that bug?

Best wishes,
Matteo Lacki

[vdemichev]

Hi Matteo,

Thanks for letting me know. Temporary workaround: option 1: run prediction on Windows and then use the resulting .predicted.speclib file on Linux, option 2: just rerun until .predicted.speclib is generated correctly. I will look into this, maybe it's a DIA-NN bug, maybe pytorch bug (DIA-NN relies on pytorch for this prediction step).

Best,
Vadim

[MatteoLacki]

Hello Vadim,

After a longer break I would like to come back to this problem with segfaults in the prediction step.
Unfortunately, option 2: i.e. reruning multiple times the prediction fails each time on linux for me (Ubuntu 20.04.3 LTS
, Focal). I am running the following command:

/usr/diann/1.8/diann-1.8  --f /home/matteo/raw_data/external/opentims_validation/60SPD_py8/20200428_Evosep_60SPD_SG06-16_MLHeLa_200ng_py8_S3-A4_1_2450.d
 --threads 7  --out /home/matteo/Projects/Pipelines/diann/results/20200428_Evosep_60SPD_SG06-16_MLHeLa_200ng_py8_S3-A4_1_2450.d/demo.report.tsv 
--out-lib /home/matteo/Projects/Pipelines/diann/results/20200428_Evosep_60SPD_SG06-16_MLHeLa_200ng_py8_S3-A4_1_2450.d/demo_lib.tsv
 --fasta /home/matteo/Projects/Pipelines/ProcessingSoftware/DIANN/ecoli.fasta 
--verbose 1 --qvalue 0.01 --matrices --gen-spec-lib --predictor --fasta-search --min-fr-mz 200 
--max-fr-mz 1800 --met-excision --cut K*,R*,!*P --missed-cleavages 1 --min-pep-len 7 --max-pep-len 30
--min-pr-mz 300 --max-pr-mz 1800 --min-pr-charge 1 --max-pr-charge 4 --unimod4 --reanalyse --smart-profiling
--pg-level 2 --species-genes

If I run this command, the core dumps happen at different stages of the algorithm.
Here are 2 examples:

DIA-NN 1.8 (Data-Independent Acquisition by Neural Networks)
Compiled on Jun 28 2021 10:59:57
Current date and time: Wed Dec 15 15:34:44 2021
Logical CPU cores: 128
Thread number set to 32
Output will be filtered at 0.01 FDR
Precursor/protein x samples expression level matrices will be saved along with the main report
A spectral library will be generated
Deep learning will be used to generate a new in silico spectral library from peptides provided
Library-free search enabled
Min fragment m/z set to 200
Max fragment m/z set to 1800
N-terminal methionine excision enabled
In silico digest will involve cuts at K*,R*
Maximum number of missed cleavages set to 1
Min peptide length set to 7
Max peptide length set to 30
Min precursor m/z set to 300
Max precursor m/z set to 1800
Min precursor charge set to 1
Max precursor charge set to 4
Cysteine carbamidomethylation enabled as a fixed modification
A spectral library will be created from the DIA runs and used to reanalyse them; .quant files will only be saved to disk during the first step
When generating a spectral library, in silico predicted spectra will be retained if deemed more reliable than experimental ones
Implicit protein grouping: genes; this determines which peptides are considered 'proteotypic' and thus affects protein FDR calculation
Species suffix will be added to gene annotation; this affects proteotypicity definition
DIA-NN will optimise the mass accuracy automatically using the first run in the experiment. This is useful primarily for quick initial analyses, when it is not yet known which mass accuracy setting works best for a particular acquisition scheme.
WARNING: MBR turned off, two or more raw files are required

1 files will be processed
[0:00] Loading FASTA /mnt/ms/old/data/SYMPHONY_VODKAS/fastas/latest/2020-3-2_2-0-11/PEAKS/human_20365_conts_172_2020-3-2_2-0-11.fasta
[0:02] Processing FASTA
[0:06] Assembling elution groups
[0:09] 3518143 precursors generated
[0:09] Gene names missing for some isoforms
[0:09] Library contains 20487 proteins, and 20105 genes
[0:09] Encoding peptides for spectra and RTs prediction
[0:14] Predicting spectra and IMs

It took few minutes to get the segfault.
Now, the above looks like an error that occurs during library generation.
Enters the second try:

DIA-NN 1.8 (Data-Independent Acquisition by Neural Networks)                                                       
Compiled on Jun 28 2021 10:59:57 
Current date and time: Wed Dec 15 15:39:16 2021                                                                    
Logical CPU cores: 128                                                                                             
Thread number set to 32                                                                                            
Output will be filtered at 0.01 FDR                                                                                
Precursor/protein x samples expression level matrices will be saved along with the main report                     
A spectral library will be generated                                                                               
Deep learning will be used to generate a new in silico spectral library from peptides provided                     
Library-free search enabled                                                                                        
Min fragment m/z set to 200                                                                                        
Max fragment m/z set to 1800                                                                                       
N-terminal methionine excision enabled
In silico digest will involve cuts at K*,R*                                                                        
Maximum number of missed cleavages set to 1
Min peptide length set to 7                                                                                        
Max peptide length set to 30                                                                                       
Min precursor m/z set to 300    
Max precursor m/z set to 1800                  
Min precursor charge set to 1                            
Max precursor charge set to 4                            
Cysteine carbamidomethylation enabled as a fixed modification                                                      
A spectral library will be created from the DIA runs and used to reanalyse them; .quant files will only be saved to disk during the first step                              
When generating a spectral library, in silico predicted spectra will be retained if deemed more reliable than experimental ones               
Implicit protein grouping: genes; this determines which peptides are considered 'proteotypic' and thus affects protein FDR calculation                                      
Species suffix will be added to gene annotation; this affects proteotypicity definition                            
DIA-NN will optimise the mass accuracy automatically using the first run in the experiment. This is useful primarily for quick initial analyses, when it is not yet known which mass accuracy setting works best for a particular acquisition scheme.            
WARNING: MBR turned off, two or more raw files are required                                                        
                                                         
1 files will be processed                                
[0:00] Loading FASTA /mnt/ms/old/data/SYMPHONY_VODKAS/fastas/latest/2020-3-2_2-0-11/PEAKS/human_20365_conts_172_2020-3-2_2-0-11.fasta                                       
[0:02] Processing FASTA                                                                                            
[0:06] Assembling elution groups                                                                                   
[0:09] 3518143 precursors generated                      
[0:09] Gene names missing for some isoforms                                                                        
[0:09] Library contains 20487 proteins, and 20105 genes  
[0:09] Encoding peptides for spectra and RTs prediction                                                            
[0:14] Predicting spectra and IMs                        
[6:21] Predicting RTs                                                                                              
[6:51] Decoding predicted spectra and IMs                                                                          
[6:53] Decoding RTs                                                                                                
[6:55] Saving the library to /mnt/ms/new/processed/diaNN/devel/test_pipeline_on_multiple_nodes/M210903_008_1_1_4704.d/demo_lib.predicted.speclib                                                                                      
[7:05] Initialising library 
                            
[7:09] File #1/1                                                                                                   
[7:09] Loading run /mnt/ms/old/rawdata/majestix/ARCHIVIERT/M2109/M210903_008_1_1_4704.d                            
For most diaPASEF datasets it is better to manually fix both the MS1 and MS2 mass accuracies to 10 ppm.  

Now, this looks more troubling, because it would seem that this occurs after the library has been created.
And it was, as much as I can tell:

matteo@Peaks-Core3:/mnt/ms/new/processed/diaNN/devel/test_pipeline_on_multiple_nodes/M210903_008_1_1_4704.d$ ls -lh
total 609M
-rwxrwxrwx 1 root new_server_users 905M Dez 15  2021 demo_lib.predicted.speclib

Say I would be interested, for the time being, to run prediction stage on Windows.
This won't solve that second problem, will it?

I don't know, maybe it is somehow related to me trying to push in only one raw folder.
But I want to have one folder, to have the flexible reanalysis mentioned in another thread I posted some time ago.

Now, after this bug report, a small question: how should I cut my command to make DiaNN only do the library generation without any additional step later on? Is that possible?

Best wishes,

MatteoLacki, AG Tenzer

[dingxuan1990]

By deleting "--reanalyse", you can do the library generation step only according to my experience.
With regard to your segfalt, I did not encounter such a problem. I have successfully used 699 DIA raw files to build a library by using lib-free method on linux and the result turned to be very good. You can refer to my code as below to check if there is anything of help.

/usr/diann/1.8/diann-1.8 --f 1.mzML --f 2.mzML --lib "" --threads 64 --verbose 1 --out "/storage/1/1.tsv" --qvalue 0.01 --matrices --out-lib "/storage/1/1.tsv" --gen-spec-lib --predictor --fasta /storage/1/1.fasta --fasta-search --min-fr-mz 200 --max-fr-mz 1800 --met-excision --cut K*,R* --missed-cleavages 1 --min-pep-len 7 --max-pep-len 30 --min-pr-mz 300 --max-pr-mz 1800 --min-pr-charge 1 --max-pr-charge 4 --unimod4 --var-mods 1 --var-mod UniMod:35,15.994915,M --individual-mass-acc --individual-windows --id-profiling --pg-level 1

[MatteoLacki]

Hi,

So, essentially you run it once with an input of 699 DIA raw files, right?
Are you by chance running this on a virtual machine? If so, could you share the iso?

For last 2 days I am extensively testing DiaNN on linux that are as small as they get, on a type I hypervisor (proxmox).
Essentially, I have been running it on Ubuntu Server 20.04 VM and on Ubuntu 20.04 linux container (both with qemu and without it).
As of now, I have seen one segfault only on the container, no errors on VM.

I am now making a windows VM and try the software there, for comparison. Likely it will work perfectly, as on Windows 10 my group have already analysed few hundred timsTOF folders.
I will try to strip down that Windows 10 to some bare minimum.
It might be a good idea to somehow share such images, so that the community could have a one-click solutions.
Or is there anything like that already somewhere and I have not known about it?
@vdemichev Is there something like a docker container with DiaNN?

My overall goal is to have a set of workers that will run DiaNN without any user intervention.

Best wishes you both,

Matteo

[vdemichev]

Hi Matteo,

The in silico library generation from a FASTA database can indeed be run once, and then this library can ber used to analyse experiments. This is actually the recommended way of doing 'library-free' analyses.

I can also reproduce the segfault when doing in silico prediction on a VM.
Yes, the Linux version of DIA-NN should be easily dockerisable, there's a pull request about it #149, although I haven't done it myself.

Best,
Vadim

[MatteoLacki]

OK, good to be in good company during hard times :)
I will let Dana (you met her in Bad Durkheim I guess) test this path in January: it will be perfectly OK for me to have the library generated once on Windows and rerun on Linux.

I have not made any docker images myself yet, but we will be likely investigating this for our other purposes and then contact you back if we have something cooked up.

Best,

[dingxuan1990]

I am only a user of DIANN and find it worked faster on linux.

[vdemichev]

I did try to trace when the segfault happens, seems to be inside pytorch library, when doing a neural network forward pass. Can be either a pytorch bug (in this case I will not be able to do anything about it, can only check if update works fine), or DIA-NN prepared input data for it incorrectly. But I put some checks on the latter, seems all fine.

[MatteoLacki]

Did you maybe run valgrind and traced the memory leak and reported it to pytorch guys?

[vdemichev]

valgrind is EXTREMELY slow and cumbersome to use due to lots of false hits. Maybe indeed still worth a try though.

[MatteoLacki]

It is somewhat sad to see that a ML library works better on Windows than on Linux. Nonetheless, maybe WSL on a windows VM will do what I want: I am using RedisQueue to orchestrate DiaNN, but it runs only on a fork-compatible system. I have here a true inception: Proxmox running Windows running WSL. It's virtualization all the way down :)

[vdemichev]

Well, it's still a chance there's a bug in how DIA-NN prepares input data for it, which only manifests on Linux. The thing is, Linux is indeed great for setting up automated analysis pipelines (we have on in Charite based on Wine actually I think), but for in silico library generation one probably does not need high throughput. Can just generate a human library once (or maybe several times for different modification sets), and then use these libs for years...

[hguturu]

I may be experiencing the same non-deterministic issue. Initially, I thought it was a single thread vs. multi-thread issue since I only saw the bug when I ran the command without the --threads flag. But, I was able to get the segfault when I ran the same command with --threads 30.

I am running DIA-NN with the following command:
/usr/diann/1.8/diann-1.8 --dir raw --fasta UP000005640_9606_combined.fasta --lib UP000005640_9606_combined.predicted.speclib --mass-acc-ms1 10 --mass-acc 10 --qvalue 0.01 --matrices --missed-cleavages 1 --met-excision --cut 'K*,R*' --smart-profiling --relaxed-prot-inf --threads 30
on Ubuntu 20.04.3 LTS (GNU/Linux 5.11.0-1025-aws x86_64) Linux ip-10-82-192-45 5.11.0-1025-aws #27~20.04.1-Ubuntu SMP Fri Jan 7 13:09:56 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

I ran it via GDB, to help track down to the following:

(gdb) run  --dir raw --fasta UP000005640_9606_combined.fasta  --lib UP000005640_9606_combined.predicted.speclib  --mass-acc-ms1 10 --mass-acc 10 --qvalue 0.01 --matrices --missed-cleavages 1 --met-excision --cut 'K*,R*' --smart-profiling --relaxed-prot-inf --threads 30
*** lots of output skipped ***
[6:47] Quantification information saved to raw/15891.mzML.quant.

[6:47] Cross-run analysis
[6:47] Reading quantification information: 1 files
[6:47] Quantifying peptides
[6:47] Assembling protein groups
--Type <RET> for more, q to quit, c to continue without paging--

Thread 1 "diann-1.8" received signal SIGSEGV, Segmentation fault.
0x0000555555508517 in void std::__introsort_loop<__gnu_cxx::__normal_iterator<int*, std::vector<int, std::allocator<int> > >, long, __gnu_cxx::__ops::_Iter_comp_iter<Library::infer_proteins()::{lambda(int, int)#1}> >(__gnu_cxx::__normal_iterator<int*, std::vector<int, std::allocator<int> > >, __gnu_cxx::__ops::_Iter_comp_iter<Library::infer_proteins()::{lambda(int, int)#1}>, long, __gnu_cxx::__ops::_Iter_comp_iter<Library::infer_proteins()::{lambda(int, int)#1}>) ()
(gdb) c
Continuing.

Program terminated with signal SIGSEGV, Segmentation fault.
The program no longer exists.

Its hard to debug further since the program is not open source.