diff --git a/wiki/Additional-Software-Citation.md b/wiki/Additional-Software-Citation.md index 54bc054..576ebde 100644 --- a/wiki/Additional-Software-Citation.md +++ b/wiki/Additional-Software-Citation.md @@ -57,7 +57,6 @@ CTFFIND4 from cisTEM package: #### SUSAN -The software package is not yet published, but if you use it, please, cite [the following publication](https://www.eurasip.org/Proceedings/Eusipco/eusipco2019/Proceedings/papers/1570533917.pdf) describing the main algorithm idea as the following: -- Sánchez RM, Mester R & Kudryashev M (2019) Fast Alignment of Limited Angle Tomograms -by projected Cross Correlation. *2019 27th European Signal Processing Conference -(EUSIPCO)* pp 1–5. +If you used SUSAN, please, cite as the following: + +- Sánchez RM (2023) rkms86/SUSAN: Release for TomoBEAR (v0.1-RC1-TomoBEAR). *Zenodo*. DOI: [10.5281/zenodo.7950904](https://doi.org/10.5281/zenodo.7950904) diff --git a/wiki/Home.md b/wiki/Home.md index d171ef2..f6de14c 100644 --- a/wiki/Home.md +++ b/wiki/Home.md @@ -8,21 +8,24 @@ >
Implementation details and benchmarks you can find in our preprint:
Balyschew N, Yushkevich A, Mikirtumov V, Sanchez RM, Sprink T, Kudryashev M. Streamlined Structure Determination by Cryo-Electron Tomography and Subtomogram Averaging using TomoBEAR. **[Preprint]** 2023. bioRxiv doi: [10.1101/2023.01.10.523437](https://www.biorxiv.org/content/10.1101/2023.01.10.523437v1) -## Contents +## Wiki contents -- [General description of the pipeline](#general-description) +- [General description of the pipeline](#general-description) - [Installation and setup notes](https://github.com/KudryashevLab/TomoBEAR/wiki/Installation-and-Setup) - [Frequently Asked Questions](https://github.com/KudryashevLab/TomoBEAR/wiki/Frequently-Asked-Questions) - [Tutorials on TomoBEAR setup and usage](https://github.com/KudryashevLab/TomoBEAR/wiki/Tutorials) - [Available modules description, their parameters and default values](https://github.com/KudryashevLab/TomoBEAR/wiki/Modules) - [Usage cases and tips](https://github.com/KudryashevLab/TomoBEAR/wiki/Usage) +- [List of external software to be cited](https://github.com/KudryashevLab/TomoBEAR/wiki/Additional-Software-Citation) ## Gerenal description **TomoBEAR** can assist you in large-scale processing of the tomographic data acquired on the electron microscope starting from the raw tilt series, possibly dose fractionated, or already assembled tilt stacks up to sample volume 3D reconstruction and even to biological structure of interest. More on input formats you [can read here](https://github.com/KudryashevLab/TomoBEAR/wiki/Usage.md#input-data-file-formats). -**TomoBEAR** is designed to operate in automated manner minimizing user intervention where that is possible. The pipeline wraps popular cryo-ET tools (such as IMOD, Dynamo, MotionCor2, GCTF/CTFFIN4, etc.). Since number of TomoBEAR parameters is huge,to help users cope with that we have carefully designed a predefined set of defaults which were chosen based on several different cryo-ET datasets. +**TomoBEAR** is designed to operate in automated manner minimizing user intervention where that is possible. The pipeline consists of modules, which wrap popular cryo-ET tools (such as IMOD, Dynamo, MotionCor2, GCTF/CTFFIN4, etc.) as well as developed in our laboratory StA framework called [SUSAN](https://github.com/rkms86/SUSAN). + +Since number of **TomoBEAR** parameters is huge, to help users cope with that we have carefully designed a predefined set of defaults which were chosen based on several different cryo-ET datasets. Description of all modules and corresponding default values is given on the [Modules page](https://github.com/KudryashevLab/TomoBEAR/wiki/Modules). > **Note** >
`TomoBEAR` supports workstations and single interactive nodes with GPUs on the comuting clusters at the moment. We are also working towards enabling the support of computer clusters through a queue manager like SLURM or SGE (Sun Grid Engine). @@ -71,7 +74,7 @@ ## Pipeline structure -In the following picture you can see a flow chart which visualizes pipeline steps which `TomoBEAR` can execute in an automated and parallel manner. +In the following picture you can see a flow chart which visualizes pipeline steps which `TomoBEAR` can execute in an automated and parallel manner. ![Schematic Pipeline Image](/images/pipeline_light_mode.svg) diff --git a/wiki/Modules.md b/wiki/Modules.md index 1c69e11..a9a4f34 100644 --- a/wiki/Modules.md +++ b/wiki/Modules.md @@ -3,9 +3,16 @@ In this section you can find descriptions of the implemented modules, their func and their parameters which can be setup in the JSON configuration file in their corresponding blocks. -# Global parameters configuration +## Contents +- [Global parameters configuration](#global-parameters-configuration) +- [Pipeline behavior control modules](#pipeline-behavior-control-modules) +- [CryoET data processing modules](#cryoet-data-processing-modules) +- [Template matching-associated modules](#template-matching-associated-modules) +- [Subtomogram Averaging modules](#subtomogram-averaging-modules) -## general +## Global parameters configuration + +### general **Description**
The general section is not a module but a configuration section where all the general @@ -274,9 +281,9 @@ section. ``` -# Pipeline behavior control modules +## Pipeline behavior control modules -## StopPipeline +### StopPipeline **Description**
@@ -298,14 +305,18 @@ need to be optimized. } ``` -# CryoET data processing modules -## MetaData +## CryoET data processing modules +### MetaData **Description**
The MetaData module collects descriptive statistics such as min, max, mean, std from the raw data. +**Dependencies** +
+This module does not depend on any external packages. + **Parameters**
@@ -329,13 +340,17 @@ mean, std from the raw data. } ``` -## SortFiles +### SortFiles **Description**
The SortFiles module sorts the raw files on a tomogram basis and links them to their corresponding folders for further processing. +**Dependencies** +
+This module does not depend on any external packages. + **Parameters**
@@ -359,21 +374,20 @@ them to their corresponding folders for further processing. } ``` -## MotionCor2 +### MotionCor2 **Description**
-The MotionCor2 module uses as the module's name suggests MotionCor2 to -correct for the sample movement in a given projection which is basically -a dose-fractionated movie. +This module uses MotionCor2 to correct for the sample movement in a given projection can be a dose-fractionated movie or EER sequence. -For some of the options it is advised to look also into the manual of -MotionCor2 as there are some more detailed descriptions which you can -find -[here](https://docs.google.com/forms/d/e/1FAIpQLSfAQm5MA81qTx90W9JL6ClzSrM77tytsvyyHh1ZZWrFByhmfQ/viewform) -or [here](https://emcore.ucsf.edu/ucsf-software) when you download the -archive of MotionCor2. +For some of the options it is advised to look also into the original manual of MotionCor2 provided along with the source code as there are some more detailed descriptions. + +**Dependencies** +
+- MotionCor2 +- IMOD +- CUDA (compatible with chosen MotionCor2 version, if GPU parallelization is needed) **Parameters**
@@ -462,11 +476,15 @@ archive of MotionCor2. ``` -## GridEdgeEraser +### GridEdgeEraser **Description**
This module performs grid edge identification and erases it for Au grids data. +**Dependencies** +
+- Dynamo + **Parameters**
```json @@ -486,7 +504,7 @@ This module performs grid edge identification and erases it for Au grids data. } ``` -## CreateStacks +### CreateStacks **Description**
@@ -498,6 +516,11 @@ automatically if you are using high-dose images and divides them by their corresponding frame count in contrast to low-dose images where the frame-count is different. +**Dependencies** +
+- Dynamo +- IMOD + **Parameters**
@@ -533,7 +556,7 @@ frame-count is different. } ``` -## DynamoTiltSeriesAlignment +### DynamoTiltSeriesAlignment **Description**
@@ -545,6 +568,10 @@ ET projects are set. Some of them are dynamically derived. The option to override non-dynamically derived parameters is still available and can be done in the json configuration file. +**Dependencies** +
+- Dynamo + **Parameters**
@@ -692,7 +719,7 @@ be done in the json configuration file. ``` -## DynamoCleanStacks +### DynamoCleanStacks **Description**
@@ -702,6 +729,11 @@ that **DynamoCleanStacks** uses the output from dynamo tilt stacks alignment which states on which projections the fiducials could be fit. The others are then removed from the tilt stacks for further processing. +**Dependencies** +
+- Dynamo +- IMOD + **Parameters**
@@ -725,36 +757,43 @@ The others are then removed from the tilt stacks for further processing. } ``` -## AreTomo +### AreTomo **Description**
This module performs AreTomo-based fiducial-free alignment. +**Dependencies** +
+- AreTomo +- IMOD +- CUDA (compatible with AreTomo) + **Parameters**
```json "AreTomo": { - "execution_method": "in_order", - "input_stack_binning": 1, - "reconstruction": false, - "weighted_back_projection": true, - "tilt_axis_refine_flag": 1, - "apply_tilt_axis_offset": 0, - "tilt_axis_offset": 0, - "align_height_ratio": 0.75, - "apply_dose_weighting": false, - "sart": "20 5", - "roi": "0 0", - "roi_file": "", - "patch": "0 0", - "flip_volume": 1, - "flip_intensity": 0, - "citation": "" + "execution_method": "in_order", + "input_stack_binning": 1, + "reconstruction": false, + "weighted_back_projection": true, + "tilt_axis_refine_flag": 1, + "correct_tilt_axis_offset": 0, + "apply_given_tilt_axis_offset": false, + "tilt_axis_offset": 0, + "align_height_ratio": 0.75, + "apply_dose_weighting": false, + "sart": "20 5", + "roi": "0 0", + "roi_file": "", + "patch": "0 0", + "flip_volume": 1, + "flip_intensity": 0, + "citation": "" } ``` -## BatchRunTomo +### BatchRunTomo **Description**
@@ -787,6 +826,10 @@ fulfill all the steps batchruntomo normally can do: - 20: Post Processing with Trimvol - 21: NAD (Nonlinear anisotropic diffusion) +**Dependencies** +
+- IMOD + **Parameters**
@@ -937,7 +980,7 @@ tomoBEAR and should not be replaced or changed by the user. ``` -## GCTFCtfphaseflipCTFCorrection +### GCTFCtfphaseflipCTFCorrection **Description**
@@ -945,6 +988,12 @@ The GCTFCtfphaseflipCTFCorrection module is detecting the CTF for the tomograms which are reconstructed for template matching or particle cropping. +**Dependencies** +
+- IMOD +- GCTF/CTFFIND4 +- CUDA (compatible with GCTF/CTFFIND4) + **Parameters**
@@ -1018,13 +1067,17 @@ cropping. ``` -## BinStacks +### BinStacks **Description**
The BinStacks module is used for binning the stacks to be able to reconstruct them with the Reconstruct module which should be used after your stacks are binned. +**Dependencies** +
+- IMOD + **Parameters**
@@ -1044,23 +1097,23 @@ your stacks are binned. | doi | "" | this variable holds the information for automatically generating the citations, **note** functionality is not fully implemented and tested | | ```json - "BinStacks":{ + "BinStacks":{ "execution_method": "parallel", - "use_ctf_corrected_aligned_stack": true, - "antialias_filter": 6, - "defocus_tolerance": 20, - "iWidth": 2, + "use_ctf_corrected_aligned_stack": true, + "antialias_filter": 6, + "defocus_tolerance": 20, + "iWidth": 2, "ampContrast": 0.1, "run_ctf_phaseflip": false, - "ctf_corrected_stack_suffix": "ctfc", - "skip": false, + "ctf_corrected_stack_suffix": "ctfc", + "skip": false, "citation": "", - "citation_link": "", - "doi": "" + "citation_link": "", + "doi": "" } ``` -## Reconstruct +### Reconstruct **Description**
@@ -1071,6 +1124,10 @@ greater than one. The module is set up by default to reconstruct binned stacks. If you otherwise want to reconstruct unbinned stacks you need to set up the Reconstruct module properly. +**Dependencies** +
+- IMOD + **Parameters**
@@ -1107,7 +1164,108 @@ set up the Reconstruct module properly. } ``` -## DynamoImportTomograms +### IsoNet + +**Description** +
+This module provides functionality of the IsoNet - DL framework for pre-processing, training and prediciton of restored information in the missing wedge region on the reconstructed cryo-elecrtron tomography volumes. + +**Dependencies** +
+- Anaconda +- IsoNet (and its Python dependencies) +- CUDA + +
+
+ Default parameters values (click to expand) + +```json +"IsoNet": { + "execution_method": "once", + "isonet_env": "", + "repository_path": "", + "tomograms_to_use": [], + "steps_to_execute": {}, + "steps_to_execute_defaults": { + "prepare_star": { + "use_ctf_corrected_tomograms": false, + "add_defocus_to_star": true, + "tomograms_binning": -1, + "folder_name": "tomograms", + "output_star": "tomograms.star", + "number_subtomos": -1 + }, + "deconv": { + "star_file": "tomograms.star", + "deconv_folder": "deconv", + "snrfalloff": 0.7, + "deconvstrength": -1, + "highpassnyquist": -1, + "chunk_size": -1, + "overlap_rate": -1, + "ncpu": -1 + }, + "make_mask": { + "star_file": "tomograms.star", + "mask_folder": "mask", + "patch_size": -1, + "mask_boundary": "", + "density_percentage": 50, + "std_percentage": 50, + "z_crop": -1, + "use_deconv_tomo": true + }, + "extract": { + "star_file": "tomograms.star", + "subtomo_folder": "subtomo", + "subtomo_star": "subtomo.star", + "cube_size": -1, + "crop_size": -1, + "use_deconv_tomo": true + }, + "refine": { + "subtomo_star": "subtomo.star", + "iterations": 30, + "data_dir": "", + "pretrained_model": "", + "result_dir": "results", + "preprocessing_ncpus": -1, + "continue_from": "", + "epochs": -1, + "batch_size": -1, + "steps_per_epoch": -1, + "noise_level": [0.05,0.1,0.15,0.2], + "noise_start_iter": [10,15,20,25], + "noise_mode": "", + "noise_dir": "", + "learning_rate": -1, + "drop_out": -1, + "convs_per_depth": -1, + "kernel": [], + "unet_depth": -1, + "filter_base": -1, + "batch_normalization": -1, + "normalize_percentile": -1 + }, + "predict": { + "star_file": "tomograms.star", + "model": "results/model_iter30.h5", + "output_dir": "corrected_tomos", + "cube_size": -1, + "crop_size": -1, + "batch_size": -1, + "normalize_percentile": -1 + } + } + } +``` +
+ + +## Template matching-associated modules + +### DynamoImportTomograms **Description**
@@ -1116,6 +1274,10 @@ and inputs the tomograms to that catalogue. After that you can call the dynamo catalogue manager (dcm) to generate the models for the tomograms or pick in them particles by hand. +**Dependencies** +
+- Dynamo + **Parameters**
@@ -1139,8 +1301,8 @@ or pick in them particles by hand. } ``` -# Template matching-associated modules -## EMDTemplateGeneration +### EMDTemplateGeneration + **Description**
The EMDTemplateGeneration module is used to automatically download a @@ -1149,6 +1311,10 @@ matching binning. Besides that an automated routine to generate the mask is also implemented. This module needs to be run before template matching is executed. +**Dependencies** +
+- Dynamo + **Parameters**
@@ -1173,11 +1339,16 @@ matching is executed. } ``` -## TemplateGenerationFromFile +### TemplateGenerationFromFile + **Description**
The concept of this module is the same as in EMDTemplateMatching, with the only difference of taking the template from a user-defined path instead of fetching it directly from EMDB. +**Dependencies** +
+- Dynamo + **Parameters**
@@ -1201,7 +1372,7 @@ The concept of this module is the same as in EMDTemplateMatching, with the only } ``` -## DynamoTemplateMatching +### DynamoTemplateMatching **Description**
@@ -1210,6 +1381,10 @@ matching from Dynamo but on a GPU. Because of the GPU usage the whole thing runs up to 15 times faster than the normal template matching implementation of dynamo. +**Dependencies** +
+- Dynamo + **Parameters**
@@ -1243,11 +1418,16 @@ implementation of dynamo. ``` -## TemplateMatchingPostProcessing +### TemplateMatchingPostProcessing **Description**
This module creates Dynamo-like table of particles based on the results of the template matching procedure from the module DynamoTemplateMatching. As well, using this module you can extract subtomograms of the identified particles. +**Dependencies** +
+- Dynamo +- SUSAN (if you chose it as the particles generation method) + **Parameters**
@@ -1294,12 +1474,53 @@ This module creates Dynamo-like table of particles based on the results of the t ``` -# Subtomogram Averaging (StA) modules -## DynamoAlignmentProject +## Subtomogram Averaging modules + +### GenerateParticles.m +**Description** +
+This module uses Dynamo-like particles table to either extract Dynamo-like particles from the tomogram or to crop SUSAN-like substacks and make from them particles reconstructions. + +**Dependencies** +
+- Dynamo +- SUSAN (if you chose it as the particles generation method) + +**Parameters** +
+ +
+ Default parameters values (click to expand) + +```json + "GenerateParticles": { + "execution_method": "once", + "generate_particles_method": "", + "particles_table_path": "", + "box_size": 1.0, + "particles_binning": -1, + "use_SUSAN": false, + "ctf_correction_method": "defocus_file", + "susan_padding": 200, + "per_particle_ctf_correction": "phase_flip", + "padding_policy": "zero", + "normalization": "zm" + } +``` +
+ + +### DynamoAlignmentProject + **Description**
This module is basically a wrapper for performing Dynamo alignment projects. +**Dependencies** +
+- Dynamo +- SUSAN (if you chose it as the classification method) + **Parameters**
@@ -1348,58 +1569,58 @@ This module is basically a wrapper for performing Dynamo alignment projects. "classes": 0, "selected_classes": [], "sampling": 0, - "ite_r1": 0, - "nref_r1": 0, - "cone_range_r1": 0, - "cone_sampling_r1": 0, - "cone_flip_r1": 0, - "cone_check_peak_r1": 0, + "ite_r1": 0, + "nref_r1": 0, + "cone_range_r1": 0, + "cone_sampling_r1": 0, + "cone_flip_r1": 0, + "cone_check_peak_r1": 0, "cone_freeze_reference_r1": 0, - "inplane_range_r1": 0, - "inplane_sampling_r1": 0, - "inplane_flip_r1": 0, - "inplane_check_peak_r1": 0, - "inplane_freeze_reference_r1": 0, - "refine_r1": 0, - "refine_factor_r1": 0, - "high_r1": 0, - "low_r1": 0, - "sym_r1": 0, - "dim_r1": 0, + "inplane_range_r1": 0, + "inplane_sampling_r1": 0, + "inplane_flip_r1": 0, + "inplane_check_peak_r1": 0, + "inplane_freeze_reference_r1": 0, + "refine_r1": 0, + "refine_factor_r1": 0, + "high_r1": 0, + "low_r1": 0, + "sym_r1": 0, + "dim_r1": 0, "area_search_r1": 0, - "area_search_modus_r1": 0, - "separation_in_tomogram_r1": 0, - "limit_xy_check_peak_r1": 0, - "limit_z_check_peak_r1": 0, - "use_CC_r1": 0, - "localnc_r1": 0, - "mra_r1": 0, - "threshold_r1": 0, - "threshold_modus_r1": 0, - "threshold2_r1": 0, - "threshold2_modus_r1": 0, - "ccmatrix_r1": 0, - "ccmatrix_type_r1": 0, - "ccmatrix_batch_r1": 0, - "Xmatrix_r1": 0, - "Xmatrix_maxMb_r1": 0, - "PCA_r1": 0, - "PCA_neigs_r1": 0, - "kmeans_r1": "", - "kmeans_ncluster_r1": 0, - "kmeans_ncoefficients_r1": 0, - "nclass_r1": 0, - "plugin_align_r1": 0, - "plugin_post_r1": 0, - "plugin_iter_r1": 0, - "plugin_align_order_r1": 0, - "plugin_post_order_r1": 0, - "plugin_iter_order_r1": 0, - "flags_r1": 0, - "convergence_type_r1": 0, - "convergence_r1": 0, - "rings_r1": 0, - "rings_random_r1": 0, + "area_search_modus_r1": 0, + "separation_in_tomogram_r1": 0, + "limit_xy_check_peak_r1": 0, + "limit_z_check_peak_r1": 0, + "use_CC_r1": 0, + "localnc_r1": 0, + "mra_r1": 0, + "threshold_r1": 0, + "threshold_modus_r1": 0, + "threshold2_r1": 0, + "threshold2_modus_r1": 0, + "ccmatrix_r1": 0, + "ccmatrix_type_r1": 0, + "ccmatrix_batch_r1": 0, + "Xmatrix_r1": 0, + "Xmatrix_maxMb_r1": 0, + "PCA_r1": 0, + "PCA_neigs_r1": 0, + "kmeans_r1": "", + "kmeans_ncluster_r1": 0, + "kmeans_ncoefficients_r1": 0, + "nclass_r1": 0, + "plugin_align_r1": 0, + "plugin_post_r1": 0, + "plugin_iter_r1": 0, + "plugin_align_order_r1": 0, + "plugin_post_order_r1": 0, + "plugin_iter_order_r1": 0, + "flags_r1": 0, + "convergence_type_r1": 0, + "convergence_r1": 0, + "rings_r1": 0, + "rings_random_r1": 0, "dynamic_mask_r1": 0, "mask_path": "", "mask_apix": 1, diff --git a/wiki/Usage.md b/wiki/Usage.md index 8926c6f..d2216e0 100644 --- a/wiki/Usage.md +++ b/wiki/Usage.md @@ -1,6 +1,6 @@ # TomoBEAR Configuration -If you followed the instructions thoroughly then you should have a +If you followed the [installation instructions](https://github.com/KudryashevLab/TomoBEAR/wiki/Installation-and-Setup) thoroughly then you should have a working copy of the TomoBEAR. Now you need to generate a JSON file for your project to start to process your acquired cryo-ET data. @@ -58,7 +58,7 @@ You may use the script above ```softlink_files.sh``` to rename input data as sof ```bash softlink_files.sh PREFIX1_PREFIX2_PREFIX3_ PREFIXNEW_ original/dir/path renamed/dir/path ``` -where for each ```TIF``` file in the location ```original/dir/path``` a set of prefixes ```PREFIX1_PREFIX2_PREFIX3_``` will be substituted with a single prefix ```PREFIXNEW_``` and softlinks with the new filenames to the corresponding original files will be saved in the ```renamed/dir/path``` directory. +where for each ```TIF``` file in the location ```original/dir/path``` a set of prefixes ```PREFIX1_PREFIX2_PREFIX3_``` will be substituted with a single prefix ```PREFIXNEW_``` and softlinks with the new filenames to the corresponding original files will be saved in the ```renamed/dir/path``` directory. ### EER input file format pre-processing @@ -97,7 +97,7 @@ where the parameters are the following: * ```eer_total_number_of_fractions```: total number of fractions (data slices) present in raw EER data; * ```eer_fraction_grouping```: number of fractions (data slices) to group and integrate into a frame; * ```eer_exposure_per_fraction```: total electron dose accumulated per frame (group of fractions). -These parameters relate to the corresponding MotionCor2 parameters such as ```–EerSampling``` and three parameter values denoted in the file passed to ```–FmIntFile``` option, namely - total number of fractions (1st column), number of fractions in group (2nd column) and total accumulated dose per group (3rd column). +These parameters relate to the corresponding MotionCor2 parameters such as ```–EerSampling``` and three parameter values denoted in the file passed to ```–FmIntFile``` option, namely - total number of fractions (1st column), number of fractions in group (2nd column) and total accumulated dose per group (3rd column). ## JSON Configuration Templates @@ -867,7 +867,7 @@ Since live data processing mainly serves for sample quality check, in order to a In order to improve contrast in reconstructions we would recommend to enable `"generate_exact_filtered_tomograms"` and setup `"exact_filter_size"` to define filter to be used to produce contrast-enhanced reconstructions. -If you start collection from zero-tilt, to avoid problems of files perception and sorting caused by `-0.0`/`+0.0` appearing as the angle for untilted views (due to small initial tilting offset being present) instead of expected `0.0` it is also recommended to add `"first_tilt_angle": 0` to `general` section of your input JSON file. +If you start collection from zero-tilt, to avoid problems of files perception and sorting caused by `-0.0`/`+0.0` appearing as the angle for untilted views (due to small initial tilting offset being present) instead of expected `0.0` it is also recommended to add `"first_tilt_angle": 0` to `general` section of your input JSON file. # Executing the Workflow