Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(dataProcess): Add parallel processing functionality to dataProcess #116

Merged
merged 2 commits into from
Mar 22, 2024

Conversation

tonywu1999
Copy link
Contributor

Motivation and Context

In theory, MSstats should produce more meaningful results if clients set up experiments with more bioreplicates. However, as clients begin to input larger datasets into MSstats, they have been running into issues regarding speed & memory. This pull request aims to speed up the MSstats dataProcess function by adding a parameter called numberOfCores that should allow users to perform data processing (specifically the summarization step) in parallel per protein.

This is an extension of the parallel processing PR for groupComparison

Changes

  • Added new numberOfCores parameter to dataProcess function. Default is 1 core, which goes through the regular for loop approach with a progress bar indicator.
  • Used the parallel package's parLapply function to process MSstatsSummarizeSingleTmp or MSstatsSummarizeSingleLinear in parallel per protein.

Testing

  • Tested speed on this 2.3 GB dataset and encountered a ~30% increase in performance from 1 core to 2 cores on my M1 mac that has 8 cores (i.e. 102 seconds -> 78 seconds).
  • Added unit tests for dataProcess verifying that parallel processed function emitted the same number of rows as dataProcess with default parameters.
  • Callout: For some reason, input_split = split(input, input$PROTEIN) creates a variable that is listed as 35GB in R despite this dataset being 2.3GB. If this variable is created prior to exporting cluster variables, the parallel processing stalls since the input_split object takes up a huge amount of space in memory. As a result, I've refactored the multi-core approach to not create this object and instead represent the split using a list of indices instead (see protein_indices variable).

Checklist Before Requesting a Review

  • I have read the MSstats contributing guidelines
  • My changes generate no new warnings
  • Any dependent changes have been merged and published in downstream modules

@mstaniak
Copy link
Contributor

Are there any new imports or were all these functions used in groupComparison update already?
I will review this very soon

@tonywu1999
Copy link
Contributor Author

@mstaniak

All these functions were used in the groupComparison update already.

R/dataProcess.R Outdated Show resolved Hide resolved
@mstaniak
Copy link
Contributor

mstaniak commented Mar 18, 2024

Then the changes are OK, just two names requires an update, well done

R/dataProcess.R Show resolved Hide resolved
@tonywu1999 tonywu1999 merged commit 2b23788 into devel Mar 22, 2024
1 check passed
@tonywu1999 tonywu1999 deleted the feature-parallel-dataProcess branch March 22, 2024 15:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants