Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: Add GP error analysis experiment script #227

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

jhlegarreta
Copy link
Collaborator

Add GP error analysis experiment script.

@jhlegarreta
Copy link
Collaborator Author

jhlegarreta commented Sep 25, 2024

A few notes:

  • Pardon my not very elaborated commit message.
  • Started placing the methods in new modules, but I do not think that is useful: even if e.g. a crossing fiber simulation script would benefit from some methods, these methods are not required for the framework to work/not necessary for the package, they would require adding tests, etc. So unnecessary work. So I would sacrifice that for having duplicated code across script in this case. At least for now.
  • The script can be launched as e.g.
pyhon /path/to/eddymotion/scripts/dwi_estimation_error_analysis.py \ 
  60 1000 100 \
  --evals1 0.0015 0.0003 0.0003 \
  --snr 20 \
  --repeats 3 \
  --kfold 1 3 5 15 30

The result of the above command should be:

Unclear why the error is not monotonically increasing.


# Simulate the fitting a number of times: every time the signal created will be a little
# different
for _ in range(repeats):
Copy link
Collaborator Author

@jhlegarreta jhlegarreta Sep 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing about these repeats is that the GP itself is able to give us a std dev,
https://scikit-learn.org/stable/modules/generated/sklearn.gaussian_process.GaussianProcessRegressor.html#sklearn.gaussian_process.GaussianProcessRegressor.predict

but we are not using it,

return model.predict(gtab, return_std=False)

so we may want to avoid an additional parameter and loop and use the former. Although I confess that as far as I know, none of DIPY's predict methods return such value.

Add GP error analysis experiment script.
Use scikit-learn GP returned std error value.
Copy link

codecov bot commented Sep 29, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 62.98%. Comparing base (b1d9928) to head (acf683b).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #227      +/-   ##
==========================================
- Coverage   63.97%   62.98%   -1.00%     
==========================================
  Files          18       18              
  Lines         905      905              
  Branches      154      154              
==========================================
- Hits          579      570       -9     
- Misses        282      288       +6     
- Partials       44       47       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@jhlegarreta
Copy link
Collaborator Author

Using the std deviation value of scikit-learn's GP regressor class (committed in
ea93c4a for the records, but most likely will drop when we get this sorted out) we get the below results, which is again not what we would expect

@jhlegarreta
Copy link
Collaborator Author

jhlegarreta commented Oct 6, 2024

@oesteban Adopted scikit learn's KFold to create folds in acf683b. One would call the script as e.g.

pyhon /path/to/eddymotion/scripts/dwi_estimation_error_analysis.py \
  60 1000 100 --evals1 0.0015 0.0003 0.0003 --repeats 3 --snr 20 --kfold 3 5 7 10 20 120
  • Kept the repeats parameter but not using them now.
  • The output is not what I would expect: error and std dev do not decrease as we add more volumes to the training set and predict less volumes each time (i.e. at increasing values of N; at 120 we are training with 119 volumes and predicting 1), and the RMSE with SNR 20 is smaller than with SNR 10

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants