-
Notifications
You must be signed in to change notification settings - Fork 341
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to dispose loaded model? #49
Comments
+1 I ran into the same issue. |
+1 Here, although dispose works, something else (no idea what) is using memory afterwards (~15 GB). Could this be because I am using a stateful executor? after using the Chat method, I set everything to null and run the GC. Is there any method I could call to free the memory? |
Could you test this out using the master branch? My multi_context PR was merged yesterday and introduces an entirely new way of managing memory. The main change is that model weights and contexts are split, so you can load the weights just once and create multiple contexts sharing the same weights. |
How can I install the package from source? |
Sorry I never got back to you. The multi context stuff is now released as part of 0.5.1 - does it resolve this issue? |
I'll close this issue for now since I think it has been fixed for a while. Feel free to re-open if it's still a problem! |
If the GPU Memory has been occupied, then the Dispose method call leads to a crash.
So, how to dispose an already loaded model correctly?
The text was updated successfully, but these errors were encountered: