How to dispose loaded model?

[ChezzPlaya]

If the GPU Memory has been occupied, then the Dispose method call leads to a crash.

So, how to dispose an already loaded model correctly?

[fwaris]

+1 I ran into the same issue.

[candlelightner]

+1 Here, although dispose works, something else (no idea what) is using memory afterwards (~15 GB). Could this be because I am using a stateful executor? after using the Chat method, I set everything to null and run the GC. Is there any method I could call to free the memory?

[martindevans]

Could you test this out using the master branch? My multi_context PR was merged yesterday and introduces an entirely new way of managing memory.

The main change is that model weights and contexts are split, so you can load the weights just once and create multiple contexts sharing the same weights.

[candlelightner]

How can I install the package from source?

[martindevans]

Sorry I never got back to you. The multi context stuff is now released as part of 0.5.1 - does it resolve this issue?

[martindevans]

I'll close this issue for now since I think it has been fixed for a while. Feel free to re-open if it's still a problem!