BatchedExecutor Save/Load

[martindevans]

Added the ability to save and load individual conversations in a batched executor.

• Added BatchedExecutor.Load(filepath) method
• Added Conversation.Save(filepath) method
• Added new (currently internal) SaveState/LoadState methods in LLamaContext which can stash some extra binary data in the header. This can be used where other state needs to be saved alongside the raw KV cache.
• New example showing saving/loading a conversation

[Lyrcaxis]

Nice!

[martindevans]

I've just added the ability to save and load an entire Conversation to an in-memory State object, instead of to file.

This was inspired by some of the scheduler stuff @AsakusaRinne has been talking about. With this a scheduler could "swap out" a Conversation object and dispose it, freeing up space in the KV cache for other conversations to be evaluated, all without touching disk.

[zsogitbe]

I've just added the ability to save and load an entire Conversation to an in-memory State object, instead of to file.

This was inspired by some of the scheduler stuff @AsakusaRinne has been talking about. With this a scheduler could "swap out" a Conversation object and dispose it, freeing up space in the KV cache for other conversations to be evaluated, all without touching disk.

Multi-model execution is an important use case. Having such a scheduler would be very useful. Great work!