GGUF

[martindevans]

Initial changes required to support GGUF. Using these changes and llama.cpp b1081 successfully loaded this model. Resolve #121.

This is just a preview, required changes (all resolved)

• Modify LLama.Unittest project to auto download a gguf version of basic llama2-chat (closest equivalent of llama-2-7b-chat.ggmlv3, which is currently used)
  • I've switched to codellama-7b
  • Switched back to LLama2-7B
• New binaries
  • Most binaries generated in new GitHub action pipeline
  • MacOS Metal binaries created by SignalRT
• Testing on MacOS
  • Tested by SignalRT and swrhim

[saddam213]

Would it make sense to usellama_token_to_piece instead of llama_token_to_piece_with_model and move the Tokenize methods from LLamaModelHandle to LLamaContextHandle now?

[martindevans]

Keepig it in the model handle allows tokenizing without having to allocate a context, so I think keeping it in model makes more sense.

[AsakusaRinne]

Good job!. Is it ready to publish a new version of binary now? And maybe along with a new version of the main package?

[martindevans]

Is it ready to publish a new version of binary now

We do need new binaries for this to work.

The new GitHub action I've been working on builds everything except for Metal. It's running right now, I'll copy across the binaries in this PR once it finishes.

However, that doesn't do Metal yet. I need some help building that and/or setting it up in the action.

And maybe along with a new version of the main package?

Definitely! There's a lot of new features and I think not supporting GGUF will confuse people using the library (e.g. #129).

[martindevans]

• Added some of the binaries, generated by this action
• Based on this version: ggerganov/llama.cpp@6b73ef1

Does Not Include:

• Linux CUDA (added from GitHub action)
• MacOS metal (added by SignalRT)

[AsakusaRinne]

That's great. I could help to compile Linux CUDA and metal binary. However unfortunately I don't know how to added these two processes to actions, either.

[martindevans]

I managed to figure out how to build the Linux CUDA binaries (I'll add them once the action finishes running).

I really have no idea how to do metal though. If you could add the metal stuff into this PR that would be great.

Do you have a script you use to build the metal stuff? If you do then I can probably take that and adapt it into the GitHub action.

[drasticactions]

Compiling Metal should be adding -DLLAMA_METAL=ON to the macOS Cmake config

You could probably do what was done for the Windows builds to have an Includes for a non-metal and metal build.

[martindevans]

Aha, thanks for the tip @drasticactions!

One thing I'm a bit confused about with the metal build is there's libllama-metal.dylib in the runtimes folder (which I assume -DLLAMA_METAL=ON will produce) but there's also ggml-metal.metal which seems to be a source file. Is this really something we need to distribute with the binaries? If so, I assume I should just grab the file from llama.cpp (https://github.com/ggerganov/llama.cpp/blob/master/ggml-metal.metal)?

[drasticactions]

@martindevans Yes, it's part of the build output and read in the native library, https://github.com/ggerganov/llama.cpp/blob/bcce96ba4dd95482824700c4ce2455fe8c49055a/ggml-metal.m#L143-L156

But it is a build artifact, so instead of having it in source control, IMO, bundling it as part of the runtime library would make more sense, since that's where it's used and will ensure the exact version is put into the right place.

[martindevans]

Thanks again, I think that should be able to piece together a metal build now 👍

@AsakusaRinne I probably won't be able to look into doing this GitHub action+Metal stuff until next weekend, so if you want to go ahead with manually building the metal stuff and doing the nuget release that's fine by me.

[AsakusaRinne]

@martindevans Is the metal binary and this GGUF support the only barrier before the next release? I'm free to compile them and make a release these days.

[martindevans]

Yes, I think so 🥳

[swrhim]

I was able to build this branch and implement the example from the docs. One replacement I had to do from the example was replace LLamaModel with LLamaContext. After that, I was able to load the model successfully and run it locally on mac arm64

[SignalRT]

@martindevans
In this branch (https://github.com/SignalRT/LLamaSharp/tree/GGUF-MERGE) I update MacOS binaries (memory and metal), change code llama for a llama2 model and change on test that doesn't compile en VS Mac if your are interested.

All the test pass correctly and at least the basic manual test also works

[martindevans]

@SignalRT could you PR those changes into this branch (on my fork)? I'll merge it and it'll become part of this PR.

[martindevans]

@AsakusaRinne The final binaries have been added by SignalRT, so this is now fully ready to merge and release when you're ready.

[Arlodotexe]

Eagerly awaiting 👀

[theolivenbaum]

Same here! 🚀
(if it helps, tested the PR on Windows, works fine with CPU and CUDA)

[AsakusaRinne]

Great job! I'll merge it and publish a new release today. Thank you for the contributions from martindevans and other developers.❤️