Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update binaries feb 2024 #479

Merged
merged 6 commits into from
Feb 6, 2024

Conversation

martindevans
Copy link
Member

Previous update (#460) was cancelled, due to some other work which changed things (introducing CLBLAST binaries, renaming windows binaries).

@martindevans
Copy link
Member Author

martindevans commented Feb 1, 2024

Testing:

  • Windows (CPU)
  • Windows (Nvidia GPU)
  • Windows (AMDGPU)
  • Linux (CPU)
  • Linux (GPU)
  • MacOS (ARM)
  • MacOS (x86)

@m0nsky
Copy link
Contributor

m0nsky commented Feb 2, 2024

My application seems to run fine in both avx2 and cu12.1.0, however, upping the batch size from 4096 to 8192 still throws:

System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.

when using or cu12.1.0 (while avx2 has no issues with 8192 batch size). I don't think I'll need a 8192 batch size and I'm not sure if this is expected, but just wanted to let you know.

Also, clblast throws a LLama.Exceptions.RuntimeError: The native library cannot be correctly loaded. for me. Does it require anything else to be installed?

Edit
I'm on Windows 10, RTX 3080

@martindevans
Copy link
Member Author

@jasoncouture any ideas about the CLBLAST loading issue?

@zsogitbe
Copy link
Contributor

zsogitbe commented Feb 3, 2024

I have tested the new code with several models on Windows 10 & GPU and it works fine. Thank you Martin!
The above errors are most likely errors in llame.cpp. Try another version. I remember having issues like this with some versions of the C++ code. Also, it seems to me that they have corrected a bug in the batch processing code just yesterday.

@m0nsky
Copy link
Contributor

m0nsky commented Feb 3, 2024

@martindevans Got clblast to work. I needed to copy clblast.dll to my executable dir (alongside the existing llama.dll), I just grabbed it from the latest llama.cpp release for now.

@martindevans
Copy link
Member Author

Thanks for testing that @m0nsky, I've added a note to #464 (which is the tracking issue for OpenCL support) about distributing clblast.dll in the nuget packages.

@ladeak
Copy link

ladeak commented Feb 4, 2024

Is there a preview nuget package to be tested?

@martindevans
Copy link
Member Author

@ladeak there's no preview package, if you want to test it your best option is to clone this branch

@jasoncouture
Copy link
Contributor

@martindevans should we use something like nbgv and publish preview builds?

@ladeak
Copy link

ladeak commented Feb 4, 2024

@martindevans I was able to test Phi-2 on Windows 11 x64. However, I notice there is a breaking change in the sense if I want to load a model for the LLamaEmbedder and for the LLamaSharpChatCompletion I would need to load it with EmbeddingMode set to true. Is that right?

EDIT: I tested through the semantic-kernel integration.

@martindevans
Copy link
Member Author

I notice there is a breaking change in the sense if I want to load a model for the LLamaEmbedder and for the LLamaSharpChatCompletion I would need to load it with EmbeddingMode set to true. Is that right?

Yep that's a deliberate change 👍

@jasoncouture
Copy link
Contributor

@martindevans I posted the PRs for full CLBLAST including the nuspec, see #489 , solves the issues of the missing clblast.dll, and adds linux AMDGPU support as well.

@martindevans
Copy link
Member Author

New run started here, using the same commit version.

@jasoncouture
Copy link
Contributor

@martindevans looks like it failed due to that sha256 check. I'll put up a PR to pull it for now, and get it figured out.

@martindevans
Copy link
Member Author

Started another run here

@martindevans martindevans merged commit 17385e1 into SciSharp:master Feb 6, 2024
3 checks passed
@martindevans martindevans deleted the update_binaries_feb_2024 branch February 6, 2024 01:08
@ladeak
Copy link

ladeak commented Feb 7, 2024

When do you plan to release the next version?

@martindevans
Copy link
Member Author

@jasoncouture is doing some tweaks to CLBLAST support, I think if that's done soon we'll probably release then.

@liqngliz
Copy link

Any update when this release will be out?

@martindevans
Copy link
Member Author

When it's ready! Hopefully this week.

I think if it's not done soon we'll push CLBLAST to the next release and get everything else out the door :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants