-
Notifications
You must be signed in to change notification settings - Fork 341
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unable to load SYCL compiled backend #746
Comments
Are you sure you're using the correct version? Each version of LLamaSharp is compatible with one version of llama.cpp - you must use exactly the right version when compiling the binaries! See the table at the bottom of the readme for the versions: https://github.com/SciSharp/LLamaSharp?tab=readme-ov-file#map-of-llamasharp-and-llamacpp-versions |
Thanks for your reply. Yes I've compiled against the correct commit. The error I receive is: An error occurred during startup I've verified that LlamaSharp is pointing to a valid dll by adding a log callback (the 'Path' parameter shown is correct) |
Is there more to that error dump? I'm hoping for a more specific message at the bottom of the stack trace. |
Unfortunately not much, the inner exception is: LLama.Exceptions.RuntimeError: Failed to load the native library [my-app/bin/Debug/net8.0/CustomBackends/IntelARC/llama.dll] you specified. |
I've hidden the path for anonymity, but other than that it's unchanged |
Ah, unfortunately that's a pretty generic error. I don't really know anything about SYCL, but from a bit of googling I think your guess about the set-vars script looks likely to be the issue. If you run the script before launching your application does that not fix the issue? |
Maybe you could try to use dotnet cli in the intel one-API CMD line? Such as |
I ran into this too, I think it's more an issue with llama.cpp than with LLamaSharp. you need to copy all the dependencies to wherever llama.dll is being loaded from - it's these files:
..which are all included in the sycl package but then that package doesn't include llama.dll 😂 so you have to build that yourself (if you do that then at least you'll already have all these dependencies on your system!). here's an issue to include the shared libs in the sycl package: ggerganov/llama.cpp#7361 I would love a backend nuget for SYCL! |
Hi All, Thank you for all your input, @gfody I think that was it. Got the built dependencies alongside llama.dll and it seemingly works. The backend loads and I'm able to start inference. However, I have noticed a interesting issue: Inference is wildly unstable. Unlike when using the cpu backend, the sycl based backend just spits out garbage. Here's a copy of what my Llama 3 8b instruct model responded to me with when I prompted it with "Hello, how are you?": "I gravel ?? enced pm ["$ Equipment secondary initWithStyle O zcze / assistant equality standby Perkins .Is balances opak I or 378 juices Is shel )data" And another run: ' I """""""""""""""""""""""""""""""""""""""" ' I believe this is something unique to my sycl backend, though I'm not sure what yet. @AsakusaRinne & @martindevans, I've added a call to Process to call the setvars.bat of the intel oneAPI tool from within my app before the backend is loaded. I originally thought that it wouldn't work since it'd be run against the context of the child process, but it does? I'm not really sure what happens under the hood here so idk. But I will try and do more testing. Would LlamaSharp accept a SYCL nuget package if we / I am able to create one? Apologies, I'm new here (and to collaborative open source dev). Thank you again |
Scratch that, a different model file works. Unsure why that particular model runs fine on cpu / nvidia but fails with sycl. |
@gfody @mrtopkid Thank you for your efforts here. :)
We certainly would not reject such a PR as long as it could run stably. I'll appreciate it if you'd like to contribute. If you would like to add such a backend, here's some tips for you:
Could you please test the same model with llama.cpp main.exe compiled with SYCL? I'm not sure if its a LLamaSharp issue. |
@mrtopkid I've noticed this as well increasing @AsakusaRinne I've tested this and you can indeed just put the sycl dependencies in your path, they don't necessarily have to be in the same location as llama.dll. presumably though maybe this could cause some unintended consequences for whatever reason why the oneAPI Toolkit doesn't just put these in our path and why also llama.cpp bundle these with their sycl release package - I don't know |
Description
I tried compiling a sycl backend for use with intel gpus. By itself, it works fine (usual testing via the outputted main.exe of llama.cpp). However when pointing LlamaSharp to the compiled libs (I set shared libs to 'ON'), it fails to load and provides no further information.
I believe this is because with normal, non-LLamaSharp usage, one needs to run it via the intel one-API CMD line or to call the set vars script of the one-API tool. LlamaSharp won't do this. Is there any workaround for this?
The text was updated successfully, but these errors were encountered: