Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Llamacpp generation incoherent (always <eos>). Driver version on ubuntu 22.04.5? #12258

Open
ultoris opened this issue Oct 23, 2024 · 1 comment
Assignees

Comments

@ultoris
Copy link

ultoris commented Oct 23, 2024

I've tried using llamacpp in both docker and native versions using the provided guides:
https://github.com/intel-analytics/ipex-llm/blob/main/docs/mddocs/Quickstart/llama_cpp_quickstart.md
https://github.com/intel-analytics/ipex-llm/blob/main/docs/mddocs/DockerGuides/docker_cpp_xpu_quickstart.md
and cannot get correct model output in either case.

When using docker the generation stops at first token as model outputs <eos> no matter the prompt.
When using the native version I can get an answer but the model quality is heavily degraded, uses a lot of * tokens and gets incoherent after a few hundred tokens.
I've tried the latest version of the docker image and pip packages and also used older versions like ipex-llm[cpp]==2.1.0 (pip) that have slight variations but the problem persists.

I'm using Bartowski's GGUF Q8_0 versions of gemma2 (27b) and llama3.1 (8b) models. The models work fine on pure cpp ggerganov/llamacpp.

This is the output of the env-check script:

-----------------------------------------------------------------
PYTHON_VERSION=3.11.10
-----------------------------------------------------------------
transformers=4.44.2
-----------------------------------------------------------------
torch=2.2.0+cu121
-----------------------------------------------------------------
ipex-llm Version: 2.2.0b20240910
-----------------------------------------------------------------
IPEX is not installed. 
-----------------------------------------------------------------
CPU Information: 
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      46 bits physical, 48 bits virtual
Byte Order:                         Little Endian
CPU(s):                             18
On-line CPU(s) list:                0-17
Vendor ID:                          GenuineIntel
Model name:                         Intel(R) Core(TM) Ultra 5 125H
CPU family:                         6
Model:                              170
Thread(s) per core:                 2
Core(s) per socket:                 14
Socket(s):                          1
Stepping:                           4
CPU max MHz:                        4400,0000
CPU min MHz:                        400,0000
BogoMIPS:                           5990.40
-----------------------------------------------------------------
Total CPU Memory: 93.6115 GB
Memory Type: DDR5 
-----------------------------------------------------------------
Operating System: 
Ubuntu 22.04.5 LTS \n \l

-----------------------------------------------------------------
Linux mini 6.5.0-1027-oem #28-Ubuntu SMP PREEMPT_DYNAMIC Thu Jul 25 13:32:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
-----------------------------------------------------------------
env-check.sh: line 148: xpu-smi: command not found
-----------------------------------------------------------------
  Driver Version                                  2023.16.12.0.12_195853.xmain-hotfix
  Driver Version                                  2023.16.12.0.12_195853.xmain-hotfix
  Driver UUID                                     32342e33-352e-3330-3837-320000000000
  Driver Version                                  24.35.30872
-----------------------------------------------------------------
Driver related package version:
ii  intel-fw-gpu                                   2024.24.5-337~22.04                     all          Firmware package for Intel integrated and discrete GPUs
ii  intel-i915-dkms                                1.24.5.15.240718.18+i24-1               all          Out of tree i915 driver.
ii  intel-level-zero-gpu                           1.3.29735.27-914~22.04                  amd64        Intel(R) Graphics Compute Runtime for oneAPI Level Zero.
-----------------------------------------------------------------
igpu detected
[opencl:gpu:2] Intel(R) OpenCL Graphics, Intel(R) Arc(TM) Graphics OpenCL 3.0 NEO  [24.35.30872]
[ext_oneapi_level_zero:gpu:0] Intel(R) Level-Zero, Intel(R) Arc(TM) Graphics 1.3 [1.3.29735]
-----------------------------------------------------------------
xpu-smi is not installed. Please install xpu-smi according to README.md

I suspect the problem is related to the driver version 24.35.30872 which is lower than the 31.0.101.5522 specified on the FAQ of the llama_cpp_quickstart.md guide. I've followed the instructions on https://github.com/intel-analytics/ipex-llm/blob/main/docs/mddocs/Quickstart/install_linux_gpu.md#install-gpu-driver (option 1 for kernel 6.5) and the version that gets installed via apt is 24.35.30872.

@rnwang04 rnwang04 self-assigned this Oct 24, 2024
@JinheTang
Copy link
Contributor

JinheTang commented Oct 24, 2024

Hi @ultoris , we didn't reproduce this issue on our native Linux MTL and the llama-3.1 Q8_0 model output is normal:

./llama-cli -m Meta-Llama-3.1-8B-Instruct-q8_0.gguf -n 32 --prompt "Once upon a time, there existed a little girl who liked to have adventures. She wanted to go to places and meet new people, and have fun" -t 8 -e -ngl 33 --color  

output:
image

our env check result:

-----------------------------------------------------------------
PYTHON_VERSION=3.11.10
-----------------------------------------------------------------
transformers=4.44.2
-----------------------------------------------------------------
torch=2.2.0+cu121
-----------------------------------------------------------------
ipex-llm Version: 2.2.0b20241023
-----------------------------------------------------------------
IPEX is not installed. 
-----------------------------------------------------------------
CPU Information: 
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      46 bits physical, 48 bits virtual
Byte Order:                         Little Endian
CPU(s):                             18
On-line CPU(s) list:                0-17
Vendor ID:                          GenuineIntel
Model name:                         Intel(R) Core(TM) Ultra 5 125H
CPU family:                         6
Model:                              170
Thread(s) per core:                 2
Core(s) per socket:                 14
Socket(s):                          1
Stepping:                           4
CPU max MHz:                        4500.0000
CPU min MHz:                        400.0000
BogoMIPS:                           5990.40
-----------------------------------------------------------------
Total CPU Memory: 30.9502 GB
-----------------------------------------------------------------
Operating System: 
Ubuntu 22.04.5 LTS \n \l

-----------------------------------------------------------------
Linux Xiaoxin03-Ubuntu 6.5.0-35-generic #35~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue May  7 09:00:52 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
-----------------------------------------------------------------
/home/arda/jinhe/ipex-llm/python/llm/scripts/env-check.sh: line 148: xpu-smi: command not found
-----------------------------------------------------------------
  Driver UUID                                     32342e33-352e-3330-3837-320000000000
  Driver Version                                  24.35.30872
-----------------------------------------------------------------
Driver related package version:
ii  intel-level-zero-gpu                           1.3.29735.27-914~22.04                            amd64        Intel(R) Graphics Compute Runtime for oneAPI Level Zero.
-----------------------------------------------------------------
/home/arda/jinhe/ipex-llm/python/llm/scripts/env-check.sh: line 167: sycl-ls: command not found
igpu not detected
-----------------------------------------------------------------
xpu-smi is not installed. Please install xpu-smi according to README.md

We notice that you are not using the latest ipex-llm version. You may try to upgrade ipex-llm[cpp] to the latest version
and see if it works:

pip install --pre --upgrade ipex-llm[cpp]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants