Llamacpp generation incoherent (always <eos>). Driver version on ubuntu 22.04.5?

[ultoris]

I've tried using llamacpp in both docker and native versions using the provided guides:
https://github.com/intel-analytics/ipex-llm/blob/main/docs/mddocs/Quickstart/llama_cpp_quickstart.md
https://github.com/intel-analytics/ipex-llm/blob/main/docs/mddocs/DockerGuides/docker_cpp_xpu_quickstart.md
and cannot get correct model output in either case.

When using docker the generation stops at first token as model outputs <eos> no matter the prompt.
When using the native version I can get an answer but the model quality is heavily degraded, uses a lot of * tokens and gets incoherent after a few hundred tokens.
I've tried the latest version of the docker image and pip packages and also used older versions like ipex-llm[cpp]==2.1.0 (pip) that have slight variations but the problem persists.

I'm using Bartowski's GGUF Q8_0 versions of gemma2 (27b) and llama3.1 (8b) models. The models work fine on pure cpp ggerganov/llamacpp.

This is the output of the env-check script:

-----------------------------------------------------------------
PYTHON_VERSION=3.11.10
-----------------------------------------------------------------
transformers=4.44.2
-----------------------------------------------------------------
torch=2.2.0+cu121
-----------------------------------------------------------------
ipex-llm Version: 2.2.0b20240910
-----------------------------------------------------------------
IPEX is not installed. 
-----------------------------------------------------------------
CPU Information: 
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      46 bits physical, 48 bits virtual
Byte Order:                         Little Endian
CPU(s):                             18
On-line CPU(s) list:                0-17
Vendor ID:                          GenuineIntel
Model name:                         Intel(R) Core(TM) Ultra 5 125H
CPU family:                         6
Model:                              170
Thread(s) per core:                 2
Core(s) per socket:                 14
Socket(s):                          1
Stepping:                           4
CPU max MHz:                        4400,0000
CPU min MHz:                        400,0000
BogoMIPS:                           5990.40
-----------------------------------------------------------------
Total CPU Memory: 93.6115 GB
Memory Type: DDR5 
-----------------------------------------------------------------
Operating System: 
Ubuntu 22.04.5 LTS \n \l

-----------------------------------------------------------------
Linux mini 6.5.0-1027-oem #28-Ubuntu SMP PREEMPT_DYNAMIC Thu Jul 25 13:32:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
-----------------------------------------------------------------
env-check.sh: line 148: xpu-smi: command not found
-----------------------------------------------------------------
  Driver Version                                  2023.16.12.0.12_195853.xmain-hotfix
  Driver Version                                  2023.16.12.0.12_195853.xmain-hotfix
  Driver UUID                                     32342e33-352e-3330-3837-320000000000
  Driver Version                                  24.35.30872
-----------------------------------------------------------------
Driver related package version:
ii  intel-fw-gpu                                   2024.24.5-337~22.04                     all          Firmware package for Intel integrated and discrete GPUs
ii  intel-i915-dkms                                1.24.5.15.240718.18+i24-1               all          Out of tree i915 driver.
ii  intel-level-zero-gpu                           1.3.29735.27-914~22.04                  amd64        Intel(R) Graphics Compute Runtime for oneAPI Level Zero.
-----------------------------------------------------------------
igpu detected
[opencl:gpu:2] Intel(R) OpenCL Graphics, Intel(R) Arc(TM) Graphics OpenCL 3.0 NEO  [24.35.30872]
[ext_oneapi_level_zero:gpu:0] Intel(R) Level-Zero, Intel(R) Arc(TM) Graphics 1.3 [1.3.29735]
-----------------------------------------------------------------
xpu-smi is not installed. Please install xpu-smi according to README.md

I suspect the problem is related to the driver version 24.35.30872 which is lower than the 31.0.101.5522 specified on the FAQ of the llama_cpp_quickstart.md guide. I've followed the instructions on https://github.com/intel-analytics/ipex-llm/blob/main/docs/mddocs/Quickstart/install_linux_gpu.md#install-gpu-driver (option 1 for kernel 6.5) and the version that gets installed via apt is 24.35.30872.

[JinheTang]

Hi @ultoris , we didn't reproduce this issue on our native Linux MTL and the llama-3.1 Q8_0 model output is normal:

./llama-cli -m Meta-Llama-3.1-8B-Instruct-q8_0.gguf -n 32 --prompt "Once upon a time, there existed a little girl who liked to have adventures. She wanted to go to places and meet new people, and have fun" -t 8 -e -ngl 33 --color  

output:

our env check result:

-----------------------------------------------------------------
PYTHON_VERSION=3.11.10
-----------------------------------------------------------------
transformers=4.44.2
-----------------------------------------------------------------
torch=2.2.0+cu121
-----------------------------------------------------------------
ipex-llm Version: 2.2.0b20241023
-----------------------------------------------------------------
IPEX is not installed. 
-----------------------------------------------------------------
CPU Information: 
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      46 bits physical, 48 bits virtual
Byte Order:                         Little Endian
CPU(s):                             18
On-line CPU(s) list:                0-17
Vendor ID:                          GenuineIntel
Model name:                         Intel(R) Core(TM) Ultra 5 125H
CPU family:                         6
Model:                              170
Thread(s) per core:                 2
Core(s) per socket:                 14
Socket(s):                          1
Stepping:                           4
CPU max MHz:                        4500.0000
CPU min MHz:                        400.0000
BogoMIPS:                           5990.40
-----------------------------------------------------------------
Total CPU Memory: 30.9502 GB
-----------------------------------------------------------------
Operating System: 
Ubuntu 22.04.5 LTS \n \l

-----------------------------------------------------------------
Linux Xiaoxin03-Ubuntu 6.5.0-35-generic #35~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue May  7 09:00:52 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
-----------------------------------------------------------------
/home/arda/jinhe/ipex-llm/python/llm/scripts/env-check.sh: line 148: xpu-smi: command not found
-----------------------------------------------------------------
  Driver UUID                                     32342e33-352e-3330-3837-320000000000
  Driver Version                                  24.35.30872
-----------------------------------------------------------------
Driver related package version:
ii  intel-level-zero-gpu                           1.3.29735.27-914~22.04                            amd64        Intel(R) Graphics Compute Runtime for oneAPI Level Zero.
-----------------------------------------------------------------
/home/arda/jinhe/ipex-llm/python/llm/scripts/env-check.sh: line 167: sycl-ls: command not found
igpu not detected
-----------------------------------------------------------------
xpu-smi is not installed. Please install xpu-smi according to README.md

We notice that you are not using the latest ipex-llm version. You may try to upgrade ipex-llm[cpp] to the latest version
and see if it works:

pip install --pre --upgrade ipex-llm[cpp]