Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ipex-llm] A significant deviation in accuracy between ipex llm 2.2.0b1 and 2.1.0b20240515 when running the codegeex model #12294

Open
johnysh opened this issue Oct 30, 2024 · 0 comments
Assignees

Comments

@johnysh
Copy link

johnysh commented Oct 30, 2024

When we ran codegeex models in ipex -llm 2.2.0b1 env , it generated a significant deviation in accuracy than in ipex-llm 2.1.0b2024515 or 2.1.0b2. We make test_logits test to get some data as below:

下面是各个ipex-llm版本间的logits结果(optimize_model=True)比较:
2.1.0b2 <-> 2.1.0b20240515 : 一致
2.1.0b2 <-> 2.1.0 : 不一致
2.1.0b2 <-> 2.2.0b1 : 不一致
2.1.0 <-> 2.2.0b1 : 不一致

下列版本做了optimize_model=True和optimize_model=False间的logits结果比较:
2.1.0b20240515: 一致
2.1.0b2: 一致
2.1.0: 不一致
2.2.0b1: 不一致

pls help me to debug this issue , thx.

@qiuxin2012 qiuxin2012 self-assigned this Oct 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants