Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

QNN Module API(new frontend) Preliminary Support #158

Merged
merged 57 commits into from
Oct 26, 2024

Conversation

oreomaker
Copy link
Collaborator

@oreomaker oreomaker commented Oct 11, 2024

!!!Do not merge until QNN backend qwen is runable!!!

Support assigning backends for modules and setup & executing computing graphs.
Better design for module backend splitting and 'tensor.to()' should be discussed.

TODO:

  • KVCacheNPU create and reuse
  • shadow execute used in QNN model
  • QNN tensor scale loading and setting

@yirongjie yirongjie requested review from liang1232018 and yirongjie and removed request for liang1232018 October 11, 2024 07:54
@yirongjie
Copy link
Collaborator

  1. add Layer::use_layername_2_tensorname to select whether use layername_2_tensorname;
  2. add to function GetBackendCreator in file Backend.cpp;
    if (type == MLLM_QNN) {
        Layer::use_layername_2_tensorname = false;
    }
  1. remove :
#ifdef USE_QNN
       // when use QNN, tensors are not reused
       // TODO: tensor reuse and pipeline support
       layername_2_tensorname[layer_next_name] = layer_next_name;
#else

@yirongjie yirongjie merged commit 64dd7b6 into UbiquitousLearning:main Oct 26, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants