Operation problem of independent GPU 4090 24GB

[harrisxyh88]

When I was running this, I encountered the problem of gpu out of memory.
I am not sure if there is any solution, or if it can actually be run on the 4090 24GB, but I may be doing it wrong.

Thanks.

[Cherrysaber]

# pytorch >= 2.1.0
model = Qwen2AudioForConditionalGeneration.from_pretrained(
    "Qwen/Qwen2-Audio-7B", device_map="cuda:0", torch_dtype=torch.bfloat16)

❯ nvidia-smi
Fri Sep  6 20:22:58 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.90.07              Driver Version: 550.90.07      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4090        Off |   00000000:01:00.0 Off |                  Off |
| 30%   47C    P2             69W /  450W |   16867MiB /  24564MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 4090        Off |   00000000:02:00.0 Off |                  Off |
| 30%   47C    P8             12W /  450W |      22MiB /  24564MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+