Inference latency measurement.

[foreverpiano]

System Info / 系統信息

A100-SXM 80GB

Information / 问题信息

• The official example scripts / 官方的示例脚本
• My own modified scripts / 我自己修改的脚本和任务

Reproduction / 复现过程

Single A100: ~180 seconds
Single H100: ~90 seconds
how did you get this result.

use official cli_demo.py
python cli_demo.py --prompt "A golden retriever, sporting sleek black sunglasses, with its lengthy fur flowing in the breeze, sprints playfully across a rooftop terrace, recently refreshed by a light rain. The scene unfolds from a distance, the dog's energetic bounds growing larger as it approaches the camera, its tail wagging with unrestrained joy, while droplets of water glisten on the concrete behind it. The overcast sky provides a dramatic backdrop, emphasizing the vibrant golden coat of the canine as it dashes towards the viewer." --model_path "/workspace/data/CogVideoX-5b" --generate_type "t2v"

Loading checkpoint shards: 100%|███████████████████████████████| 2/2 [00:02<00:00, 1.05s/it]
Loading pipeline components...: 100%|██████████████████████████| 5/5 [00:05<00:00, 1.16s/it]
100%|████████████████████████████████████████████████████████| 50/50 [08:31<00:00, 10.23s/it]

Expected behavior / 期待表现

slower than expected

Mon Oct 7 07:30:01 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.90.07 Driver Version: 550.90.07 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA A100-SXM4-80GB On | 00000000:07:00.0 Off | 0 |
| N/A 35C P0 63W / 400W | 1MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+