Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

reduce concurrency of dorado jobs #2224

Merged
merged 1 commit into from
Oct 7, 2024
Merged

reduce concurrency of dorado jobs #2224

merged 1 commit into from
Oct 7, 2024

Conversation

cat-bro
Copy link
Collaborator

@cat-bro cat-bro commented Oct 7, 2024

Set dorado mem the same as alphafold mem so that not too many jobs will run at once

Set dorado mem the same as alphafold mem so that not too many jobs will run at once
@cat-bro
Copy link
Collaborator Author

cat-bro commented Oct 7, 2024

they are running out of GPU memory

nvidia-smi
Mon Oct  7 10:06:33 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.90.07              Driver Version: 550.90.07      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A100-PCIE-40GB          Off |   00000000:00:08.0 Off |                    0 |
| N/A   58C    P0            251W /  250W |   27095MiB /  40960MiB |    100%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A      8243      C   dorado                                       4964MiB |
|    0   N/A  N/A      8246      C   dorado                                       4794MiB |
|    0   N/A  N/A      8248      C   dorado                                       4794MiB |
|    0   N/A  N/A      8330      C   dorado                                       4794MiB |
|    0   N/A  N/A      8332      C   dorado                                       4794MiB |
|    0   N/A  N/A      8822      C   dorado                                       2918MiB |
+-----------------------------------------------------------------------------------------+

@cat-bro cat-bro merged commit 5ba0a2e into master Oct 7, 2024
1 check passed
@cat-bro cat-bro deleted the cat-bro-patch-35 branch October 7, 2024 10:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant