Skip to content

Commit

Permalink
Include all GPU tests in test_a100 job. (#14765)
Browse files Browse the repository at this point in the history
See #14169 for context.

Tha `test_a100` job runs on a100 machines, which are in low supply
relative to demand and cost more to run, so we only run them on
postsubmit by default. It can take hours for a runner to be available,
but then actual job time is usually around 10 minutes, only ~20 seconds
of which is actually running tests ([sample
logs](https://github.com/openxla/iree/actions/runs/5928273900/job/16074197115#step:7:265)).
Previously, we only ran a100-specific tests (via the `requires-gpu-sm80`
label). This PR changes to run all GPU tests (well, this should be all
"a100-compatible GPU tests", but we don't have any
AMD/ROCm/etc.-specific tests today). Differences between GPU
architectures and driver versions are common, so it will help to have
baseline feature test coverage on as many GPUs as we can get.

configuration | total time | ctest time | number of tests | sample log
link
-- | -- | -- | -- | --
baseline | 7m 51s | 20 seconds | 13 |
[logs](https://github.com/openxla/iree/actions/runs/5922237543/job/16056162777#step:7:272)
all GPU tests, parallel 2 | 12m 32s | 292 seconds | 370 |
[logs](https://github.com/openxla/iree/actions/runs/5931611100/job/16084038030?pr=14765#step:7:1008)
all GPU tests, parallel 4 | 11m 40s | 248 seconds | 370 |
[logs](https://github.com/openxla/iree/actions/runs/5932490304/job/16086531533?pr=14765#step:7:1010)
all GPU tests, parallel 8 | 11m 42s | 245 seconds | 370 (1 timeout) |
[logs](https://github.com/openxla/iree/actions/runs/5931784428/job/16084543385?pr=14765#step:7:1053)

ci-exactly: build_all, test_a100
  • Loading branch information
ScottTodd authored Aug 22, 2023
1 parent 2a914c1 commit d6d64df
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -350,7 +350,7 @@ jobs:
env:
IREE_NVIDIA_SM80_TESTS_DISABLE: 0
IREE_MULTI_DEVICE_TESTS_DISABLE: 1
IREE_CTEST_LABEL_REGEX: ^requires-gpu-sm80$
IREE_CTEST_LABEL_REGEX: ^requires-gpu-sm80|^requires-gpu|^driver=vulkan$|^driver=cuda$
run: |
./build_tools/github_actions/docker_run.sh \
--env IREE_NVIDIA_SM80_TESTS_DISABLE \
Expand All @@ -359,7 +359,7 @@ jobs:
--env IREE_VULKAN_F16_DISABLE=0 \
--env IREE_CUDA_DISABLE=0 \
--env IREE_NVIDIA_GPU_TESTS_DISABLE=0 \
--env CTEST_PARALLEL_LEVEL=2 \
--env CTEST_PARALLEL_LEVEL=4 \
--env NVIDIA_DRIVER_CAPABILITIES=all \
--gpus all \
gcr.io/iree-oss/nvidia@sha256:1717431fd46b8b1e96d95fa72508e3e3eacb5c95f1245b9b7dbeec23ae823d02 \
Expand Down

0 comments on commit d6d64df

Please sign in to comment.