From d6d64df5e70ae07b2e09d3f61feabcbbef5e8306 Mon Sep 17 00:00:00 2001 From: Scott Todd Date: Tue, 22 Aug 2023 09:17:16 -0700 Subject: [PATCH] Include all GPU tests in test_a100 job. (#14765) See https://github.com/openxla/iree/issues/14169 for context. Tha `test_a100` job runs on a100 machines, which are in low supply relative to demand and cost more to run, so we only run them on postsubmit by default. It can take hours for a runner to be available, but then actual job time is usually around 10 minutes, only ~20 seconds of which is actually running tests ([sample logs](https://github.com/openxla/iree/actions/runs/5928273900/job/16074197115#step:7:265)). Previously, we only ran a100-specific tests (via the `requires-gpu-sm80` label). This PR changes to run all GPU tests (well, this should be all "a100-compatible GPU tests", but we don't have any AMD/ROCm/etc.-specific tests today). Differences between GPU architectures and driver versions are common, so it will help to have baseline feature test coverage on as many GPUs as we can get. configuration | total time | ctest time | number of tests | sample log link -- | -- | -- | -- | -- baseline | 7m 51s | 20 seconds | 13 | [logs](https://github.com/openxla/iree/actions/runs/5922237543/job/16056162777#step:7:272) all GPU tests, parallel 2 | 12m 32s | 292 seconds | 370 | [logs](https://github.com/openxla/iree/actions/runs/5931611100/job/16084038030?pr=14765#step:7:1008) all GPU tests, parallel 4 | 11m 40s | 248 seconds | 370 | [logs](https://github.com/openxla/iree/actions/runs/5932490304/job/16086531533?pr=14765#step:7:1010) all GPU tests, parallel 8 | 11m 42s | 245 seconds | 370 (1 timeout) | [logs](https://github.com/openxla/iree/actions/runs/5931784428/job/16084543385?pr=14765#step:7:1053) ci-exactly: build_all, test_a100 --- .github/workflows/ci.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index bfb4fd4f3072..7b46dbfa1e11 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -350,7 +350,7 @@ jobs: env: IREE_NVIDIA_SM80_TESTS_DISABLE: 0 IREE_MULTI_DEVICE_TESTS_DISABLE: 1 - IREE_CTEST_LABEL_REGEX: ^requires-gpu-sm80$ + IREE_CTEST_LABEL_REGEX: ^requires-gpu-sm80|^requires-gpu|^driver=vulkan$|^driver=cuda$ run: | ./build_tools/github_actions/docker_run.sh \ --env IREE_NVIDIA_SM80_TESTS_DISABLE \ @@ -359,7 +359,7 @@ jobs: --env IREE_VULKAN_F16_DISABLE=0 \ --env IREE_CUDA_DISABLE=0 \ --env IREE_NVIDIA_GPU_TESTS_DISABLE=0 \ - --env CTEST_PARALLEL_LEVEL=2 \ + --env CTEST_PARALLEL_LEVEL=4 \ --env NVIDIA_DRIVER_CAPABILITIES=all \ --gpus all \ gcr.io/iree-oss/nvidia@sha256:1717431fd46b8b1e96d95fa72508e3e3eacb5c95f1245b9b7dbeec23ae823d02 \