disable goodput recorder in test-maxtest.sh #3432

The self-hosted runner: A100-07b83d0b1737 lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.

amd64 / test-upstream-pax / test-upstream-pax-outcome

Process completed with exit code 1.

amd64 / test-upstream-t5x / test-upstream-t5x-outcome

Process completed with exit code 1.

amd64 / test-rosetta-pax / single-process-evaluation-te (1, 8, 1, 1)

The job running on runner GitHub Actions 284 has exceeded the maximum execution time of 360 minutes.

amd64 / test-rosetta-pax / single-process-evaluation-te (1, 8, 1, 1)

The operation was canceled.

amd64 / test-rosetta-pax / rosetta-pax-multi-node (4, 2, 1, 2)

The job running on runner GitHub Actions 455 has exceeded the maximum execution time of 360 minutes.

amd64 / test-rosetta-pax / rosetta-pax-multi-node (4, 2, 1, 2)

The operation was canceled.

amd64 / test-rosetta-pax / rosetta-pax-multi-node (1, 8, 1, 1)

The job running on runner GitHub Actions 360 has exceeded the maximum execution time of 360 minutes.

amd64 / test-rosetta-pax / rosetta-pax-multi-node (1, 8, 1, 1)

The operation was canceled.

amd64 / test-rosetta-pax / rosetta-pax-multi-node (4, 2, 1, 1)

The job running on runner GitHub Actions 325 has exceeded the maximum execution time of 360 minutes.

amd64 / test-rosetta-pax / rosetta-pax-multi-node (4, 2, 1, 1)

The operation was canceled.

amd64 / test-rosetta-pax / rosetta-pax-multi-node-te (5B_fused_attn_0, 1, 1, 8, 1, 2, --model-type 5B --disable-fused-attn)

The job running on runner GitHub Actions 108 has exceeded the maximum execution time of 360 minutes.

amd64 / test-rosetta-pax / rosetta-pax-multi-node-te (5B_fused_attn_0, 1, 1, 8, 1, 2, --model-type 5B --disable-fused-attn)

The operation was canceled.

amd64 / test-rosetta-pax / single-process-multi-device-te (1, 8, 1, 1)

The job running on runner GitHub Actions 366 has exceeded the maximum execution time of 360 minutes.

amd64 / test-rosetta-pax / single-process-multi-device-te (1, 8, 1, 1)

The operation was canceled.

amd64 / test-rosetta-pax / single-process-multi-device-te (1, 1, 2, 4)

The job running on runner GitHub Actions 30 has exceeded the maximum execution time of 360 minutes.

amd64 / test-rosetta-pax / single-process-multi-device-te (1, 1, 2, 4)

The operation was canceled.

amd64 / test-rosetta-pax / rosetta-pax-multi-node-te (5B_fused_attn_1, 1, 1, 8, 1, 2, --model-type 5B)

The job running on runner GitHub Actions 448 has exceeded the maximum execution time of 360 minutes.

amd64 / test-rosetta-pax / rosetta-pax-multi-node-te (5B_fused_attn_1, 1, 1, 8, 1, 2, --model-type 5B)

The operation was canceled.

amd64 / test-rosetta-pax / rosetta-pax-multi-node-te (8DP1FSDP1TP1PP_TE, 1, 8, 1, 1, 4)

The job running on runner GitHub Actions 444 has exceeded the maximum execution time of 360 minutes.

amd64 / test-rosetta-pax / rosetta-pax-multi-node-te (8DP1FSDP1TP1PP_TE, 1, 8, 1, 1, 4)

The operation was canceled.

amd64 / test-rosetta-pax / rosetta-pax-multi-node-te (LLaMA_eval_TE, 1, 1, 8, 1, 4, true, --model-type LLaMA70BProxy --evalu...

The job running on runner GitHub Actions 460 has exceeded the maximum execution time of 360 minutes.

amd64 / test-rosetta-pax / rosetta-pax-multi-node-te (LLaMA_eval_TE, 1, 1, 8, 1, 4, true, --model-type LLaMA70BProxy --evalu...

The operation was canceled.

amd64 / test-rosetta-pax / test-pax-rosetta-outcome

Process completed with exit code 1.

amd64 / test-rosetta-t5x / vit-multi-gpu-multi-node (8, 1)

The job running on runner GitHub Actions 180 has exceeded the maximum execution time of 360 minutes.

amd64 / test-rosetta-t5x / vit-multi-gpu-multi-node (8, 1)

The operation was canceled.

amd64 / test-rosetta-t5x / vit-multi-gpu-multi-node (8, 2)

The job running on runner GitHub Actions 73 has exceeded the maximum execution time of 360 minutes.

amd64 / test-rosetta-t5x / vit-multi-gpu-multi-node (8, 2)

The operation was canceled.

amd64 / test-rosetta-t5x / vit-multi-gpu-multi-node (1, 2)

The job running on runner GitHub Actions 446 has exceeded the maximum execution time of 360 minutes.

amd64 / test-rosetta-t5x / vit-multi-gpu-multi-node (1, 2)

The operation was canceled.

amd64 / test-rosetta-t5x / vit-multi-gpu-multi-node (1, 1)

The job running on runner GitHub Actions 493 has exceeded the maximum execution time of 360 minutes.

amd64 / test-rosetta-t5x / vit-multi-gpu-multi-node (1, 1)

The operation was canceled.

amd64 / test-rosetta-t5x / multi-gpu-multi-node (2N8G-te-1, 8, 2, --gin.train/utils.DatasetConfig.pack=False --gin.train_eva...

The job running on runner GitHub Actions 48 has exceeded the maximum execution time of 360 minutes.

amd64 / test-rosetta-t5x / multi-gpu-multi-node (2N8G-te-1, 8, 2, --gin.train/utils.DatasetConfig.pack=False --gin.train_eva...

The operation was canceled.

amd64 / test-rosetta-t5x / single-process-multi-device (1P8G_te-1, 8, --gin.train/utils.DatasetConfig.pack=False --gin.train...

The job running on runner GitHub Actions 144 has exceeded the maximum execution time of 360 minutes.

amd64 / test-rosetta-t5x / single-process-multi-device (1P8G_te-1, 8, --gin.train/utils.DatasetConfig.pack=False --gin.train...

The operation was canceled.

amd64 / test-rosetta-t5x / single-process-multi-device (1P1G_te-0, 1, --enable-te 0)

The job running on runner GitHub Actions 377 has exceeded the maximum execution time of 360 minutes.

amd64 / test-rosetta-t5x / single-process-multi-device (1P1G_te-0, 1, --enable-te 0)

The operation was canceled.

amd64 / test-rosetta-t5x / multi-gpu-multi-node (2N2G_te-0, 2, 2, --enable-te 0)

The job running on runner GitHub Actions 15 has exceeded the maximum execution time of 360 minutes.

amd64 / test-rosetta-t5x / multi-gpu-multi-node (2N2G_te-0, 2, 2, --enable-te 0)

The operation was canceled.

amd64 / test-rosetta-t5x / vit-single-process-multi-device (8)

The job running on runner GitHub Actions 8 has exceeded the maximum execution time of 360 minutes.

amd64 / test-rosetta-t5x / vit-single-process-multi-device (8)

The operation was canceled.

amd64 / test-rosetta-t5x / test-t5x-rosetta-sitrep / sitrep

Process completed with exit code 2.

amd64 / test-rosetta-t5x / test-t5x-rosetta-outcome

Process completed with exit code 2.

Artifacts

Produced during runtime

Name	Size
artifact-base-build-amd64	567 Bytes
artifact-base-build-arm64	566 Bytes
artifact-equinox-build-amd64	570 Bytes
artifact-equinox-build-arm64	568 Bytes
artifact-final-report	6.34 KB
artifact-gemma-build-amd64	561 Bytes
artifact-jax-build-amd64	552 Bytes
artifact-jax-build-arm64	553 Bytes
artifact-levanter-build-amd64	572 Bytes
artifact-levanter-build-arm64	573 Bytes
artifact-maxtext-build-amd64	568 Bytes
artifact-maxtext-build-arm64	568 Bytes
artifact-maxtext-test	670 Bytes
artifact-pax-build-amd64	566 Bytes
artifact-pax-build-arm64	568 Bytes
artifact-rosetta-build-pax-amd64	584 Bytes
artifact-rosetta-build-pax-arm64	583 Bytes
artifact-rosetta-build-t5x-amd64	584 Bytes
artifact-rosetta-build-t5x-arm64	585 Bytes
artifact-rosetta-pax-mgmn-test	681 Bytes
artifact-t5x-build-amd64	568 Bytes
artifact-t5x-build-arm64	567 Bytes
artifact-triton-build-amd64	564 Bytes
artifact-upstream-pax-mgmn-test	2.77 KB
artifact-upstream-t5x-mgmn-test	644 Bytes
artifact-workflow-metadata	277 Bytes
bumped-manifest	71.3 KB
final-base	254 Bytes
final-equinox	263 Bytes
final-gemma	257 Bytes
final-jax	251 Bytes
final-levanter	266 Bytes
final-maxtext	263 Bytes
final-pax	251 Bytes
final-t5x	251 Bytes
final-triton	259 Bytes
final-upstream-pax	277 Bytes
final-upstream-t5x	277 Bytes
gemma-unit-test-A100	4.08 KB
gemma-unit-test-V100	2.74 KB
jax-unit-test-A100	18.1 KB
jax-unit-test-V100	18.2 KB
levanter-unit-test-A100	6.85 KB
levanter-unit-test-V100	6.87 KB
mealkit-equinox	271 Bytes
mealkit-gemma	265 Bytes
mealkit-jax	261 Bytes
mealkit-levanter	275 Bytes
mealkit-maxtext	271 Bytes
mealkit-pax	261 Bytes
mealkit-t5x	261 Bytes
mealkit-triton	268 Bytes
mealkit-upstream-pax	286 Bytes
mealkit-upstream-t5x	286 Bytes
nsys-jax-unit-test-A100	29.2 MB
nsys-jax-unit-test-V100	16 MB
pallas-unit-test-A100	7.59 KB
pallas-unit-test-V100	7.48 KB
rosetta-pax-11374571521-16DP1FSDP1TP1PP_TE	477 KB
rosetta-pax-11374571521-1DP1FSDP1TP1PP_TE	86.7 KB
rosetta-pax-11374571521-1DP8FSDP1TP1PP_TE	259 KB
rosetta-pax-11374571521-4DP1FSDP2TP1PP	362 KB
rosetta-pax-11374571521-4DP1FSDP2TP1PP_TE	266 KB
rosetta-pax-11374571521-8DP_TE_dropout	258 KB
te-2GPU	5.92 KB
te-4GPU	5.94 KB
te-8GPU	6 KB
te-multigpu-test	527 Bytes
te-unit-test-V100	607 KB
triton-unit-test-A100	4.9 KB
triton-unit-test-V100	4.87 KB
upstream-maxtext-11374571521-1DP1FSDP1TP1PP	14 KB
upstream-maxtext-11374571521-1DP1FSDP8TP1PP	19.4 KB
upstream-maxtext-11374571521-1DP2FSDP4TP1PP_single_process	14.1 KB
upstream-maxtext-11374571521-1DP4FSDP2TP1PP	19.2 KB
upstream-maxtext-11374571521-1DP8FSDP1TP1PP	19.6 KB
upstream-maxtext-11374571521-2DP2FSDP2TP1PP	19.4 KB
upstream-maxtext-11374571521-4DP2FSDP2TP1PP	26.2 KB
upstream-maxtext-metrics-test-log	1.86 KB
upstream-pax-11374571521-16DP1FSDP1TP1PP	762 KB
upstream-pax-11374571521-1DP1FSDP1TP1PP	141 KB
upstream-pax-11374571521-1DP2FSDP4TP1PP_single_process	142 KB
upstream-pax-11374571521-1DP8FSDP1TP1PP	415 KB
upstream-pax-11374571521-2DP1FSDP1TP4PP	292 KB
upstream-pax-11374571521-2DP1FSDP2TP4PP	553 KB
upstream-pax-11374571521-4DP1FSDP2TP1PP	419 KB
upstream-pax-11374571521-8DP1FSDP1TP1PP	414 KB
upstream-pax-11374571521-8DP1FSDP1TP1PP_eval	108 KB
upstream-pax-11374571521-8DP1FSDP1TP1PP_single_process	142 KB
upstream-pax-11374571521-LLaMA_eval	243 KB
upstream-pax-metrics-test-log	7.11 KB
upstream-t5x-11374571521-1P2G_fmha	34 KB
upstream-t5x-11374571521-1P8G	33.6 KB
upstream-t5x-11374571521-8G2N	54.6 KB
upstream-t5x-11374571521-8G2N_fmha	55.3 KB
upstream-t5x-metrics-test-log	4.47 KB

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

disable goodput recorder in test-maxtest.sh #3432

Summary

disable goodput recorder in test-maxtest.sh #3432

Jobs

Run details

ci.yaml

Annotations

Artifacts