buildx failed with: ERROR: failed to solve: process "/bin/sh -c pip-finalize.sh" did not complete successfully: exit code: 1

buildx failed with: ERROR: failed to solve: process "/bin/sh -c <<\"EOF\" bash -ex\npushd /opt/checkout/external/triton\nmkdir dist\n# Do not compile with -Werror\nsed -i -e 's|-Werror||g' CMakeLists.txt\n# The LLVM build above does not enable these libraries\nsed -i -e 's|\\(LLVMAMDGPU.*\\)|# \\1|g' CMakeLists.txt\n# Do not build tests\nsed -i -e 's|^\\s*add_subdirectory(unittest)|# unit tests disabled|' CMakeLists.txt\n# Do not build the AMD GPU backend\nsed -i -e 's|BackendInstaller.copy(\\[\"nvidia\", \"amd\"\\])|BackendInstaller.copy([\"nvidia\"])|g' python/setup.py\n# Google has patches that mess with include paths in source files\nsed -i -e '/include_directories(${PROJECT_SOURCE_DIR}\\/third_party)/a include_directories(${PROJECT_SOURCE_DIR}/third_party/nvidia/include)' CMakeLists.txt\nsed -i -e '/include_directories(${PROJECT_BINARY_DIR}\\/third_party)/a include_directories(${PROJECT_BINARY_DIR}/third_party/nvidia/include)' CMakeLists.txt\n# Extra patches to Triton maintained in XLA. These are already applied in the working directory.\nXLA_TRITON_PATCHES=\"${SRC_PATH_XLA}/third_party/triton\"\n# This patch adds two files that are not known to CMake\nif [[ -f \"${XLA_TRITON_PATCHES}/xla_extensions/sparse_dot_base.patch\" ]]; then\n sed -i -e '/ConvertLayoutOpToLLVM.cpp/a ConvertLayoutOpToLLVM/SharedToSparseDotOperand.cpp' third_party/nvidia/lib/TritonNVIDIAGPUToLLVM/CMakeLists.txt\n sed -i -e '/DotOpToLLVM.cpp/a DotOpToLLVM/Sparse.cpp' third_party/nvidia/lib/TritonNVIDIAGPUToLLVM/CMakeLists.txt\nfi\n# Use clang to match Google etc.\nLLVM_INCLUDE_DIRS=/opt/llvm-build/include \\\n LLVM_LIBRARY_DIR=/opt/llvm-build/lib \\\n LLVM_SYSPATH=/opt/llvm-build \\\n TRITON_BUILD_WITH_CLANG_LLD=1 \\\n pip wheel --verbose --wheel-dir=dist/ python/\n# Clean up the wheel build directory so it doesn't end up bloating the container\nrm -rf python/build\nEOF" did not complete successfully: exit code: 1

buildx failed with: ERROR: failed to solve: process "/bin/sh -c <<\"EOF\" bash -exu\nMANIFEST_DIR=$(dirname ${MANIFEST_FILE})\nif [[ \"${UPDATE_PATCHES}\" != \"true\" && \"${UPDATE_PATCHES}\" != \"false\" ]]; then\n echo \"UPDATE_PATCHES can only be true or false\"\n exit 1\nfi\nif [[ \"${UPDATE_PATCHES}\" == \"true\" ]]; then\n cp -r /mnt/jax-toolbox/.github/container/patches ${MANIFEST_DIR}/\n cp /mnt/jax-toolbox/.github/container/manifest.yaml ${MANIFEST_DIR}/manifest.yaml\n cp /mnt/jax-toolbox/.github/container/create-distribution.sh ${MANIFEST_DIR}/create-distribution.sh\n # TODO: remove\n cp /mnt/jax-toolbox/.github/container/pip-finalize.sh /usr/local/bin/\nfi\ncp -r /mnt/jax-toolbox/rosetta /opt/rosetta\n\nif [[ -n \"${UPDATED_TE_REF}\" ]]; then\n TE_INSTALL_DIR=/opt/transformer-engine\n yq e \".transformer-engine.latest_verified_commit = \\\"${UPDATED_TE_REF}\\\"\" -i $MANIFEST_FILE\n # Install from source instead of pre-built wheel\n sed -i -E 's@( file:///opt/transformer-engine)/dist/[^ ]*@\\1@' /opt/pip-tools.d/requirements-te.in\n git -C $TE_INSTALL_DIR fetch -a\n if [[ \"${UPDATED_TE_REF}\" =~ ^pull/ ]]; then\n PR_ID=$(cut -d/ -f2 <<<\"${UPDATED_TE_REF}\")\n git -C $TE_INSTALL_DIR fetch origin ${UPDATED_TE_REF}:PR-${PR_ID}\n git -C $TE_INSTALL_DIR checkout PR-${PR_ID}\n else\n git -C $TE_INSTALL_DIR checkout ${UPDATED_TE_REF}\n fi\nfi\n\n# Setting the username/email is required to author commits from patches\ngit config --global user.email \"${GIT_USER_EMAIL}\"\ngit config --global user.name \"${GIT_USER_NAME}\"\n\nbash ${MANIFEST_DIR}/create-distribution.sh \\\n --manifest ${MANIFEST_FILE} \\\n --package t5x\nbash ${MANIFEST_DIR}/create-distribution.sh \\\n --manifest ${MANIFEST_FILE} \\\n --package flax\n# Remove .gitconfig to avoid end-user authoring commits as the \"build user\"\nrm -f ~/.gitconfig\n\necho \"--extra-index-url https://developer.download.nvidia.com/compute/redist\" >> /opt/pip-tools.d/requirements-t5x.in\necho \"-e file:///opt/rosetta\" >> /opt/pip-tools.d/requirements-t5x.in\nEOF" did not complete successfully: exit code: 1

buildx failed with: ERROR: failed to solve: process "/bin/sh -c <<\"EOF\" bash -exu\nMANIFEST_DIR=$(dirname ${MANIFEST_FILE})\nif [[ \"${UPDATE_PATCHES}\" != \"true\" && \"${UPDATE_PATCHES}\" != \"false\" ]]; then\n echo \"UPDATE_PATCHES can only be true or false\"\n exit 1\nfi\nif [[ \"${UPDATE_PATCHES}\" == \"true\" ]]; then\n cp -r /mnt/jax-toolbox/.github/container/patches ${MANIFEST_DIR}/\n cp /mnt/jax-toolbox/.github/container/manifest.yaml ${MANIFEST_DIR}/manifest.yaml\n cp /mnt/jax-toolbox/.github/container/create-distribution.sh ${MANIFEST_DIR}/create-distribution.sh\n # TODO: remove\n cp /mnt/jax-toolbox/.github/container/pip-finalize.sh /usr/local/bin/\nfi\ncp -r /mnt/jax-toolbox/rosetta /opt/rosetta\n\nif [[ -n \"${UPDATED_TE_REF}\" ]]; then\n TE_INSTALL_DIR=/opt/transformer-engine\n yq e \".transformer-engine.latest_verified_commit = \\\"${UPDATED_TE_REF}\\\"\" -i $MANIFEST_FILE\n # Install from source instead of pre-built wheel\n sed -i -E 's@( file:///opt/transformer-engine)/dist/[^ ]*@\\1@' /opt/pip-tools.d/requirements-te.in\n git -C $TE_INSTALL_DIR fetch -a\n if [[ \"${UPDATED_TE_REF}\" =~ ^pull/ ]]; then\n PR_ID=$(cut -d/ -f2 <<<\"${UPDATED_TE_REF}\")\n git -C $TE_INSTALL_DIR fetch origin ${UPDATED_TE_REF}:PR-${PR_ID}\n git -C $TE_INSTALL_DIR checkout PR-${PR_ID}\n else\n git -C $TE_INSTALL_DIR checkout ${UPDATED_TE_REF}\n fi\nfi\n\n# Setting the username/email is required to author commits from patches\ngit config --global user.email \"${GIT_USER_EMAIL}\"\ngit config --global user.name \"${GIT_USER_NAME}\"\n\nbash ${MANIFEST_DIR}/create-distribution.sh \\\n --manifest ${MANIFEST_FILE} \\\n --package paxml\nbash ${MANIFEST_DIR}/create-distribution.sh \\\n --manifest ${MANIFEST_FILE} \\\n --package praxis\n# Remove .gitconfig to avoid end-user authoring commits as the \"build user\"\nrm -f ~/.gitconfig\nEOF" did not complete successfully: exit code: 1

buildx failed with: ERROR: failed to solve: process "/bin/sh -c <<\"EOF\" bash -exu\nMANIFEST_DIR=$(dirname ${MANIFEST_FILE})\nif [[ \"${UPDATE_PATCHES}\" != \"true\" && \"${UPDATE_PATCHES}\" != \"false\" ]]; then\n echo \"UPDATE_PATCHES can only be true or false\"\n exit 1\nfi\nif [[ \"${UPDATE_PATCHES}\" == \"true\" ]]; then\n cp -r /mnt/jax-toolbox/.github/container/patches ${MANIFEST_DIR}/\n cp /mnt/jax-toolbox/.github/container/manifest.yaml ${MANIFEST_DIR}/manifest.yaml\n cp /mnt/jax-toolbox/.github/container/create-distribution.sh ${MANIFEST_DIR}/create-distribution.sh\n # TODO: remove\n cp /mnt/jax-toolbox/.github/container/pip-finalize.sh /usr/local/bin/\nfi\ncp -r /mnt/jax-toolbox/rosetta /opt/rosetta\n\nif [[ -n \"${UPDATED_TE_REF}\" ]]; then\n TE_INSTALL_DIR=/opt/transformer-engine\n yq e \".transformer-engine.latest_verified_commit = \\\"${UPDATED_TE_REF}\\\"\" -i $MANIFEST_FILE\n # Install from source instead of pre-built wheel\n sed -i -E 's@( file:///opt/transformer-engine)/dist/[^ ]*@\\1@' /opt/pip-tools.d/requirements-te.in\n git -C $TE_INSTALL_DIR fetch -a\n if [[ \"${UPDATED_TE_REF}\" =~ ^pull/ ]]; then\n PR_ID=$(cut -d/ -f2 <<<\"${UPDATED_TE_REF}\")\n git -C $TE_INSTALL_DIR fetch origin ${UPDATED_TE_REF}:PR-${PR_ID}\n git -C $TE_INSTALL_DIR checkout PR-${PR_ID}\n else\n git -C $TE_INSTALL_DIR checkout ${UPDATED_TE_REF}\n fi\nfi\n\n# Setting the username/email is required to author commits from patches\ngit config --global user.email \"${GIT_USER_EMAIL}\"\ngit config --global user.name \"${GIT_USER_NAME}\"\n\nbash ${MANIFEST_DIR}/create-distribution.sh \\\n --manifest ${MANIFEST_FILE} \\\n --package t5x\nbash ${MANIFEST_DIR}/create-distribution.sh \\\n --manifest ${MANIFEST_FILE} \\\n --package flax\n# Remove .gitconfig to avoid end-user authoring commits as the \"build user\"\nrm -f ~/.gitconfig\n\necho \"--extra-index-url https://developer.download.nvidia.com/compute/redist\" >> /opt/pip-tools.d/requirements-t5x.in\necho \"-e file:///opt/rosetta\" >> /opt/pip-tools.d/requirements-t5x.in\nEOF" did not complete successfully: exit code: 1

buildx failed with: ERROR: failed to solve: process "/bin/sh -c <<\"EOF\" bash -exu\nMANIFEST_DIR=$(dirname ${MANIFEST_FILE})\nif [[ \"${UPDATE_PATCHES}\" != \"true\" && \"${UPDATE_PATCHES}\" != \"false\" ]]; then\n echo \"UPDATE_PATCHES can only be true or false\"\n exit 1\nfi\nif [[ \"${UPDATE_PATCHES}\" == \"true\" ]]; then\n cp -r /mnt/jax-toolbox/.github/container/patches ${MANIFEST_DIR}/\n cp /mnt/jax-toolbox/.github/container/manifest.yaml ${MANIFEST_DIR}/manifest.yaml\n cp /mnt/jax-toolbox/.github/container/create-distribution.sh ${MANIFEST_DIR}/create-distribution.sh\n # TODO: remove\n cp /mnt/jax-toolbox/.github/container/pip-finalize.sh /usr/local/bin/\nfi\ncp -r /mnt/jax-toolbox/rosetta /opt/rosetta\n\nif [[ -n \"${UPDATED_TE_REF}\" ]]; then\n TE_INSTALL_DIR=/opt/transformer-engine\n yq e \".transformer-engine.latest_verified_commit = \\\"${UPDATED_TE_REF}\\\"\" -i $MANIFEST_FILE\n # Install from source instead of pre-built wheel\n sed -i -E 's@( file:///opt/transformer-engine)/dist/[^ ]*@\\1@' /opt/pip-tools.d/requirements-te.in\n git -C $TE_INSTALL_DIR fetch -a\n if [[ \"${UPDATED_TE_REF}\" =~ ^pull/ ]]; then\n PR_ID=$(cut -d/ -f2 <<<\"${UPDATED_TE_REF}\")\n git -C $TE_INSTALL_DIR fetch origin ${UPDATED_TE_REF}:PR-${PR_ID}\n git -C $TE_INSTALL_DIR checkout PR-${PR_ID}\n else\n git -C $TE_INSTALL_DIR checkout ${UPDATED_TE_REF}\n fi\nfi\n\n# Setting the username/email is required to author commits from patches\ngit config --global user.email \"${GIT_USER_EMAIL}\"\ngit config --global user.name \"${GIT_USER_NAME}\"\n\nbash ${MANIFEST_DIR}/create-distribution.sh \\\n --manifest ${MANIFEST_FILE} \\\n --package paxml\nbash ${MANIFEST_DIR}/create-distribution.sh \\\n --manifest ${MANIFEST_FILE} \\\n --package praxis\n# Remove .gitconfig to avoid end-user authoring commits as the \"build user\"\nrm -f ~/.gitconfig\nEOF" did not complete successfully: exit code: 1

Process completed with exit code 1.

The job running on runner GitHub Actions 449 has exceeded the maximum execution time of 360 minutes.

The operation was canceled.

The job running on runner GitHub Actions 426 has exceeded the maximum execution time of 360 minutes.

The operation was canceled.

Process completed with exit code 1.

The job running on runner GitHub Actions 204 has exceeded the maximum execution time of 360 minutes.

The operation was canceled.

The job running on runner GitHub Actions 383 has exceeded the maximum execution time of 360 minutes.

The operation was canceled.

Process completed with exit code 1.

The job running on runner GitHub Actions 246 has exceeded the maximum execution time of 360 minutes.

The operation was canceled.

The job running on runner GitHub Actions 480 has exceeded the maximum execution time of 360 minutes.

The operation was canceled.

The job running on runner GitHub Actions 375 has exceeded the maximum execution time of 360 minutes.

The operation was canceled.

The job running on runner GitHub Actions 308 has exceeded the maximum execution time of 360 minutes.

The operation was canceled.

The job running on runner GitHub Actions 160 has exceeded the maximum execution time of 360 minutes.

The operation was canceled.

The job running on runner GitHub Actions 406 has exceeded the maximum execution time of 360 minutes.

The operation was canceled.

The job running on runner GitHub Actions 479 has exceeded the maximum execution time of 360 minutes.

The operation was canceled.

The job running on runner GitHub Actions 258 has exceeded the maximum execution time of 360 minutes.

The operation was canceled.

The job running on runner GitHub Actions 339 has exceeded the maximum execution time of 360 minutes.

The operation was canceled.

The job running on runner GitHub Actions 357 has exceeded the maximum execution time of 360 minutes.

The operation was canceled.

The job running on runner GitHub Actions 266 has exceeded the maximum execution time of 360 minutes.

The operation was canceled.

The job running on runner GitHub Actions 168 has exceeded the maximum execution time of 360 minutes.

The operation was canceled.

The job running on runner GitHub Actions 463 has exceeded the maximum execution time of 360 minutes.

The operation was canceled.

The job running on runner GitHub Actions 402 has exceeded the maximum execution time of 360 minutes.

The operation was canceled.

The job running on runner GitHub Actions 135 has exceeded the maximum execution time of 360 minutes.

The operation was canceled.

Process completed with exit code 1.

The job running on runner GitHub Actions 142 has exceeded the maximum execution time of 360 minutes.

The operation was canceled.

The job running on runner GitHub Actions 289 has exceeded the maximum execution time of 360 minutes.

The operation was canceled.

The job running on runner GitHub Actions 34 has exceeded the maximum execution time of 360 minutes.

The operation was canceled.

The job running on runner GitHub Actions 45 has exceeded the maximum execution time of 360 minutes.

The operation was canceled.

The job running on runner GitHub Actions 344 has exceeded the maximum execution time of 360 minutes.

The operation was canceled.

The job running on runner GitHub Actions 330 has exceeded the maximum execution time of 360 minutes.

The operation was canceled.

The job running on runner GitHub Actions 148 has exceeded the maximum execution time of 360 minutes.

The operation was canceled.

The job running on runner GitHub Actions 57 has exceeded the maximum execution time of 360 minutes.

The operation was canceled.

The job running on runner GitHub Actions 73 has exceeded the maximum execution time of 360 minutes.

The operation was canceled.

The job running on runner GitHub Actions 27 has exceeded the maximum execution time of 360 minutes.

The operation was canceled.

The job running on runner GitHub Actions 26 has exceeded the maximum execution time of 360 minutes.

The operation was canceled.

The job running on runner GitHub Actions 103 has exceeded the maximum execution time of 360 minutes.

The operation was canceled.

The job running on runner GitHub Actions 461 has exceeded the maximum execution time of 360 minutes.

The operation was canceled.

The job running on runner GitHub Actions 175 has exceeded the maximum execution time of 360 minutes.

The operation was canceled.

The job running on runner GitHub Actions 155 has exceeded the maximum execution time of 360 minutes.

The operation was canceled.

Process completed with exit code 2.

Process completed with exit code 2.

The self-hosted runner: A100-5d5c17c15425 lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.

The self-hosted runner: A100-30a40ee31675 lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.