Intermittent MPI test failures

[adam-smnk]

The examples/mpi/feed-forward-mpi.py randomly fails.
Root cause should be investigated and the test made stable before reenabling it.

Example failed job dump:

-- Testing: 28 tests, 4 workers --
PASS: Lighthouse test suite :: examples/basic/transform_a_payload_according_to_a_schedule.py (1 of 28)
PASS: Lighthouse test suite :: examples/ingress/mlir_gen/generate-linalg-3layer-mlp.sh (2 of 28)
PASS: Lighthouse test suite :: examples/basic/compile_and_run.py (3 of 28)
PASS: Lighthouse test suite :: examples/ingress/torch/mlp_from_file.py (4 of 28)
PASS: Lighthouse test suite :: examples/ingress/torch/mlp_from_model.py (5 of 28)
PASS: Lighthouse test suite :: examples/ingress/torch/compile_model.py (6 of 28)
FAIL: Lighthouse test suite :: examples/mpi/feed-forward-mpi.py (7 of 28)
******************** TEST 'Lighthouse test suite :: examples/mpi/feed-forward-mpi.py' FAILED ********************
Exit Code: 1

Command Output (stderr):
--
RUN: at line 2: mpirun -n 4 python /home/runner/work/lighthouse/lighthouse/examples/mpi/feed-forward-mpi.py --mpilib=/home/runner/work/lighthouse/lighthouse/.venv/lib/libmpi.so.12 | /usr/bin/FileCheck-18 /home/runner/work/lighthouse/lighthouse/examples/mpi/feed-forward-mpi.py
+ mpirun -n 4 python /home/runner/work/lighthouse/lighthouse/examples/mpi/feed-forward-mpi.py --mpilib=/home/runner/work/lighthouse/lighthouse/.venv/lib/libmpi.so.12
+ /usr/bin/FileCheck-18 /home/runner/work/lighthouse/lighthouse/examples/mpi/feed-forward-mpi.py
Abort(404899215): Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71).: MPI_Init_thread(argc=(nil), argv=(nil), required=3, provided=0x7ffe8a676060) failed
MPII_Init_thread(261)....: 
MPID_Init(549)...........: 
MPIDI_UCX_init_local(234): 
MPIDI_UCX_init_worker(86):  ucx function returned with failed status(ucx_init.c 86 MPIDI_UCX_init_worker Input/output error)
Abort(1008878991): Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71).: MPI_Init_thread(argc=(nil), argv=(nil), required=3, provided=0x7ffc2b7a8fc0) failed
MPII_Init_thread(261)....: 
MPID_Init(549)...........: 
MPIDI_UCX_init_local(234): 
MPIDI_UCX_init_worker(86):  ucx function returned with failed status(ucx_init.c 86 MPIDI_UCX_init_worker Input/output error)
Abort(136463759): Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71).: MPI_Init_thread(argc=(nil), argv=(nil), required=3, provided=0x7ffe591ac650) failed
MPII_Init_thread(261)....: 
MPID_Init(549)...........: 
MPIDI_UCX_init_local(234): 
MPIDI_UCX_init_worker(86):  ucx function returned with failed status(ucx_init.c 86 MPIDI_UCX_init_worker Input/output error)
/home/runner/work/lighthouse/lighthouse/examples/mpi/feed-forward-mpi.py:5:10: error: CHECK: expected string not found in input
# CHECK: PASSED
         ^
<stdin>:1:1: note: scanning from here
[1785831715.877809] [runnervmvrwv9:2894 :0] ib_iface.c:1316 UCX ERROR mana_0: iface 0x3cb598a0 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
^
<stdin>:9:49: note: possible intended match here
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
                                                ^

Input file: <stdin>
Check file: /home/runner/work/lighthouse/lighthouse/examples/mpi/feed-forward-mpi.py

-dump-input=help explains the following input dump.

Input was:
<<<<<<
           1: [1785831715.877809] [runnervmvrwv9:2894 :0] ib_iface.c:1316 UCX ERROR mana_0: iface 0x3cb598a0 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported 
check:5'0     X~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ error: no match found
           2: [1785831715.878738] [runnervmvrwv9:2894 :0] ucp_worker.c:1415 UCX ERROR uct_iface_open(ud_verbs/mana_0:1) failed: Input/output error 
check:5'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           3: [1785831715.879088] [runnervmvrwv9:2891 :0] ib_iface.c:1316 UCX ERROR mana_0: iface 0x384043c0 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported 
check:5'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           4: [1785831715.880186] [runnervmvrwv9:2892 :0] ib_iface.c:1316 UCX ERROR mana_0: iface 0x1f45cc60 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported 
check:5'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           5: [1785831715.880779] [runnervmvrwv9:2891 :0] ucp_worker.c:1415 UCX ERROR uct_iface_open(ud_verbs/mana_0:1) failed: Input/output error 
check:5'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           6: [1785831715.881213] [runnervmvrwv9:2892 :0] ucp_worker.c:1415 UCX ERROR uct_iface_open(ud_verbs/mana_0:1) failed: Input/output error 
check:5'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           7:  
check:5'0     ~
           8: =================================================================================== 
check:5'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           9: = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES 
check:5'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
check:5'1                                                     ?       possible intended match
          10: = PID 2893 RUNNING AT runnervmvrwv9 
check:5'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          11: = EXIT CODE: 9 
check:5'0     ~~~~~~~~~~~~~~~
          12: = CLEANING UP REMAINING PROCESSES 
check:5'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          13: = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES 
check:5'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          14: =================================================================================== 
check:5'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           .
           .
           .
>>>>>>

--

********************
PASS: Lighthouse test suite :: examples/llama/test_llama3.py (8 of 28)
PASS: Lighthouse test suite :: examples/xegpu/fused_attention.py (9 of 28)
PASS: Lighthouse test suite :: examples/xegpu/layer_norm.py (10 of 28)
PASS: Lighthouse test suite :: examples/llama/torch_ingress.py (11 of 28)
PASS: Lighthouse test suite :: examples/xegpu/matmul.py (12 of 28)
PASS: Lighthouse test suite :: examples/xegpu/nanoGPT.py (13 of 28)
PASS: Lighthouse test suite :: examples/xegpu/softmax.py (14 of 28)
UNSUPPORTED: Lighthouse test suite :: examples/xegpu/torch_matmul.py (15 of 28)
PASS: Lighthouse test suite :: examples/xegpu/tune_matmul_costmodel.py (16 of 28)
PASS: Lighthouse test suite :: test/dialects/dialect_loading.py (17 of 28)
PASS: Lighthouse test suite :: test/dialects/transform_ext.py (18 of 28)
PASS: Lighthouse test suite :: test/dialects/transform_ext_subview.py (19 of 28)
PASS: Lighthouse test suite :: examples/xegpu/mlp.py (20 of 28)
PASS: Lighthouse test suite :: test/pipeline/find_pipeline.py (21 of 28)
PASS: Lighthouse test suite :: test/run/pipeline-check.mlir (22 of 28)
PASS: Lighthouse test suite :: test/transform/test_foreach.py (23 of 28)
PASS: Lighthouse test suite :: test/transform/test_pack_lowering.py (24 of 28)
PASS: Lighthouse test suite :: test/transform/test_tile_size_ops.py (25 of 28)
PASS: Lighthouse test suite :: test/opt/pipeline-check.mlir (26 of 28)
PASS: Lighthouse test suite :: examples/xegpu/kernel_bench.py (27 of 28)
PASS: Lighthouse test suite :: examples/KernelBench/test-kernel-bench.py (28 of 28)
********************
Failed Tests (1):
  Lighthouse test suite :: examples/mpi/feed-forward-mpi.py