Issue with UCC Initialization: "mca_coll_ucc_save_coll_handlers"

[j34ni]

Description:

Users may encounter this kind of error related to UCC initialization in OpenMPI:

Error: coll_ucc_module.c:474 - mca_coll_ucc_module_enable() mca_coll_ucc_save_coll_handlers failed

This was documented in the OpenMPI repository (see: open-mpi/ompi#9885).

Apparently the problem arises because the default priority for UCC is 10, which conflicts with the priorities of basic and libnbc.

Workaround:
Users can bypass this issue by setting a higher priority for UCC:

export OMPI_MCA_coll_ucc_priority=100

Recommendation:
Should we not inform users about this issue (and workaround), and possibly look into adjusting the default configurations to avoid such conflict?

[dalcinl]

If there is an upstream patch accepted, once that patch is merged, then we can add that patch here and produce a new build.