--set-env PATH conflicts with pre-defiend PATH on host

[garykuok]

Hello,
I'm testing to run some applications on the ch image (e.g., Paraview). Since this software is not exist on host, I need to run --set-env param to correctly configure the bins when using related commands (e.g., pvbatch, pvpython). However, on the host, I defined some PATH variables in .bashrc, then when running the ch-run commands, it raises error:

pvbatch: symbol lookup error: /paraview/ParaView-5.7.0-osmesa-MPI-Linux-Python3.7-64bit/bin/../lib/libmpifort.so.12: undefined symbol: MPII_F_TRUE

This indicates a conflict with pre-defined MPI paths on the host. If I don't set them in the host, I can successfully run the ch-run command. Any idea of how to resolve this?

[heasterday]

Hello Gary,

I can think of two things to try first:

• Use --unset-env=<MPI VARS> to remove the MPI environment variables pointing to host paths
• Use the ENV directive in your Dockerfile to specify the problematic MPI env variables with values correct for the container environment. These will be exported to the file located at /ch/environment, then when you use --set-env=<path to image>/ch/environment they will be overridden by the correct values.

Do you think one of these may work for your use case?

[garykuok]

Hello Heaster,
I tried the first method with --unset-env. What is the actual syntax? My MPI env is set as path export PATH="/mpich-3.3.2/DIR/bin:$PATH" at host, I tested with --unset-env=PATH or with the whole line, both with no luck.

The second method only happens in ch environment. Since docker is using an isolated container, the host environment doesn't affect inside the container.
The command I used in docker is ENV PATH="/mpich-3.3.2/DIR/bin:$PATH". It works in docker, and in singularity it works as well (but it raises some warning of rootless container when running this image).

[heasterday]

--unset-env supports globbing, so as an example if you want to unset every environment variable that starts with "MPI" you can do something like this, --unset-env=MPI*. --unset-env=PATH should work but instead what I would recommend is --set-env=<path to image>/ch/environment which will override PATH's previous value (if PATH is set in that file).

Could you provide the contents of the image's /ch/environment as well as the command you are using to start the container?

[garykuok]

I have this in my ch/environment:

PATH=/paraview/ParaView-5.7.0-osmesa-MPI-Linux-Python3.7-64bit/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DEBIAN_FRONTEND=noninteractive

Then I execute this command to run the image:

ch-run -w pv_pure --cd=$(pwd) --set-env=pv_pure/ch/environment -- pvbatch $(pwd)/alpaca2dpy.py --xdmf $(pwd)/$i

I have this in my bashrc that I think caused the problem:

export PATH="/home/ckkuok/ALPACA_INSTALL/mpich-3.3.2/DIR/bin:$PATH"
export LD_LIBRARY_PATH="/home/ckkuok/ALPACA_INSTALL/mpich-3.3.2/DIR/include:$LD_LIBRARY_PATH"
export LD_LIBRARY_PATH="/home/ckkuok/ALPACA_INSTALL/mpich-3.3.2/DIR/lib:$LD_LIBRARY_PATH"

I used --set-env already but doesn't work..
Still thank you for the reply.

[heasterday]

After looking at this more the error doesn't look like it's a host/container conflict as it's only reporting the container paths that I can see. Please verify that the linker in the container can find the MPI libraries. If they aren't in the default search paths you will want to set LD_LIBRARY_PATH to include them.

[garykuok]

Another follow related to path problem:
I want to use intel-mpi inside the container, and on host I also have intel-mpi. The intel-mpi inside the container is a little newer than the one on the host. Both of them are of version 2019.
The problem is, the intel libraries require to set proper path to use them (e.g., use "ENV ..." in the ch/environment file). If I set the paths, then it will have conflicts to the host, causing this error:

MPIR_Init_thread(136)........:
MPID_Init(1138)..............:
MPIDI_OFI_mpi_init_hook(1061): OFI addrinfo() failed (ofi_init.c:1061:MPIDI_OFI_mpi_init_hook:No data available)

I searched this online, it's mostly related to FI_PROVIDER_PATH. I don't know how to fix this..

Where as using MPICH inside the container, no particular ENV need to be set. This way it can communicate with the host, but the performance is not as good. Thus, I want to use intel-mpi in the container as well.

[garykuok]

If I do a guest launch with mpiexec inside the container, it also raises error:

[mpiexec@i22r07c05s11] check_exit_codes (../../../../../src/pm/i_hydra/libhydra/demux/hydra_demux_poll.c:117): unable to run bstrap_proxy on i22r07c05s11 (pid 51863, exit code 65280)
[mpiexec@i22r07c05s11] poll_for_event (../../../../../src/pm/i_hydra/libhydra/demux/hydra_demux_poll.c:159): check exit codes error

[garykuok]

Update:

I solved the problem by specifying "FI_PROVIDER=verbs" in the ENV. By default one can use tcp and no need to install anything extra. If infiniband network is available, this can be used to speedup the communication by installing libibverbs and librdmacm in the image, as described in the example Dockerfiles.