Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Compilers broken with undefined references to GLIBC_2.2.5 when installed on Rocky 9 OS #130

Open
1 task done
christopherwharrop-noaa opened this issue Dec 1, 2023 · 21 comments
Open
1 task done

Compilers broken with undefined references to GLIBC_2.2.5 when installed on Rocky 9 OS #130

christopherwharrop-noaa opened this issue Dec 1, 2023 · 21 comments
Labels
bug

Comments

@christopherwharrop-noaa
Copy link

Solution to issue cannot be found in the documentation.

  • I checked the documentation.

Issue

Installing OpenMPI and wrappers with Miniconda3 on a Rocky 9 system results in compilers having GLIBC issues. Performing the exact same sequence of installation commands on an old CentOS 7 system works fine. A Spack installation of OpenMPI on the same Rocky 9 system also works just fine.

The environment installed:

name: chiltepin-env

channels:
  - conda-forge

dependencies:
  - openmpi
  - openmpi-mpicc
  - openmpi-mpicxx
  - openmpi-mpifort

Attempts to use mpif90 result in the following:

(chiltepin) [src]$ mpif90 mpi_hello.f90 
$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: $HOME/miniconda3/envs/chiltepin/lib/./libopen-pal.so.40: undefined reference to `openpty@GLIBC_2.2.5'
$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: $HOME/miniconda3/envs/chiltepin/lib/./libopen-pal.so.40: undefined reference to `dlopen@GLIBC_2.2.5'
$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: $HOME/miniconda3/envs/chiltepin/lib/./libopen-pal.so.40: undefined reference to `dlerror@GLIBC_2.2.5'
$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /usr/lib64/libz.so.1: undefined reference to `memcpy@GLIBC_2.14'
$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: $HOME/miniconda3/envs/chiltepin/lib/./libopen-pal.so.40: undefined reference to `dlclose@GLIBC_2.2.5'
$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: $HOME/miniconda3/envs/chiltepin/lib/./libopen-pal.so.40: undefined reference to `dlsym@GLIBC_2.2.5'

Attempts to use mpicc results in the same output, with a few additional undefined references:

$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /usr/lib64/libm.so.6: undefined reference to `__strtold_nan@GLIBC_PRIVATE'
$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /usr/lib64/libm.so.6: undefined reference to `__strtod_nan@GLIBC_PRIVATE'
$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /usr/lib64/libm.so.6: undefined reference to `__strtof_nan@GLIBC_PRIVATE'
$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /usr/lib64/libm.so.6: undefined reference to `__strtof128_nan@GLIBC_PRIVATE'

I tried export OMPI_FC=$FC, but it did not change the output.

Installed packages

# packages in environment at $HOME/miniconda3/envs/chiltepin:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
binutils_impl_linux-64    2.40                 hf600244_0    conda-forge
binutils_linux-64         2.40                 hbdbef99_2    conda-forge
gcc_impl_linux-64         12.3.0               he2b93b0_3    conda-forge
gcc_linux-64              12.3.0               h76fc315_2    conda-forge
gfortran_impl_linux-64    12.3.0               hfcedea8_3    conda-forge
gfortran_linux-64         12.3.0               h7fe76b4_2    conda-forge
gxx_impl_linux-64         12.3.0               he2b93b0_3    conda-forge
gxx_linux-64              12.3.0               h8a814eb_2    conda-forge
kernel-headers_linux-64   2.6.32              he073ed8_16    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
libgcc-devel_linux-64     12.3.0             h8bca6fd_103    conda-forge
libgcc-ng                 13.2.0               h807b86a_3    conda-forge
libgfortran-ng            13.2.0               h69a702a_3    conda-forge
libgfortran5              13.2.0               ha4646dd_3    conda-forge
libgomp                   13.2.0               h807b86a_3    conda-forge
libsanitizer              12.3.0               h0f45ef3_3    conda-forge
libstdcxx-devel_linux-64  12.3.0             h8bca6fd_103    conda-forge
libstdcxx-ng              13.2.0               h7e041cc_3    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
mpi                       1.0                     openmpi    conda-forge
openmpi                   4.1.6              hc5af2df_101    conda-forge
openmpi-mpicc             4.1.6              hd590300_101    conda-forge
openmpi-mpicxx            4.1.6              h00ab1b0_101    conda-forge
openmpi-mpifort           4.1.6              heb67821_101    conda-forge
sysroot_linux-64          2.12                he073ed8_16    conda-forge
zlib                      1.2.13               hd590300_5    conda-forge

Environment info

active environment : chiltepin
    active env location : $HOME/miniconda3/envs/chiltepin
            shell level : 2
       user config file : $HOME/.condarc
 populated config files : 
          conda version : 23.10.0
    conda-build version : not installed
         python version : 3.11.5.final.0
       virtual packages : __archspec=1=icelake
                          __glibc=2.34=0
                          __linux=5.14.0=0
                          __unix=0=0
       base environment : $HOME/miniconda3  (writable)
      conda av data dir : $HOME/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : $HOME/miniconda3/pkgs
                          $HOME/.conda/pkgs
       envs directories : $HOME/miniconda3/envs
                          $HOME/.conda/envs
               platform : linux-64
             user-agent : conda/23.10.0 requests/2.31.0 CPython/3.11.5 Linux/5.14.0-162.12.1.el9_1.0.2.x86_64 rocky/9.1 glibc/2.34 solver/libmamba conda-libmamba-solver/23.11.1 libmambapy/1.5.3
                UID:GID : 7966:17000
             netrc file : None
           offline mode : False
@dalcinl
Copy link
Contributor

dalcinl commented Dec 2, 2023

What's the output of
ldd $HOME/miniconda3/envs/chiltepin/lib/./libopen-pal.so.40
and
readelf -d $HOME/miniconda3/envs/chiltepin/lib/./libopen-pal.so.40
?

I'm not sure what's going on. Any ideas, @minrk ?

@dalcinl
Copy link
Contributor

dalcinl commented Dec 2, 2023

BTW, do this happen only with the Fortran compiler wrapper? Does the mpicc compiler exhibit similar issues?

@christopherwharrop-noaa
Copy link
Author

@dalcinl - The same issue is also present for the mpicc wrapper as well. I have some additional information to report, but first the information you asked for:

(chiltepin) [src]$ ldd $HOME/miniconda3/envs/chiltepin/lib/./libopen-pal.so.40
	linux-vdso.so.1 (0x00007ffdf81c7000)
	libdl.so.2 => /usr/lib64/libdl.so.2 (0x00007f1e9722d000)
	libm.so.6 => /usr/lib64/libm.so.6 (0x00007f1e97152000)
	libutil.so.1 => /usr/lib64/libutil.so.1 (0x00007f1e9714d000)
	libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x00007f1e97148000)
	libc.so.6 => /usr/lib64/libc.so.6 (0x00007f1e96f3f000)
	librt.so.1 => /usr/lib64/librt.so.1 (0x00007f1e96f38000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f1e97334000)
(chiltepin) [src]$ readelf -d $HOME/miniconda3/envs/chiltepin/lib/./libopen-pal.so.40

Dynamic section at offset 0xf17a0 contains 34 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libutil.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [librt.so.1]
 0x000000000000000e (SONAME)             Library soname: [libopen-pal.so.40]
 0x000000000000000f (RPATH)              Library rpath: [$ORIGIN/.]
 0x000000000000000c (INIT)               0x2a000
 0x000000000000000d (FINI)               0xbe6ac
 0x0000000000000019 (INIT_ARRAY)         0xef5f0
 0x000000000000001b (INIT_ARRAYSZ)       8 (bytes)
 0x000000000000001a (FINI_ARRAY)         0xef5f8
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x0000000000000004 (HASH)               0x200
 0x000000006ffffef5 (GNU_HASH)           0x2ea0
 0x0000000000000005 (STRTAB)             0x10c68
 0x0000000000000006 (SYMTAB)             0x5fd0
 0x000000000000000a (STRSZ)              47089 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000003 (PLTGOT)             0xf1a00
 0x0000000000000002 (PLTRELSZ)           24 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x29a88
 0x0000000000000007 (RELA)               0x1d410
 0x0000000000000008 (RELASZ)             50808 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x0000000000000018 (BIND_NOW)           
 0x000000006ffffffb (FLAGS_1)            Flags: NOW
 0x000000006ffffffe (VERNEED)            0x1d2c0
 0x000000006fffffff (VERNEEDNUM)         6
 0x000000006ffffff0 (VERSYM)             0x1c45a
 0x000000006ffffff9 (RELACOUNT)          633
 0x0000000000000000 (NULL)               0x0
(chiltepin) [src]$

I noticed that the environment output shows __glibc=2.34=0 but the installed packages shows binutils_linux-64 2.40. My understanding is that should not be a problem. But, if I add binutils_linux-64=2.34 to my environment yaml and install, the errors go away and the wrapper compiles my code. However, a side effect is that the underlying compiler versions are ancient, 7.5.0.

name: chiltepin-env

channels:
  - conda-forge

dependencies:
  - binutils_linux-64=2.34
  - openmpi
  - openmpi-mpicc
  - openmpi-mpicxx
  - openmpi-mpifort

(chiltepin) [src]$ mpif90 mpi_hello.f90 
(chiltepin) [src]$ mpif90 -show
x86_64-conda-linux-gnu-gfortran -I$HOME/miniconda3/envs/chiltepin/include -I$HOME/miniconda3/envs/chiltepin/include -I$HOME/miniconda3/envs/chiltepin/lib -L$HOME/miniconda3/envs/chiltepin/lib -L$HOME/miniconda3/envs/chiltepin/lib -Wl,-rpath,$HOME/miniconda3/envs/chiltepin/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi

(chiltepin) [src]$ x86_64-conda-linux-gnu-gfortran --version
GNU Fortran (crosstool-NG 1.24.0.133_b0863d8_dirty) 7.5.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

(chiltepin) [src]$

And, of course, there is also the other problem that having to specify specific versions of binutils for each machine makes my conda environment spec completely unportable and difficult to manage.

@christopherwharrop-noaa
Copy link
Author

Oh, and one other thing you might need to know... The /usr/lib64/libc.so.6 is not a symlink.

(chiltepin) [src]$ ls -l /usr/lib64/libc.so.6
-rwxr-xr-x 1 root root 2386744 Jan 23  2023 /usr/lib64/libc.so.6*

(chiltepin) [src]$ file /usr/lib64/libc.so.6
/usr/lib64/libc.so.6: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=82f7ae28e16376aa97cc3bf50b40ab2d1043924a, for GNU/Linux 3.2.0, not stripped
(chiltepin) [src]$

@christopherwharrop-noaa
Copy link
Author

It appears that the compiler that gets installed in my chiltepin environment (when I don't include the binutils spec) doesn't work properly, but the include files and libraries for OpenMPI does work. I say this because if I use the wrappers that conda installs, it doesn't work, but if I use the system gfortran and manually append all of the MPI related includes and libs for the conda OpenMPI install, it does work. The system's default GNU is 11.3.1. See the following (I don't know why mpif90 -show is showing duplicate -I and -L paths):

(chiltepin) [src]$ mpif90 -show
x86_64-conda-linux-gnu-gfortran -I$HOME/miniconda3/envs/chiltepin/include -pthread -I$HOME/miniconda3/envs/chiltepin/include -I$HOME/miniconda3/envs/chiltepin/lib -L$HOME/miniconda3/envs/chiltepin/lib -L$HOME/miniconda3/envs/chiltepin/lib -Wl,-rpath,$HOME/miniconda3/envs/chiltepin/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi

(chiltepin) [src]$ x86_64-conda-linux-gnu-gfortran --version
GNU Fortran (conda-forge gcc 12.3.0-3) 12.3.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

(chiltepin) [src]$ which gcc
/usr/bin/gcc
(chiltepin) [src]$ gcc --version
gcc (GCC) 11.3.1 20220421 (Red Hat 11.3.1-2)
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

(chiltepin) [src]$ which gfortran
/usr/bin/gfortran
(chiltepin) [src]$ gfortran --version
GNU Fortran (GCC) 11.3.1 20220421 (Red Hat 11.3.1-2)
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

(chiltepin) [src]$ gfortran mpi_hello.f90
mpi_hello.f90:4:7:

    4 |   use mpi
      |       1
Fatal Error: Cannot open module file ‘mpi.mod’ for reading at (1): No such file or directory
compilation terminated.

(chiltepin) [src]$ gfortran mpi_hello.f90 -I$HOME/miniconda3/envs/chiltepin/include -pthread -I$HOME/miniconda3/envs/chiltepin/include -I$HOME/miniconda3/envs/chiltepin/lib -L$HOME/miniconda3/envs/chiltepin/lib -L$HOME/miniconda3/envs/chiltepin/lib -Wl,-rpath,$HOME/miniconda3/envs/chiltepin/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi
(chiltepin) [src]$ 

(chiltepin) [src]$ mpif90 mpi_hello.f90 
$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: $HOME/miniconda3/envs/chiltepin/lib/./libopen-pal.so.40: undefined reference to `openpty@GLIBC_2.2.5'
$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: $HOME/miniconda3/envs/chiltepin/lib/./libopen-pal.so.40: undefined reference to `dlopen@GLIBC_2.2.5'
$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: $HOME/miniconda3/envs/chiltepin/lib/./libopen-pal.so.40: undefined reference to `dlerror@GLIBC_2.2.5'
$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /usr/lib64/libz.so.1: undefined reference to `memcpy@GLIBC_2.14'
$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: $HOME/miniconda3/envs/chiltepin/lib/./libopen-pal.so.40: undefined reference to `dlclose@GLIBC_2.2.5'
$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: $HOME/miniconda3/envs/chiltepin/lib/./libopen-pal.so.40: undefined reference to `dlsym@GLIBC_2.2.5'
collect2: error: ld returned 1 exit status
(chiltepin) [src]$

@dalcinl
Copy link
Contributor

dalcinl commented Dec 5, 2023

So the system compiler is OLDER than than the conda-forge compiler? That may be the root of all evil.

@conda-forge/core I'm not sure what's the way to deal with these ABI issues.

@isuruf
Copy link
Member

isuruf commented Dec 5, 2023

What's the output of mpif90 mpi_hello.f90 -v -Wl,-v?

@isuruf
Copy link
Member

isuruf commented Dec 5, 2023

And what's the output of mpif90 mpi_hello.f90 -v -Wl,-v -Wl,-rpath-link,$HOME/miniconda3/envs/chiltepin/lib

@christopherwharrop-noaa
Copy link
Author 

(chiltepin) [src]$ mpif90 mpi_hello.f90 -v -Wl,-v
Driving: $HOME/miniconda3/envs/chiltepin/bin/x86_64-conda-linux-gnu-gfortran mpi_hello.f90 -v -Wl,-v -I$HOME/miniconda3/envs/chiltepin/include -pthread -I$HOME/miniconda3/envs/chiltepin/include -I$HOME/miniconda3/envs/chiltepin/lib -L$HOME/miniconda3/envs/chiltepin/lib -L$HOME/miniconda3/envs/chiltepin/lib -Wl,-rpath,$HOME/miniconda3/envs/chiltepin/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -l gfortran -l m -shared-libgcc
Reading specs from $HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/specs
could not find specs file conda.specs
COLLECT_GCC=$HOME/miniconda3/envs/chiltepin/bin/x86_64-conda-linux-gnu-gfortran
COLLECT_LTO_WRAPPER=$HOME/miniconda3/envs/chiltepin/bin/../libexec/gcc/x86_64-conda-linux-gnu/12.3.0/lto-wrapper
Target: x86_64-conda-linux-gnu
Configured with: ../configure --prefix=/home/conda/feedstock_root/build_artifacts/gcc_compilers_1699751145211/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho --with-slibdir=/home/conda/feedstock_root/build_artifacts/gcc_compilers_1699751145211/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib --libdir=/home/conda/feedstock_root/build_artifacts/gcc_compilers_1699751145211/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib --mandir=/home/conda/feedstock_root/build_artifacts/gcc_compilers_1699751145211/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/man --build=x86_64-conda-linux-gnu --host=x86_64-conda-linux-gnu --target=x86_64-conda-linux-gnu --enable-default-pie --enable-languages=c,c++,fortran,objc,obj-c++ --enable-__cxa_atexit --disable-libmudflap --enable-libgomp --disable-libssp --enable-libquadmath --enable-libquadmath-support --enable-libsanitizer --enable-lto --enable-threads=posix --enable-target-optspace --enable-plugin --enable-gold --disable-nls --disable-bootstrap --disable-multilib --enable-long-long --with-sysroot=/home/conda/feedstock_root/build_artifacts/gcc_compilers_1699751145211/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/x86_64-conda-linux-gnu/sysroot --with-build-sysroot=/home/conda/feedstock_root/build_artifacts/gcc_compilers_1699751145211/_build_env/x86_64-conda-linux-gnu/sysroot --with-gxx-include-dir=/home/conda/feedstock_root/build_artifacts/gcc_compilers_1699751145211/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/x86_64-conda-linux-gnu/include/c++/12.3.0 --with-pkgversion='conda-forge gcc 12.3.0-3' --with-bugurl=https://github.com/conda-forge/ctng-compilers-feedstock/issues/new/choose
Thread model: posix
Supported LTO compression algorithms: zlib
gcc version 12.3.0 (conda-forge gcc 12.3.0-3) 
COLLECT_GCC_OPTIONS='-v' '-I' '$HOME/miniconda3/envs/chiltepin/include' '-pthread' '-I' '$HOME/miniconda3/envs/chiltepin/include' '-I' '$HOME/miniconda3/envs/chiltepin/lib' '-L$HOME/miniconda3/envs/chiltepin/lib' '-L$HOME/miniconda3/envs/chiltepin/lib' '-shared-libgcc' '-mtune=generic' '-march=x86-64' '-dumpdir' 'a-'
 $HOME/miniconda3/envs/chiltepin/bin/../libexec/gcc/x86_64-conda-linux-gnu/12.3.0/f951 mpi_hello.f90 -I $HOME/miniconda3/envs/chiltepin/include -I $HOME/miniconda3/envs/chiltepin/include -I $HOME/miniconda3/envs/chiltepin/lib -quiet -dumpdir a- -dumpbase mpi_hello.f90 -dumpbase-ext .f90 -mtune=generic -march=x86-64 -version -fintrinsic-modules-path $HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/finclude -o /tmp/ccVjR6RW.s
GNU Fortran (conda-forge gcc 12.3.0-3) version 12.3.0 (x86_64-conda-linux-gnu)
	compiled by GNU C version 12.3.0, GMP version 6.2.1, MPFR version 4.1.0, MPC version 1.2.1, isl version isl-0.24-GMP

GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
GNU Fortran2008 (conda-forge gcc 12.3.0-3) version 12.3.0 (x86_64-conda-linux-gnu)
	compiled by GNU C version 12.3.0, GMP version 6.2.1, MPFR version 4.1.0, MPC version 1.2.1, isl version isl-0.24-GMP

GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
COLLECT_GCC_OPTIONS='-v' '-I' '$HOME/miniconda3/envs/chiltepin/include' '-pthread' '-I' '$HOME/miniconda3/envs/chiltepin/include' '-I' '$HOME/miniconda3/envs/chiltepin/lib' '-L$HOME/miniconda3/envs/chiltepin/lib' '-L$HOME/miniconda3/envs/chiltepin/lib' '-shared-libgcc' '-mtune=generic' '-march=x86-64' '-dumpdir' 'a-'
 $HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/as -v -I $HOME/miniconda3/envs/chiltepin/include -I $HOME/miniconda3/envs/chiltepin/include -I $HOME/miniconda3/envs/chiltepin/lib --64 -o /tmp/ccdEl19P.o /tmp/ccVjR6RW.s
GNU assembler version 2.40 (x86_64-conda-linux-gnu) using BFD version (GNU Binutils) 2.40
Reading specs from $HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/lib/../lib/libgfortran.spec
rename spec lib to liborig
COLLECT_GCC_OPTIONS='-v' '-I' '$HOME/miniconda3/envs/chiltepin/include' '-pthread' '-I' '$HOME/miniconda3/envs/chiltepin/include' '-I' '$HOME/miniconda3/envs/chiltepin/lib' '-L$HOME/miniconda3/envs/chiltepin/lib' '-L$HOME/miniconda3/envs/chiltepin/lib' '-shared-libgcc' '-mtune=generic' '-march=x86-64' '-dumpdir' 'a-'
COMPILER_PATH=$HOME/miniconda3/envs/chiltepin/bin/../libexec/gcc/x86_64-conda-linux-gnu/12.3.0/:$HOME/miniconda3/envs/chiltepin/bin/../libexec/gcc/:$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/
LIBRARY_PATH=$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/:$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/:$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/lib/../lib/:$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../lib/:$HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot/lib/../lib/:$HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib/../lib/:$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/lib/:$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../:$HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot/lib/:$HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib/
COLLECT_GCC_OPTIONS='-v' '-I' '$HOME/miniconda3/envs/chiltepin/include' '-pthread' '-I' '$HOME/miniconda3/envs/chiltepin/include' '-I' '$HOME/miniconda3/envs/chiltepin/lib' '-L$HOME/miniconda3/envs/chiltepin/lib' '-L$HOME/miniconda3/envs/chiltepin/lib' '-shared-libgcc' '-mtune=generic' '-march=x86-64' '-dumpdir' 'a.'
 $HOME/miniconda3/envs/chiltepin/bin/../libexec/gcc/x86_64-conda-linux-gnu/12.3.0/collect2 -plugin $HOME/miniconda3/envs/chiltepin/bin/../libexec/gcc/x86_64-conda-linux-gnu/12.3.0/liblto_plugin.so -plugin-opt=$HOME/miniconda3/envs/chiltepin/bin/../libexec/gcc/x86_64-conda-linux-gnu/12.3.0/lto-wrapper -plugin-opt=-fresolution=/tmp/ccdeg3f3.res -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lquadmath -plugin-opt=-pass-through=-lm -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lpthread -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc --sysroot=$HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -pie $HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib/../lib/Scrt1.o $HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib/../lib/crti.o $HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/crtbeginS.o -L$HOME/miniconda3/envs/chiltepin/lib -L$HOME/miniconda3/envs/chiltepin/lib -L$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0 -L$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc -L$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/lib/../lib -L$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../lib -L$HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot/lib/../lib -L$HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib/../lib -L$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/lib -L$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../.. -L$HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot/lib -L$HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib /tmp/ccdEl19P.o -v -rpath $HOME/miniconda3/envs/chiltepin/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -lgcc_s -lgcc -lquadmath -lm -lgcc_s -lgcc -lpthread -lc -lgcc_s -lgcc $HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/crtendS.o $HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib/../lib/crtn.o -rpath $HOME/miniconda3/envs/chiltepin/lib
collect2 version 12.3.0
$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld -plugin $HOME/miniconda3/envs/chiltepin/bin/../libexec/gcc/x86_64-conda-linux-gnu/12.3.0/liblto_plugin.so -plugin-opt=$HOME/miniconda3/envs/chiltepin/bin/../libexec/gcc/x86_64-conda-linux-gnu/12.3.0/lto-wrapper -plugin-opt=-fresolution=/tmp/ccdeg3f3.res -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lquadmath -plugin-opt=-pass-through=-lm -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lpthread -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc --sysroot=$HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -pie $HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib/../lib/Scrt1.o $HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib/../lib/crti.o $HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/crtbeginS.o -L$HOME/miniconda3/envs/chiltepin/lib -L$HOME/miniconda3/envs/chiltepin/lib -L$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0 -L$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc -L$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/lib/../lib -L$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../lib -L$HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot/lib/../lib -L$HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib/../lib -L$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/lib -L$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../.. -L$HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot/lib -L$HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib /tmp/ccdEl19P.o -v -rpath $HOME/miniconda3/envs/chiltepin/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -lgcc_s -lgcc -lquadmath -lm -lgcc_s -lgcc -lpthread -lc -lgcc_s -lgcc $HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/crtendS.o $HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib/../lib/crtn.o -rpath $HOME/miniconda3/envs/chiltepin/lib
GNU ld (GNU Binutils) 2.40
$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: $HOME/miniconda3/envs/chiltepin/lib/./libopen-pal.so.40: undefined reference to `openpty@GLIBC_2.2.5'
$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: $HOME/miniconda3/envs/chiltepin/lib/./libopen-pal.so.40: undefined reference to `dlopen@GLIBC_2.2.5'
$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: $HOME/miniconda3/envs/chiltepin/lib/./libopen-pal.so.40: undefined reference to `dlerror@GLIBC_2.2.5'
$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /usr/lib64/libz.so.1: undefined reference to `memcpy@GLIBC_2.14'
$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: $HOME/miniconda3/envs/chiltepin/lib/./libopen-pal.so.40: undefined reference to `dlclose@GLIBC_2.2.5'
$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: $HOME/miniconda3/envs/chiltepin/lib/./libopen-pal.so.40: undefined reference to `dlsym@GLIBC_2.2.5'
collect2: error: ld returned 1 exit status
(chiltepin) [src]$

@christopherwharrop-noaa
Copy link
Author 

(chiltepin) [src]$ mpif90 mpi_hello.f90 -v -Wl,-v -Wl,-rpath-link,$HOME/miniconda3/envs/chiltepin/lib
Driving: $HOME/miniconda3/envs/chiltepin/bin/x86_64-conda-linux-gnu-gfortran mpi_hello.f90 -v -Wl,-v -Wl,-rpath-link,$HOME/miniconda3/envs/chiltepin/lib -I$HOME/miniconda3/envs/chiltepin/include -pthread -I$HOME/miniconda3/envs/chiltepin/include -I$HOME/miniconda3/envs/chiltepin/lib -L$HOME/miniconda3/envs/chiltepin/lib -L$HOME/miniconda3/envs/chiltepin/lib -Wl,-rpath,$HOME/miniconda3/envs/chiltepin/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -l gfortran -l m -shared-libgcc
Reading specs from $HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/specs
could not find specs file conda.specs
COLLECT_GCC=$HOME/miniconda3/envs/chiltepin/bin/x86_64-conda-linux-gnu-gfortran
COLLECT_LTO_WRAPPER=$HOME/miniconda3/envs/chiltepin/bin/../libexec/gcc/x86_64-conda-linux-gnu/12.3.0/lto-wrapper
Target: x86_64-conda-linux-gnu
Configured with: ../configure --prefix=/home/conda/feedstock_root/build_artifacts/gcc_compilers_1699751145211/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho --with-slibdir=/home/conda/feedstock_root/build_artifacts/gcc_compilers_1699751145211/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib --libdir=/home/conda/feedstock_root/build_artifacts/gcc_compilers_1699751145211/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib --mandir=/home/conda/feedstock_root/build_artifacts/gcc_compilers_1699751145211/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/man --build=x86_64-conda-linux-gnu --host=x86_64-conda-linux-gnu --target=x86_64-conda-linux-gnu --enable-default-pie --enable-languages=c,c++,fortran,objc,obj-c++ --enable-__cxa_atexit --disable-libmudflap --enable-libgomp --disable-libssp --enable-libquadmath --enable-libquadmath-support --enable-libsanitizer --enable-lto --enable-threads=posix --enable-target-optspace --enable-plugin --enable-gold --disable-nls --disable-bootstrap --disable-multilib --enable-long-long --with-sysroot=/home/conda/feedstock_root/build_artifacts/gcc_compilers_1699751145211/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/x86_64-conda-linux-gnu/sysroot --with-build-sysroot=/home/conda/feedstock_root/build_artifacts/gcc_compilers_1699751145211/_build_env/x86_64-conda-linux-gnu/sysroot --with-gxx-include-dir=/home/conda/feedstock_root/build_artifacts/gcc_compilers_1699751145211/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/x86_64-conda-linux-gnu/include/c++/12.3.0 --with-pkgversion='conda-forge gcc 12.3.0-3' --with-bugurl=https://github.com/conda-forge/ctng-compilers-feedstock/issues/new/choose
Thread model: posix
Supported LTO compression algorithms: zlib
gcc version 12.3.0 (conda-forge gcc 12.3.0-3) 
COLLECT_GCC_OPTIONS='-v' '-I' '$HOME/miniconda3/envs/chiltepin/include' '-pthread' '-I' '$HOME/miniconda3/envs/chiltepin/include' '-I' '$HOME/miniconda3/envs/chiltepin/lib' '-L$HOME/miniconda3/envs/chiltepin/lib' '-L$HOME/miniconda3/envs/chiltepin/lib' '-shared-libgcc' '-mtune=generic' '-march=x86-64' '-dumpdir' 'a-'
 $HOME/miniconda3/envs/chiltepin/bin/../libexec/gcc/x86_64-conda-linux-gnu/12.3.0/f951 mpi_hello.f90 -I $HOME/miniconda3/envs/chiltepin/include -I $HOME/miniconda3/envs/chiltepin/include -I $HOME/miniconda3/envs/chiltepin/lib -quiet -dumpdir a- -dumpbase mpi_hello.f90 -dumpbase-ext .f90 -mtune=generic -march=x86-64 -version -fintrinsic-modules-path $HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/finclude -o /tmp/ccx0OhjF.s
GNU Fortran (conda-forge gcc 12.3.0-3) version 12.3.0 (x86_64-conda-linux-gnu)
	compiled by GNU C version 12.3.0, GMP version 6.2.1, MPFR version 4.1.0, MPC version 1.2.1, isl version isl-0.24-GMP

GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
GNU Fortran2008 (conda-forge gcc 12.3.0-3) version 12.3.0 (x86_64-conda-linux-gnu)
	compiled by GNU C version 12.3.0, GMP version 6.2.1, MPFR version 4.1.0, MPC version 1.2.1, isl version isl-0.24-GMP

GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
COLLECT_GCC_OPTIONS='-v' '-I' '$HOME/miniconda3/envs/chiltepin/include' '-pthread' '-I' '$HOME/miniconda3/envs/chiltepin/include' '-I' '$HOME/miniconda3/envs/chiltepin/lib' '-L$HOME/miniconda3/envs/chiltepin/lib' '-L$HOME/miniconda3/envs/chiltepin/lib' '-shared-libgcc' '-mtune=generic' '-march=x86-64' '-dumpdir' 'a-'
 $HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/as -v -I $HOME/miniconda3/envs/chiltepin/include -I $HOME/miniconda3/envs/chiltepin/include -I $HOME/miniconda3/envs/chiltepin/lib --64 -o /tmp/cc62qNDD.o /tmp/ccx0OhjF.s
GNU assembler version 2.40 (x86_64-conda-linux-gnu) using BFD version (GNU Binutils) 2.40
Reading specs from $HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/lib/../lib/libgfortran.spec
rename spec lib to liborig
COLLECT_GCC_OPTIONS='-v' '-I' '$HOME/miniconda3/envs/chiltepin/include' '-pthread' '-I' '$HOME/miniconda3/envs/chiltepin/include' '-I' '$HOME/miniconda3/envs/chiltepin/lib' '-L$HOME/miniconda3/envs/chiltepin/lib' '-L$HOME/miniconda3/envs/chiltepin/lib' '-shared-libgcc' '-mtune=generic' '-march=x86-64' '-dumpdir' 'a-'
COMPILER_PATH=$HOME/miniconda3/envs/chiltepin/bin/../libexec/gcc/x86_64-conda-linux-gnu/12.3.0/:$HOME/miniconda3/envs/chiltepin/bin/../libexec/gcc/:$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/
LIBRARY_PATH=$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/:$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/:$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/lib/../lib/:$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../lib/:$HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot/lib/../lib/:$HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib/../lib/:$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/lib/:$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../:$HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot/lib/:$HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib/
COLLECT_GCC_OPTIONS='-v' '-I' '$HOME/miniconda3/envs/chiltepin/include' '-pthread' '-I' '$HOME/miniconda3/envs/chiltepin/include' '-I' '$HOME/miniconda3/envs/chiltepin/lib' '-L$HOME/miniconda3/envs/chiltepin/lib' '-L$HOME/miniconda3/envs/chiltepin/lib' '-shared-libgcc' '-mtune=generic' '-march=x86-64' '-dumpdir' 'a.'
 $HOME/miniconda3/envs/chiltepin/bin/../libexec/gcc/x86_64-conda-linux-gnu/12.3.0/collect2 -plugin $HOME/miniconda3/envs/chiltepin/bin/../libexec/gcc/x86_64-conda-linux-gnu/12.3.0/liblto_plugin.so -plugin-opt=$HOME/miniconda3/envs/chiltepin/bin/../libexec/gcc/x86_64-conda-linux-gnu/12.3.0/lto-wrapper -plugin-opt=-fresolution=/tmp/ccf9ZW7S.res -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lquadmath -plugin-opt=-pass-through=-lm -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lpthread -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc --sysroot=$HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -pie $HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib/../lib/Scrt1.o $HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib/../lib/crti.o $HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/crtbeginS.o -L$HOME/miniconda3/envs/chiltepin/lib -L$HOME/miniconda3/envs/chiltepin/lib -L$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0 -L$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc -L$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/lib/../lib -L$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../lib -L$HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot/lib/../lib -L$HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib/../lib -L$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/lib -L$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../.. -L$HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot/lib -L$HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib /tmp/cc62qNDD.o -v -rpath-link $HOME/miniconda3/envs/chiltepin/lib -rpath $HOME/miniconda3/envs/chiltepin/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -lgcc_s -lgcc -lquadmath -lm -lgcc_s -lgcc -lpthread -lc -lgcc_s -lgcc $HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/crtendS.o $HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib/../lib/crtn.o -rpath $HOME/miniconda3/envs/chiltepin/lib
collect2 version 12.3.0
$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld -plugin $HOME/miniconda3/envs/chiltepin/bin/../libexec/gcc/x86_64-conda-linux-gnu/12.3.0/liblto_plugin.so -plugin-opt=$HOME/miniconda3/envs/chiltepin/bin/../libexec/gcc/x86_64-conda-linux-gnu/12.3.0/lto-wrapper -plugin-opt=-fresolution=/tmp/ccf9ZW7S.res -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lquadmath -plugin-opt=-pass-through=-lm -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lpthread -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc --sysroot=$HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -pie $HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib/../lib/Scrt1.o $HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib/../lib/crti.o $HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/crtbeginS.o -L$HOME/miniconda3/envs/chiltepin/lib -L$HOME/miniconda3/envs/chiltepin/lib -L$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0 -L$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc -L$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/lib/../lib -L$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../lib -L$HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot/lib/../lib -L$HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib/../lib -L$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/lib -L$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../.. -L$HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot/lib -L$HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib /tmp/cc62qNDD.o -v -rpath-link $HOME/miniconda3/envs/chiltepin/lib -rpath $HOME/miniconda3/envs/chiltepin/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -lgcc_s -lgcc -lquadmath -lm -lgcc_s -lgcc -lpthread -lc -lgcc_s -lgcc $HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/crtendS.o $HOME/miniconda3/envs/chiltepin/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib/../lib/crtn.o -rpath $HOME/miniconda3/envs/chiltepin/lib
GNU ld (GNU Binutils) 2.40
$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: $HOME/miniconda3/envs/chiltepin/lib/libopen-pal.so.40: undefined reference to `openpty@GLIBC_2.2.5'
$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: $HOME/miniconda3/envs/chiltepin/lib/libopen-pal.so.40: undefined reference to `dlopen@GLIBC_2.2.5'
$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: $HOME/miniconda3/envs/chiltepin/lib/libopen-pal.so.40: undefined reference to `dlerror@GLIBC_2.2.5'
$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: $HOME/miniconda3/envs/chiltepin/lib/libopen-pal.so.40: undefined reference to `dlclose@GLIBC_2.2.5'
$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: $HOME/miniconda3/envs/chiltepin/lib/libopen-pal.so.40: undefined reference to `dlsym@GLIBC_2.2.5'
collect2: error: ld returned 1 exit status
(chiltepin) [src]$

@christopherwharrop-noaa
Copy link
Author

christopherwharrop-noaa commented Dec 5, 2023 •
edited
Loading

Another thing I tried.... I noticed that the system I'm on has gcc version 12.2.0 installed via Spack and is available as an LMOD module. So, I tried loading it to change the default system version of GCC to more closely match the one in the conda environment.

(chiltepin) [src]$ module load gcc
(chiltepin) [src]$ module list

Currently Loaded Modules:
  1) zlib/1.2.13   2) gcc/12.2.0

 
(chiltepin) [src]$ gfortran --version
GNU Fortran (Spack GCC) 12.2.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

(chiltepin) [src]$ gcc --version
gcc (Spack GCC) 12.2.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

(chiltepin) [src]$

This results in a different error:

(chiltepin) [src]$ mpif90 mpi_hello.f90 
x86_64-conda-linux-gnu-gfortran: fatal error: cannot execute 'f951': execvp: No such file or directory
compilation terminated.
(chiltepin) [src]$

I also tried completely deleting miniconda3 and reinstalling both it and my chiltepin environment while the gcc/12.2.0 module was loaded on my path. It did not change the outcome. Same error as above.

@isuruf
Copy link
Member

isuruf commented Dec 5, 2023

Is there a way to reproduce this? Docker?

@christopherwharrop-noaa
Copy link
Author

It will take me some time, but I can try to make a Docker container that reproduces it. I'm not sure I'll be successful, but I can give it a shot.

@h-vetinari
Copy link
Member 

/usr/lib64/libz.so.1: undefined reference to `memcpy@GLIBC_2.14'

That's possibly related to

sysroot_linux-64          2.12                he073ed8_16    conda-forge

(since you have glibc 2.34, I don't know why it would fail to find a symbol that exists since 2.14)

Out of curiosity, could you try installing sysroot_linux-64=2.17?

@isuruf
Copy link
Member

isuruf commented Dec 5, 2023

That's a good observation.

(since you have glibc 2.34, I don't know why it would fail to find a symbol that exists since 2.14)

It does not matter what is in the system while linking. It matter what's given to the linker.

@christopherwharrop-noaa
Copy link
Author

@h-vetinari / @isuruf - As suggested I added sysroot_linux-64=2.17:

name: chiltepin-env

channels:
  - conda-forge

dependencies:
  - sysroot_linux-64=2.17
  - openmpi
  - openmpi-mpicc
  - openmpi-mpicxx
  - openmpi-mpifort

This yields:

# packages in environment at $HOME/miniconda3/envs/chiltepin:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
_sysroot_linux-64_curr_repodata_hack 3                   h69a702a_13    conda-forge
binutils_impl_linux-64    2.40                 hf600244_0    conda-forge
binutils_linux-64         2.40                 hbdbef99_2    conda-forge
gcc_impl_linux-64         12.3.0               he2b93b0_3    conda-forge
gcc_linux-64              12.3.0               h76fc315_2    conda-forge
gfortran_impl_linux-64    12.3.0               hfcedea8_3    conda-forge
gfortran_linux-64         12.3.0               h7fe76b4_2    conda-forge
gxx_impl_linux-64         12.3.0               he2b93b0_3    conda-forge
gxx_linux-64              12.3.0               h8a814eb_2    conda-forge
kernel-headers_linux-64   3.10.0              h4a8ded7_13    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
libgcc-devel_linux-64     12.3.0             h8bca6fd_103    conda-forge
libgcc-ng                 13.2.0               h807b86a_3    conda-forge
libgfortran-ng            13.2.0               h69a702a_3    conda-forge
libgfortran5              13.2.0               ha4646dd_3    conda-forge
libgomp                   13.2.0               h807b86a_3    conda-forge
libsanitizer              12.3.0               h0f45ef3_3    conda-forge
libstdcxx-devel_linux-64  12.3.0             h8bca6fd_103    conda-forge
libstdcxx-ng              13.2.0               h7e041cc_3    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
mpi                       1.0                     openmpi    conda-forge
openmpi                   4.1.6              hc5af2df_101    conda-forge
openmpi-mpicc             4.1.6              hd590300_101    conda-forge
openmpi-mpicxx            4.1.6              h00ab1b0_101    conda-forge
openmpi-mpifort           4.1.6              heb67821_101    conda-forge
sysroot_linux-64          2.17                h4a8ded7_13    conda-forge
zlib                      1.2.13               hd590300_5    conda-forge

The memcpy reference error went away, but the others remained and additional undefined references were added:

(chiltepin) [src]$ mpif90 mpi_hello.f90 
$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: $HOME/miniconda3/envs/chiltepin/lib/./libopen-pal.so.40: undefined reference to `openpty@GLIBC_2.2.5'
$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: $HOME/miniconda3/envs/chiltepin/lib/./libopen-pal.so.40: undefined reference to `dlopen@GLIBC_2.2.5'
$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: $HOME/miniconda3/envs/chiltepin/lib/./libopen-pal.so.40: undefined reference to `dlerror@GLIBC_2.2.5'
$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: $HOME/miniconda3/envs/chiltepin/lib/./libopen-pal.so.40: undefined reference to `dlclose@GLIBC_2.2.5'
$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: $HOME/miniconda3/envs/chiltepin/lib/./libopen-pal.so.40: undefined reference to `dlsym@GLIBC_2.2.5'
$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: $HOME/miniconda3/envs/chiltepin/lib/libmpi.so: undefined reference to `clock_getres@GLIBC_2.2.5'
$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: $HOME/miniconda3/envs/chiltepin/lib/libmpi.so: undefined reference to `clock_gettime@GLIBC_2.2.5'
$HOME/miniconda3/envs/chiltepin/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /usr/lib64/librt.so.1: undefined reference to `__libc_unwind_link_get@GLIBC_PRIVATE'
collect2: error: ld returned 1 exit status
(chiltepin) [src]$

@christopherwharrop-noaa
Copy link
Author

I see that there is a Rocky base image on docker hub with a version 9 tag. I'll see if I can make a container that reproduces this. The thing is, I'm not sure what the folks who administer the machine I'm working on have done to it beyond the standard Rocky 9 OS (if anything). So, I don't know if I'll be able to reproduce. I'll try.

@christopherwharrop-noaa
Copy link
Author

I have made some attempts to build a Docker container to reproduce the problem, but have not yet been successful. I was able to get a Rocky 9.1 (the machine I'm trying to debug has Rocky 9.1) container with miniconda3 installed. And I installed my environment using the same specs. The compiler wrapper works just fine in the container.

However, I did observe some notable differences in the conda info output in the container version. In particular:

My machine:

       virtual packages : __archspec=1=icelake
                          __conda=23.11.0=0
                          __glibc=2.34=0
                          __linux=5.14.0=0
                          __unix=0=0

             user-agent : conda/23.11.0 requests/2.31.0 CPython/3.11.5 Linux/5.14.0-162.6.1.el9_1.0.1.x86_64 rocky/9.1 glibc/2.34 solver/libmamba conda-libmamba-solver/23.11.1 libmambapy/1.5.3

My container:

       virtual packages : __archspec=1=haswell
                          __conda=23.11.0=0
                          __glibc=2.34=0
                          __linux=6.4.16=0
                          __unix=0=0

             user-agent : conda/23.11.0 requests/2.31.0 CPython/3.11.5 Linux/6.4.16-linuxkit rocky/9.1 glibc/2.34 solver/libmamba conda-libmamba-solver/23.11.1 libmambapy/1.5.3

I do not know why those are different. The __linux=5.14.0=0 vs __linux=6.4.16=0 and Linux/5.14.0-162.6.1.el9_1.0.1.x86_64 vs Linux/6.4.16-linuxkit may be important clues. But I do not know how to get the container to match those from the broken system.

@christopherwharrop-noaa
Copy link
Author

I forgot to mention that the output of conda list is identical for my broken host system and the install in my container. But the differences in conda info show that they are not the same. Container works. The original host doesn't.

@isuruf
Copy link
Member

isuruf commented Dec 6, 2023

Do you have env variables like LIBRARY_PATH set?

@christopherwharrop-noaa
Copy link
Author

Yes.

(base) [~]$ env | grep LIBRARY
LD_LIBRARY_PATH=/usr/lib64:/usr/lib:/opt/slurm/lib
(base) [~]$ ls -al /opt/slurm/lib
total 74256
drwxr-xr-x 3 slurm slurm      312 Oct 25 08:58 ./
drwxr-xr-x 9 slurm slurm      109 Oct 25 13:43 ../
-rw-r--r-- 1 slurm slurm   439904 Oct 25 08:58 libpmi.a
-rwxr-xr-x 1 slurm slurm     1016 Oct 25 08:58 libpmi.la*
lrwxrwxrwx 1 slurm slurm       15 Oct 25 08:58 libpmi.so -> libpmi.so.0.0.0*
lrwxrwxrwx 1 slurm slurm       15 Oct 25 08:58 libpmi.so.0 -> libpmi.so.0.0.0*
-rwxr-xr-x 1 slurm slurm   240816 Oct 25 08:58 libpmi.so.0.0.0*
-rw-r--r-- 1 slurm slurm   491000 Oct 25 08:58 libpmi2.a
-rwxr-xr-x 1 slurm slurm      978 Oct 25 08:58 libpmi2.la*
lrwxrwxrwx 1 slurm slurm       16 Oct 25 08:58 libpmi2.so -> libpmi2.so.0.0.0*
lrwxrwxrwx 1 slurm slurm       16 Oct 25 08:58 libpmi2.so.0 -> libpmi2.so.0.0.0*
-rwxr-xr-x 1 slurm slurm   203080 Oct 25 08:58 libpmi2.so.0.0.0*
-rw-r--r-- 1 slurm slurm 66385348 Oct 25 08:56 libslurm.a
-rwxr-xr-x 1 slurm slurm      989 Oct 25 08:56 libslurm.la*
lrwxrwxrwx 1 slurm slurm       18 Oct 25 08:56 libslurm.so -> libslurm.so.39.0.0*
lrwxrwxrwx 1 slurm slurm       18 Oct 25 08:56 libslurm.so.39 -> libslurm.so.39.0.0*
-rwxr-xr-x 1 slurm slurm  8239928 Oct 25 08:56 libslurm.so.39.0.0*
drwxr-xr-x 3 slurm slurm    12288 Oct 25 08:57 slurm/
(base) [~]$

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@dalcinl @isuruf @h-vetinari @christopherwharrop-noaa