dplyr: namespace ‘Rcpp’ 0.12.14 is being loaded, but >= 0.12.15 is required

[nick-youngblut]

Creating the conda env:

$ conda create -n test_env conda-forge::r-ape conda-forge::r-dplyr conda-forge::r-tidyr conda-forge::r-ggplot2
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.5.8
  latest version: 4.5.9

Please update conda by running

    $ conda update -n base conda



## Package Plan ##

  environment location: /ebio/abt3_projects/software/dev/miniconda3_dev/envs/test_env

  added / updated specs:
    - conda-forge::r-ape
    - conda-forge::r-dplyr
    - conda-forge::r-ggplot2
    - conda-forge::r-tidyr


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    r-mgcv-1.8_24              |   r341h470a237_1         2.4 MB  conda-forge
    r-ape-5.1                  |   r341hfc679d8_1         1.5 MB  conda-forge
    r-tidyr-0.8.1              |   r341hfc679d8_1         404 KB  conda-forge
    r-matrix-1.2_14            |   r341h470a237_1         2.4 MB  conda-forge
    ------------------------------------------------------------
                                           Total:         6.7 MB

The following NEW packages will be INSTALLED:

    bzip2:            1.0.6-h470a237_2       conda-forge
    ca-certificates:  2018.4.16-0            conda-forge
    cairo:            1.14.12-he56eebe_3     conda-forge
    curl:             7.61.0-h93b3f91_1      conda-forge
    fontconfig:       2.13.0-h0c61d53_4      conda-forge
    freetype:         2.8.1-hfa320df_1       conda-forge
    gettext:          0.19.8.1-0             conda-forge
    glib:             2.55.0-h464dc38_2      conda-forge
    graphite2:        1.3.11-hfc679d8_0      conda-forge
    gsl:              2.1-2                  conda-forge
    harfbuzz:         1.8.5-h2bb21d5_0       conda-forge
    icu:              58.2-hfc679d8_0        conda-forge
    jpeg:             9c-h470a237_0          conda-forge
    krb5:             1.14.6-0               conda-forge
    libffi:           3.2.1-3                conda-forge
    libgcc:           7.2.0-h69d50b8_2
    libgcc-ng:        7.2.0-hdf63c60_3
    libgfortran:      3.0.0-1
    libiconv:         1.15-h470a237_1        conda-forge
    libpng:           1.6.34-ha92aebf_1      conda-forge
    libssh2:          1.8.0-h5b517e9_2       conda-forge
    libstdcxx-ng:     7.2.0-hdf63c60_3
    libtiff:          4.0.9-he6b73bb_1       conda-forge
    libuuid:          1.0.3-1                conda-forge
    libxcb:           1.13-h470a237_2        conda-forge
    libxml2:          2.9.8-h422b904_2       conda-forge
    ncurses:          5.9-10                 conda-forge
    openssl:          1.0.2o-h470a237_1      conda-forge
    pango:            1.40.14-h9105a7a_2     conda-forge
    pcre:             8.39-0                 conda-forge
    pixman:           0.34.0-2               conda-forge
    pthread-stubs:    0.4-h470a237_1         conda-forge
    r-ape:            5.1-r341hfc679d8_1     conda-forge
    r-assertthat:     0.2.0-r3.4.1_0         conda-forge
    r-base:           3.4.1-4                conda-forge
    r-bh:             1.66.0_1-r3.4.1_0      conda-forge
    r-bindr:          0.1.1-r3.4.1_0         conda-forge
    r-bindrcpp:       0.2.2-r341h9d2a408_0   conda-forge
    r-cli:            1.0.0-r3.4.1_0         conda-forge
    r-colorspace:     1.3_2-r341h470a237_1   conda-forge
    r-crayon:         1.3.4-r3.4.1_0         conda-forge
    r-dichromat:      2.0_0-r3.4.1_0         conda-forge
    r-digest:         0.6.15-r341h470a237_1  conda-forge
    r-dplyr:          0.7.6-r341h9d2a408_0   conda-forge
    r-ggplot2:        3.0.0-r341h6115d3f_0   conda-forge
    r-glue:           1.3.0-r341h470a237_1   conda-forge
    r-gtable:         0.2.0-r3.4.1_0         conda-forge
    r-labeling:       0.3-r3.4.1_0           conda-forge
    r-lattice:        0.20_34-r3.4.1_0       conda-forge
    r-lazyeval:       0.2.1-r341h470a237_1   conda-forge
    r-magrittr:       1.5-r3.4.1_0           conda-forge
    r-mass:           7.3_50-r341hc070d10_1  conda-forge
    r-matrix:         1.2_14-r341h470a237_1  conda-forge
    r-mgcv:           1.8_24-r341h470a237_1  conda-forge
    r-munsell:        0.5.0-r341_0           conda-forge
    r-nlme:           3.1_131-r341ha44fe06_1 conda-forge
    r-pillar:         1.2.2-r341h6115d3f_1   conda-forge
    r-pkgconfig:      2.0.1-r3.4.1_0         conda-forge
    r-plogr:          0.2.0-r341h6115d3f_0   conda-forge
    r-plyr:           1.8.4-r341hfc679d8_1   conda-forge
    r-purrr:          0.2.4-r341h470a237_1   conda-forge
    r-r6:             2.2.2-r3.4.1_0         conda-forge
    r-rcolorbrewer:   1.1_2-r3.4.1_0         conda-forge
    r-rcpp:           0.12.17-r341h9d2a408_1 conda-forge
    r-reshape2:       1.4.3-r341hfc679d8_1   conda-forge
    r-rlang:          0.2.1-r341h470a237_1   conda-forge
    r-scales:         0.5.0-r341hfc679d8_1   conda-forge
    r-stringi:        1.2.3-r341hfc679d8_1   conda-forge
    r-stringr:        1.3.1-r341_0           conda-forge
    r-tibble:         1.4.2-r341h470a237_1   conda-forge
    r-tidyr:          0.8.1-r341hfc679d8_1   conda-forge
    r-tidyselect:     0.2.4-r341hfc679d8_1   conda-forge
    r-utf8:           1.1.3-r341h470a237_1   conda-forge
    r-viridislite:    0.3.0-r3.4.1_0         conda-forge
    r-withr:          2.1.1-r3.4.1_0         conda-forge
    readline:         7.0-0                  conda-forge
    tk:               8.6.8-0                conda-forge
    xorg-kbproto:     1.0.7-h470a237_2       conda-forge
    xorg-libice:      1.0.9-h470a237_4       conda-forge
    xorg-libsm:       1.2.2-h470a237_5       conda-forge
    xorg-libx11:      1.6.5-h470a237_2       conda-forge
    xorg-libxau:      1.0.8-h470a237_6       conda-forge
    xorg-libxdmcp:    1.1.2-h470a237_7       conda-forge
    xorg-libxext:     1.3.3-h470a237_4       conda-forge
    xorg-libxrender:  0.9.10-h470a237_2      conda-forge
    xorg-renderproto: 0.11.1-h470a237_2      conda-forge
    xorg-xextproto:   7.3.0-h470a237_2       conda-forge
    xorg-xproto:      7.0.31-h470a237_7      conda-forge
    xz:               5.2.4-h470a237_0       conda-forge
    zlib:             1.2.11-h470a237_3      conda-forge

Proceed ([y]/n)?


Downloading and Extracting Packages
r-mgcv-1.8_24        |  2.4 MB | ########################################################################################################################################### | 100%
r-ape-5.1            |  1.5 MB | ########################################################################################################################################### | 100%
r-tidyr-0.8.1        |  404 KB | ########################################################################################################################################### | 100%
r-matrix-1.2_14      |  2.4 MB | ########################################################################################################################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate test_env
#
# To deactivate an active environment, use
#
#     $ conda deactivate

Testing:

$ conda activate test_env
$ R

> library(dplyr)

Error generated:

Error: package or namespace load failed for ‘dplyr’ in loadNamespace(j <- i[[1L]], c(lib.loc, .libPaths()), versionCheck = vI[[j]]):
 namespace ‘Rcpp’ 0.12.14 is being loaded, but >= 0.12.15 is required

[nick-youngblut]

It appears that conda is using the wrong Rcpp build:

$ R

R version 3.4.1 (2017-06-30) -- "Single Candle"
Copyright (C) 2017 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

> library(Rcpp)
Warning message:
package ‘Rcpp’ was built under R version 3.4.3

> library(dplyr)
Error: package or namespace load failed for ‘dplyr’ in loadNamespace(j <- i[[1L]], c(lib.loc, .libPaths()), versionCheck = vI[[j]]):
 namespace ‘Rcpp’ 0.12.14 is already loaded, but >= 0.12.15 is required

[bgruening]

Interesting, but you are installing r-rcpp: 0.12.17-r341h9d2a408_1 conda-forge so 0.12.17.
Is there anything else on your system that infers with it?

[nick-youngblut]

Yeah, r-rcpp: 0.12.17-r341h9d2a408_1 should be installed, but it doesn't appear to be used in my R sessions:

$ R

R version 3.4.1 (2017-06-30) -- "Single Candle"
Copyright (C) 2017 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> library(Rcpp)
Warning message:
package ‘Rcpp’ was built under R version 3.4.3
> sessionInfo()
R version 3.4.1 (2017-06-30)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.1 LTS

Matrix products: default
BLAS: /ebio/abt3_projects/software/dev/miniconda3_dev/envs/test_env/lib/R/lib/libRblas.so
LAPACK: /ebio/abt3_projects/software/dev/miniconda3_dev/envs/test_env/lib/R/lib/libRlapack.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
 [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] Rcpp_0.12.14

loaded via a namespace (and not attached):
[1] compiler_3.4.1

I'm definitely calling R installed in the test conda environment.

hmmm... what am I missing?

[nick-youngblut]

My base conda env doesn't have Rcpp installed at all, but Rcpp (v0.12.14) will still load in an R session running from the base conda env

[nick-youngblut]

I checked .Libpaths(), and it appears that Rcpp is being loaded from an R install in my home directory. I changed my libpaths to just the conda install libpath. That fixed the issue, and I can now load dplyr correctly. This has never happened with my conda install previously, so I'm really surprised that it happened here. I should have checked .Libpaths() prior to posting this issue.

I still wonder why Rcpp defaulted to the version outside of my conda env even though Rcpp was installed in the conda env.

[bgruening]

That is an R issue, afaik. R by default looks into your HOME directory. Nothing conda can do here I think. Or we patch R to not do so?

[nick-youngblut]

It's weird though that my older conda install (conda 4.3.29 with R 3.3.2) doesn't include my home directory R install in the libPaths, but my newer version of conda (conda 4.5.9 with R 3.4.1) does. So the old version of conda + R is not looking in my HOME directory. Why should R installed in a conda env look into a HOME directory? I thought the whole point of conda envs was to isolate the software in the envs.

[bgruening]

All I know is that this is an R thing. Python does not have this problem.

[nick-youngblut]

OK. It does appear like a potential big issue if R is looking outside of the conda env. This could reduce the reproducibility of conda envs.

Any ideas on the best way to permanently remove my HOME R install libPath from the conda R libPaths? Permanently adding to the libPaths seems easier than permanently removing a specific libPath.

[nick-youngblut]

I tried downgrading r-base from R 3.4.1 to R 3.3.2. For R 3.3.2, my libPaths just includes the conda env, and not my HOME R install, so it's just the newer versions of r-base that look into the HOME directory for an R install.

[nick-youngblut]

I moved my HOME R install (instead of completely removing it), and all of my conda environments that use R 3.4.1 seem to have been relying on my HOME R install to some extent. So moving my R install seems to have broken all of my conda environments (at least for loading R packages), and this means that my conda envs were not fully isolated and thus they cannot be fully reproduced with a yaml file of the packages in the environment. I'm just glad I didn't completely remove my HOME dir R install...

[jdblischak]

Why should R installed in a conda env look into a HOME directory? I thought the whole point of conda envs was to isolate the software in the envs.

@nick-youngblut Yes, it is super frustrating that conda does not isolate itself by default. That is what everyone expects it to do, so usually the errors are more subtle. For example, one of your collaborators can install your conda environment defined by an environment.yaml file, but still be using their local R packages to perform the analysis.

See conda-forge/r-base-feedstock#37 for an extended discussion. Unfortunately the core conda team is not overly interested in fixing this, so I don't see anything changing anytime soon. It's a shame because this is a serious weakness.

All I know is that this is an R thing. Python does not have this problem.

@bgruening Not true. conda will also use user-installed Python packages over conda packages. It's just that it is less common for Python users to have user-installed packages compared to R users due to differences in pip and install.packages(). Unlike conda, venv does properly ignore user-level Python packages. See conda-forge/python-feedstock#171 for discussion.

[bgruening]

@jdblischak this problem is only partially about packages. The problem is I still wonder why Rcpp defaulted to the version outside of my conda env even though Rcpp was installed in the conda env.. I have never seen this with Python. There are multiple ways how you can disable this behavior in Python if you want to, as linked in the thread by you.

Do you know any fancy way how we can avoid this with R? Would be cool to know as there are many users having problems with this.

[jdblischak]

this problem is only partially about packages. The problem is I still wonder why Rcpp defaulted to the version outside of my conda env even though Rcpp was installed in the conda env.. I have never seen this with Python.

@bgruening That error is completely from a package path issue. The R user library was listed before the conda R library, so the wrong version of Rcpp was loaded. Python behaves the same way. If I install an old version of numpy in my Python user library and then load pandas that is installed in my conda environment, Python throws an error:

$ docker run --rm -it condaforge/linux-anvil
$ conda install -y pandas
$ python -c 'import pandas'
$ pip install --user 'numpy==1.8.2'
$ python -c 'import numpy;print( numpy.__version__)'
1.8.2
$ python -c 'import pandas'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/conda/lib/python3.6/site-packages/pandas/__init__.py", line 23, in <module>
    from pandas.compat.numpy import *
  File "/opt/conda/lib/python3.6/site-packages/pandas/compat/numpy/__init__.py", line 24, in <module>
    'this pandas version'.format(_np_version))
ImportError: this version of pandas is incompatible with numpy < 1.9.0
your numpy version is 1.8.2.
Please upgrade numpy to >= 1.9.0 to use this pandas version

There are multiple ways how you can disable this behavior in Python if you want to, as linked in the thread by you.

Sure, there are also ways to get around it in R. But the point is that this requires manual intervention. And it also requires that users understand that their conda environments are not isolated from their user installation.

Do you know any fancy way how we can avoid this with R? Would be cool to know as there are many users having problems with this.

The solution analogous to Python setting export PYTHONNOUSERSITE=True would be to define R_LIBS_USER="" in ~/.Renviron. But both of these options are quite suboptimal. A user will likely want to be able to load the user packages when running Python or R outside of conda. Having to set and unset environment variables is tedious and error-prone.

But I think that conda should handle this by itself (just like venv). My proposed solution is to add a patch to r-base so that a conda-installed R ignores the default user library on that OS conda-forge/r-base-feedstock#37 (comment). It's not fool-proof, but it will solve the most common problem.

[bgruening]

From a normal Ubuntu 18.04 Docker container:

bag@bag:~$ pip install --user 'numpy==1.8.2'
bag@bag:~$ python -c 'import numpy;print( numpy.__version__)'
1.8.2
bag@bag:~$ conda create -n pandas -y pandas
bag@bag:~$ conda activate pandas
(pandas) bag@bag:~$ python -c 'import numpy;print( numpy.__version__)'
1.14.3
(pandas) bag@bag:~$ python -c 'import pandas'

But I guess I completely misunderstood the problem here. Nevermind, sorry for the noise.

[jdblischak]

From a normal Ubuntu 18.04 Docker container:

@bgruening I wasn't able to reproduce the behavior you observed. Using Ubuntu 18.04 and installing pandas in an environment (via conda activate), I still get an error due to the user-installation of numpy:

docker run --rm -it ubuntu:bionic
# Install conda
apt update
apt install -y --no-install-recommends bzip2 ca-certificates curl
curl -s -L https://repo.continuum.io/miniconda/Miniconda3-4.5.11-Linux-x86_64.sh > miniconda.sh
openssl md5 miniconda.sh | grep e1045ee415162f944b6aebfe560b8fee
bash miniconda.sh -b -p /opt/conda
ln -s /opt/conda/etc/profile.d/conda.sh /etc/profile.d/conda.sh
source /opt/conda/etc/profile.d/conda.sh
conda activate
conda config --set show_channel_urls True
conda config --add channels conda-forge
conda update --all --yes
# Install old version of numpy
# - Had to use Python 3.6. Always got errors with Python 3.7
# - Had to install gcc with APT. Always got error with conda
conda install -y python=3.6
apt install -y gcc
pip install --user 'numpy==1.8.2'
python -c 'import numpy;print( numpy.__version__)'
# 1.8.2
# Create conda environment
conda create -n pandas -y pandas
conda activate pandas
python -c 'import numpy;print( numpy.__version__)'
# 1.8.2
python -c 'import pandas'
# Traceback (most recent call last):
#   File "<string>", line 1, in <module>
#   File "/opt/conda/envs/pandas/lib/python3.6/site-packages/pandas/__init__.py", line 23, in <module>
#     from pandas.compat.numpy import *
#   File "/opt/conda/envs/pandas/lib/python3.6/site-packages/pandas/compat/numpy/__init__.py", line 24, in <module>
#     'this pandas version'.format(_np_version))
# ImportError: this version of pandas is incompatible with numpy < 1.9.0
# your numpy version is 1.8.2.
# Please upgrade numpy to >= 1.9.0 to use this pandas version

But I guess I completely misunderstood the problem here. Nevermind, sorry for the noise.

No need to apologize! I think it is important to discuss and understand this issue with user-installed packages. It is so often over-looked, but it can cause conda users a lot of pain (e.g. this current issue).