Mamba resolves to older packages than Conda

[johnhany97]

Hey,

I am experimenting with mamba to use in place of conda and noticed that sometimes mamba can pick different packages than conda. Namely, I noticed that sometimes mamba's resolved environment can include some packages at a whole major version lower than what conda would pick. Take for example the following:

If you try and run:

mamba create --json --dry-run -y --override-channels -p /tmp/mamba-test -c conda-forge python\=\=3.6.7 scipy scikit-learn\>\=0.23.0

we end up resolving to

  conda-forge/noarch::joblib-1.0.1-pyhd8ed1ab_0
  conda-forge/noarch::pip-21.0.1-pyhd8ed1ab_0
  conda-forge/noarch::threadpoolctl-2.1.0-pyh5ca1d4c_0
  conda-forge/noarch::wheel-0.36.2-pyhd3deb0d_0
  conda-forge/osx-64::ca-certificates-2020.12.5-h033912b_0
  conda-forge/osx-64::certifi-2020.12.5-py36h79c6626_1
  conda-forge/osx-64::libblas-3.9.0-8_openblas
  conda-forge/osx-64::libcblas-3.9.0-8_openblas
  conda-forge/osx-64::libcxx-11.1.0-habf9029_0
  conda-forge/osx-64::libffi-3.2.1-hb1e8313_1007
  conda-forge/osx-64::libgfortran-4.0.0-7_5_0_h1a10cd1_22
  conda-forge/osx-64::libgfortran4-7.5.0-h1a10cd1_22
  conda-forge/osx-64::liblapack-3.9.0-8_openblas
  conda-forge/osx-64::libopenblas-0.3.12-openmp_h63d9170_1
  conda-forge/osx-64::llvm-openmp-11.1.0-hda6cdc1_1
  conda-forge/osx-64::ncurses-6.2-h2e338ed_4
  conda-forge/osx-64::numpy-1.19.5-py36h08dc641_1
  conda-forge/osx-64::openssl-1.1.1k-h0d85af4_0
  conda-forge/osx-64::python-3.6.7-h4285619_1008_cpython
  conda-forge/osx-64::python_abi-3.6-1_cp36m
  conda-forge/osx-64::readline-8.1-h05e3726_0
  conda-forge/osx-64::scikit-learn-0.24.1-py36hed11f80_0
  conda-forge/osx-64::scipy-1.5.3-py36hca8f76d_0
  conda-forge/osx-64::setuptools-49.6.0-py36h79c6626_3
  conda-forge/osx-64::sqlite-3.35.4-h44b9ce1_0
  conda-forge/osx-64::tk-8.6.10-hb0a8c7a_1
  conda-forge/osx-64::xz-5.2.5-haf1e3a3_1
  conda-forge/osx-64::zlib-1.2.11-h7795811_1010

but if you run the same command with conda instead, we resolve to:

  conda-forge/noarch::joblib-1.0.1-pyhd8ed1ab_0
  conda-forge/noarch::pip-21.0.1-pyhd8ed1ab_0
  conda-forge/noarch::threadpoolctl-2.1.0-pyh5ca1d4c_0
  conda-forge/noarch::wheel-0.36.2-pyhd3deb0d_0
  conda-forge/osx-64::ca-certificates-2020.12.5-h033912b_0
  conda-forge/osx-64::certifi-2020.12.5-py36h79c6626_1
  conda-forge/osx-64::libblas-3.9.0-8_openblas
  conda-forge/osx-64::libcblas-3.9.0-8_openblas
  conda-forge/osx-64::libcxx-11.1.0-habf9029_0
  conda-forge/osx-64::libffi-3.2.1-hb1e8313_1007
  conda-forge/osx-64::libgfortran-5.0.0-9_3_0_h6c81a4c_22
  conda-forge/osx-64::libgfortran5-9.3.0-h6c81a4c_22
  conda-forge/osx-64::liblapack-3.9.0-8_openblas
  conda-forge/osx-64::libopenblas-0.3.12-openmp_h54245bb_1
  conda-forge/osx-64::llvm-openmp-11.1.0-hda6cdc1_1
  conda-forge/osx-64::ncurses-6.2-h2e338ed_4
  conda-forge/osx-64::numpy-1.19.5-py36h08dc641_1
  conda-forge/osx-64::openssl-1.1.1k-h0d85af4_0
  conda-forge/osx-64::python-3.6.7-h4285619_1008_cpython
  conda-forge/osx-64::python_abi-3.6-1_cp36m
  conda-forge/osx-64::readline-8.1-h05e3726_0
  conda-forge/osx-64::scikit-learn-0.24.1-py36hed11f80_0
  conda-forge/osx-64::scipy-1.5.3-py36h04de62b_0
  conda-forge/osx-64::setuptools-49.6.0-py36h79c6626_3
  conda-forge/osx-64::sqlite-3.35.4-h44b9ce1_0
  conda-forge/osx-64::tk-8.6.10-h0419947_1
  conda-forge/osx-64::xz-5.2.5-haf1e3a3_1
  conda-forge/osx-64::zlib-1.2.11-h7795811_1010

Notably, the differing packages here can be seen in this screenshot: (Conda on the left / Mamba on the right)

It's surprising that mamba resolved to libgfortran 4.X where conda picked 5.X.

Hypothesis

I did some digging and noticed that libgfortran is brought in as a transitive dependency from scipy. Conda and mamba resolved to the same scipy 1.5.3 but different buildstrings.

When I looked at them more closely, I noticed that conda picked:

{
  "arch": null,
  "build": "py36h04de62b_0",
  "build_number": 0,
  "channel": "https://conda.anaconda.org/conda-forge/osx-64",
  "constrains": [],
  "depends": [
    "libblas >=3.8.0,<4.0a0",
    "libcblas >=3.8.0,<4.0a0",
    "libcxx >=10.0.1",
    "libgfortran 5.*",
    "libgfortran5 >=9.3.0",
    "liblapack >=3.8.0,<4.0a0",
    "numpy >=1.16.5,<2.0a0",
    "python >=3.6,<3.7.0a0",
    "python_abi 3.6.* *_cp36m"
  ],
  "fn": "scipy-1.5.3-py36h04de62b_0.tar.bz2",
  "license": "BSD-3-Clause",
  "license_family": "BSD",
  "md5": "e247c8d7ae7e222548c01ffe5896aeaa",
  "name": "scipy",
  "platform": null,
  "sha256": "40c9887704fd2d0a220192f35f717f6e85d65a51f99945e54e17170092f52c5a",
  "size": 17895320,
  "subdir": "osx-64",
  "timestamp": 1604306033890,
  "url": "https://conda.anaconda.org/conda-forge/osx-64/scipy-1.5.3-py36h04de62b_0.tar.bz2",
  "version": "1.5.3"
}

while mamba picked

{
  "build": "py36hca8f76d_0",
  "build_number": 0,
  "channel": "https://conda.anaconda.org/conda-forge/osx-64",
  "constrains": [],
  "depends": [
    "python >=3.6,<3.7.0a0",
    "python_abi 3.6.* *_cp36m",
    "libcxx >=10.0.1",
    "libblas >=3.8.0,<4.0a0",
    "numpy >=1.16.5,<2.0a0",
    "libcblas >=3.8.0,<4.0a0",
    "libgfortran4 >=7.5.0",
    "libgfortran 4.*",
    "liblapack >=3.8.0,<4.0a0"
  ],
  "fn": "scipy-1.5.3-py36hca8f76d_0.tar.bz2",
  "license": "BSD-3-Clause",
  "md5": "3826befb30dd642754d33ac2c5083980",
  "name": "scipy",
  "sha256": "8635d15de2864c68c93317ccd44e3e2412b7efa1c576a72474b2be4578aa6288",
  "size": 20036553,
  "subdir": "osx-64",
  "timestamp": 1604305961000,
  "url": "https://conda.anaconda.org/conda-forge/osx-64/scipy-1.5.3-py36hca8f76d_0.tar.bz2",
  "version": "1.5.3"
}

I then converted the timestamp field and noticed that conda picked the newer package (Monday, 2 November 2020 08:33:53.890) out of the 2 (Mamba picked Monday, 2 November 2020 08:32:41) when presented with 2 packages with the same name and version, but different build strings.

I think conda has a concept called Timestamp maximization which is what kicked in here and led to the differing results.

[wolfv]

Hi @johnhany97 this could be fixed in libsolv, by modifying the sort order. The code for this is in this general area: https://github.com/openSUSE/libsolv/blob/e2ed3f290d588642b454d19e8c993d76b7a58f15/src/policy.c#L912

[beckermr]

Hey @wolfv. Some colleagues and I also hit this bug recently. Are there any good workarounds?

[wolfv]

@beckermr this appears to be fixed with this PR in libsolv: openSUSE/libsolv#449

I am considering to publish the two recent improvements to libsolv's conda support in patches to our libsolv feedstock while we wait for feedback.

[beckermr]

Sounds good!

[beckermr]

Should we be shipping builds with the conda macros enabled for libsolv separately from the upstream lib? This might be a better practice for users of libsolv outside of mamba/micromamba, though maybe no such users exist.

[wolfv]

I think if someone wants that config as a conda-package, they are free to open a PR on the feedstock :)

[wolfv]

Can you test this again with the latest libsolv on conda-forge? I pushed a build with this patch: openSUSE/libsolv#457

[astrojuanlu]

If I understand correctly, changes to libsolv haven't been merged and released yet? (Just to better understand the status of this issue)

[beckermr]

The conda-forge package has been updated ahead of upstream with the patch. So if you update your base env, you will get the changes.