Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow micromamba provisioning in Linux too #2097

Merged
merged 5 commits into from
Oct 17, 2024

Conversation

jaimergp
Copy link
Member

@jaimergp jaimergp commented Oct 17, 2024

Checklist

  • Added a news entry
  • Regenerated schema JSON if schema altered (python conda_smithy/schema.py)

Comment on lines 40 to 44
{%- if conda_install_tool == "micromamba" %}
micromamba create --root-prefix /opt/conda --prefix /opt/mamba \
--yes --quiet --override-channels --channel conda-forge --strict-channel-priority \
pip {{ conda_install_tool_deps }} {{ conda_build_tool_deps }} {{ " ".join(remote_ci_setup) }}
{%- endif %}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why can't we just install into /opt/conda?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

micromamba doesn't allow --prefix and --root-prefix to be the same, so this is the quickest way to reuse the /opt/conda cache. We also avoid history related problems.

If you want to reuse /opt/conda, we could:

  1. Move /opt/conda/pkgs to the chosen mamba root prefix (e.g. ~/.conda)
  2. Clear /opt/conda/conda-meta/history
  3. micromamba install ... into /opt/conda

Do you prefer that?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait maybe we don't need (1). Let me check something.

Copy link
Member

@isuruf isuruf Oct 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the issue with using something like

CONDA_PKGS_DIRS=/opt/conda micromamba install conda-forge-ci-setup --prefix /opt/conda --root-prefix /opt/mamba

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, yep, that's what I tested in conda-forge/dav1d-feedstock#21. Let me clean up and push.

@jaimergp
Copy link
Member Author

Ok, the cache can be reused with the current method:

2024-10-17T16:40:19.9135344Z DEBUG:conda.core.package_cache_data:instantiating ProgressiveFetchExtract with
2024-10-17T16:40:19.9136423Z   conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge
2024-10-17T16:40:19.9136914Z   conda-forge/linux-64::ca-certificates-2024.8.30-hbcca054_0
2024-10-17T16:40:19.9137384Z   conda-forge/noarch::kernel-headers_linux-64-3.10.0-he073ed8_17
2024-10-17T16:40:19.9137835Z   conda-forge/linux-64::ld_impl_linux-64-2.43-h712a8e2_1
2024-10-17T16:40:19.9138291Z   conda-forge/noarch::libgcc-devel_linux-64-13.3.0-h84ea5a7_101
2024-10-17T16:40:19.9138737Z   conda-forge/linux-64::python_abi-3.13-5_cp313
2024-10-17T16:40:19.9139144Z   conda-forge/noarch::tzdata-2024b-hc8b5060_0
2024-10-17T16:40:19.9139571Z   conda-forge/linux-64::libgomp-14.2.0-h77fa898_1
2024-10-17T16:40:19.9140050Z   conda-forge/noarch::sysroot_linux-64-2.17-h4a8ded7_17
2024-10-17T16:40:19.9140478Z   conda-forge/linux-64::_openmp_mutex-4.5-2_gnu
2024-10-17T16:40:19.9140932Z   conda-forge/linux-64::binutils_impl_linux-64-2.43-h4bf12b8_1
2024-10-17T16:40:19.9141382Z   conda-forge/linux-64::binutils_linux-64-2.43-h4852527_1
2024-10-17T16:40:19.9141812Z   conda-forge/linux-64::libgcc-14.2.0-h77fa898_1
2024-10-17T16:40:19.9142240Z   conda-forge/linux-64::libexpat-2.6.3-h5888daf_0
2024-10-17T16:40:19.9142672Z   conda-forge/linux-64::libgcc-ng-14.2.0-h69a702a_1
2024-10-17T16:40:19.9143098Z   conda-forge/linux-64::libstdcxx-14.2.0-hc0a3c3a_1
2024-10-17T16:40:19.9143519Z   conda-forge/linux-64::libzlib-1.3.1-hb9d3cd8_2
2024-10-17T16:40:19.9143941Z   conda-forge/linux-64::openssl-3.3.2-hb9d3cd8_0
2024-10-17T16:40:19.9144352Z   conda-forge/linux-64::bzip2-1.0.8-h4bc722e_7
2024-10-17T16:40:19.9144768Z   conda-forge/linux-64::libffi-3.4.2-h7f98852_5
2024-10-17T16:40:19.9145185Z   conda-forge/linux-64::libmpdec-4.0.0-h4bc722e_0
2024-10-17T16:40:19.9145602Z   conda-forge/linux-64::libsanitizer-13.3.0-heb74ff8_1
2024-10-17T16:40:19.9146035Z   conda-forge/linux-64::libsqlite-3.46.1-hadc24fc_0
2024-10-17T16:40:19.9146702Z   conda-forge/linux-64::libstdcxx-ng-14.2.0-h4852527_1
2024-10-17T16:40:19.9147139Z   conda-forge/linux-64::libuuid-2.38.1-h0b41bf4_0
2024-10-17T16:40:19.9147562Z   conda-forge/linux-64::nasm-2.16.03-h4bc722e_1
2024-10-17T16:40:19.9147974Z   conda-forge/linux-64::ncurses-6.5-he02047a_1
2024-10-17T16:40:19.9148405Z   conda-forge/linux-64::pkg-config-0.29.2-h4bc722e_1009
2024-10-17T16:40:19.9149007Z   conda-forge/linux-64::tk-8.6.13-noxft_h4845f30_101
2024-10-17T16:40:19.9149500Z   conda-forge/linux-64::xz-5.2.6-h166bdaf_0
2024-10-17T16:40:19.9149948Z   conda-forge/linux-64::gcc_impl_linux-64-13.3.0-hfea6d02_1
2024-10-17T16:40:19.9150419Z   conda-forge/linux-64::ninja-1.12.1-h297d8ca_0
2024-10-17T16:40:19.9150837Z   conda-forge/linux-64::readline-8.2-h8228510_1
2024-10-17T16:40:19.9151447Z   conda-forge/linux-64::gcc_linux-64-13.3.0-hc28eda2_4
2024-10-17T16:40:19.9151896Z   conda-forge/linux-64::python-3.13.0-h9ebbce0_100_cp313
2024-10-17T16:40:19.9152327Z   conda-forge/noarch::setuptools-75.1.0-pyhd8ed1ab_0
2024-10-17T16:40:19.9152740Z   conda-forge/noarch::meson-1.5.2-pyhd8ed1ab_0
2024-10-17T16:40:19.9152964Z 
2024-10-17T16:40:19.9176555Z INFO:conda.core.link:initializing UnlinkLinkTransaction with
2024-10-17T16:40:19.9177426Z   target_prefix: /home/conda/feedstock_root/build_artifacts/dav1d-split_1729183124081/_build_env
2024-10-17T16:40:19.9177801Z   unlink_precs:
2024-10-17T16:40:19.9178071Z     
2024-10-17T16:40:19.9178334Z   link_precs:
2024-10-17T16:40:19.9178726Z     conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge
2024-10-17T16:40:19.9179397Z     conda-forge/linux-64::ca-certificates-2024.8.30-hbcca054_0
2024-10-17T16:40:19.9179869Z     conda-forge/noarch::kernel-headers_linux-64-3.10.0-he073ed8_17
2024-10-17T16:40:19.9180325Z     conda-forge/linux-64::ld_impl_linux-64-2.43-h712a8e2_1
2024-10-17T16:40:19.9180784Z     conda-forge/noarch::libgcc-devel_linux-64-13.3.0-h84ea5a7_101
2024-10-17T16:40:19.9181208Z     conda-forge/linux-64::python_abi-3.13-5_cp313
2024-10-17T16:40:19.9181638Z     conda-forge/noarch::tzdata-2024b-hc8b5060_0
2024-10-17T16:40:19.9182062Z     conda-forge/linux-64::libgomp-14.2.0-h77fa898_1
2024-10-17T16:40:19.9182495Z     conda-forge/noarch::sysroot_linux-64-2.17-h4a8ded7_17
2024-10-17T16:40:19.9182923Z     conda-forge/linux-64::_openmp_mutex-4.5-2_gnu
2024-10-17T16:40:19.9183365Z     conda-forge/linux-64::binutils_impl_linux-64-2.43-h4bf12b8_1
2024-10-17T16:40:19.9183821Z     conda-forge/linux-64::binutils_linux-64-2.43-h4852527_1
2024-10-17T16:40:19.9184299Z     conda-forge/linux-64::libgcc-14.2.0-h77fa898_1
2024-10-17T16:40:19.9184729Z     conda-forge/linux-64::libexpat-2.6.3-h5888daf_0
2024-10-17T16:40:19.9185164Z     conda-forge/linux-64::libgcc-ng-14.2.0-h69a702a_1
2024-10-17T16:40:19.9185619Z     conda-forge/linux-64::libstdcxx-14.2.0-hc0a3c3a_1
2024-10-17T16:40:19.9186043Z     conda-forge/linux-64::libzlib-1.3.1-hb9d3cd8_2
2024-10-17T16:40:19.9186462Z     conda-forge/linux-64::openssl-3.3.2-hb9d3cd8_0
2024-10-17T16:40:19.9186881Z     conda-forge/linux-64::bzip2-1.0.8-h4bc722e_7
2024-10-17T16:40:19.9187306Z     conda-forge/linux-64::libffi-3.4.2-h7f98852_5
2024-10-17T16:40:19.9187713Z     conda-forge/linux-64::libmpdec-4.0.0-h4bc722e_0
2024-10-17T16:40:19.9188206Z     conda-forge/linux-64::libsanitizer-13.3.0-heb74ff8_1
2024-10-17T16:40:19.9188642Z     conda-forge/linux-64::libsqlite-3.46.1-hadc24fc_0
2024-10-17T16:40:19.9189078Z     conda-forge/linux-64::libstdcxx-ng-14.2.0-h4852527_1
2024-10-17T16:40:19.9189505Z     conda-forge/linux-64::libuuid-2.38.1-h0b41bf4_0
2024-10-17T16:40:19.9189931Z     conda-forge/linux-64::nasm-2.16.03-h4bc722e_1
2024-10-17T16:40:19.9190351Z     conda-forge/linux-64::ncurses-6.5-he02047a_1
2024-10-17T16:40:19.9190781Z     conda-forge/linux-64::pkg-config-0.29.2-h4bc722e_1009
2024-10-17T16:40:19.9191652Z     conda-forge/linux-64::tk-8.6.13-noxft_h4845f30_101
2024-10-17T16:40:19.9192125Z     conda-forge/linux-64::xz-5.2.6-h166bdaf_0
2024-10-17T16:40:19.9192571Z     conda-forge/linux-64::gcc_impl_linux-64-13.3.0-hfea6d02_1
2024-10-17T16:40:19.9193157Z     conda-forge/linux-64::ninja-1.12.1-h297d8ca_0
2024-10-17T16:40:19.9199635Z     conda-forge/linux-64::readline-8.2-h8228510_1
2024-10-17T16:40:19.9200466Z     conda-forge/linux-64::gcc_linux-64-13.3.0-hc28eda2_4
2024-10-17T16:40:19.9201139Z     conda-forge/linux-64::python-3.13.0-h9ebbce0_100_cp313
2024-10-17T16:40:19.9201774Z     conda-forge/noarch::setuptools-75.1.0-pyhd8ed1ab_0
2024-10-17T16:40:19.9202289Z     conda-forge/noarch::meson-1.5.2-pyhd8ed1ab_0
2024-10-17T16:40:19.9202698Z 
2024-10-17T16:40:19.9203050Z DEBUG:conda.core.package_cache_data:prepared package cache actions:
2024-10-17T16:40:19.9203591Z   cache_actions:
2024-10-17T16:40:19.9204450Z     CacheUrlAction<url='https://conda.anaconda.org/conda-forge/linux-64/gcc_impl_linux-64-13.3.0-hfea6d02_1.conda', target_full_path='/home/conda/feedstock_root/build_artifacts/pkg_cache/gcc_impl_linux-64-13.3.0-hfea6d02_1.conda'>
2024-10-17T16:40:19.9205672Z     CacheUrlAction<url='https://conda.anaconda.org/conda-forge/linux-64/python-3.13.0-h9ebbce0_100_cp313.conda', target_full_path='/home/conda/feedstock_root/build_artifacts/pkg_cache/python-3.13.0-h9ebbce0_100_cp313.conda'>
2024-10-17T16:40:19.9206781Z     CacheUrlAction<url='https://conda.anaconda.org/conda-forge/linux-64/binutils_impl_linux-64-2.43-h4bf12b8_1.conda', target_full_path='/home/conda/feedstock_root/build_artifacts/pkg_cache/binutils_impl_linux-64-2.43-h4bf12b8_1.conda'>
2024-10-17T16:40:19.9207958Z     CacheUrlAction<url='https://conda.anaconda.org/conda-forge/linux-64/libsanitizer-13.3.0-heb74ff8_1.conda', target_full_path='/home/conda/feedstock_root/build_artifacts/pkg_cache/libsanitizer-13.3.0-heb74ff8_1.conda'>
2024-10-17T16:40:19.9215617Z     CacheUrlAction<url='https://conda.anaconda.org/conda-forge/linux-64/libstdcxx-14.2.0-hc0a3c3a_1.conda', target_full_path='/home/conda/feedstock_root/build_artifacts/pkg_cache/libstdcxx-14.2.0-hc0a3c3a_1.conda'>
2024-10-17T16:40:19.9217246Z     CacheUrlAction<url='https://conda.anaconda.org/conda-forge/noarch/libgcc-devel_linux-64-13.3.0-h84ea5a7_101.conda', target_full_path='/home/conda/feedstock_root/build_artifacts/pkg_cache/libgcc-devel_linux-64-13.3.0-h84ea5a7_101.conda'>
2024-10-17T16:40:19.9218528Z     CacheUrlAction<url='https://conda.anaconda.org/conda-forge/noarch/kernel-headers_linux-64-3.10.0-he073ed8_17.conda', target_full_path='/home/conda/feedstock_root/build_artifacts/pkg_cache/kernel-headers_linux-64-3.10.0-he073ed8_17.conda'>
2024-10-17T16:40:19.9219572Z     CacheUrlAction<url='https://conda.anaconda.org/conda-forge/linux-64/libmpdec-4.0.0-h4bc722e_0.conda', target_full_path='/home/conda/feedstock_root/build_artifacts/pkg_cache/libmpdec-4.0.0-h4bc722e_0.conda'>
2024-10-17T16:40:19.9220752Z     CacheUrlAction<url='https://conda.anaconda.org/conda-forge/linux-64/libgcc-ng-14.2.0-h69a702a_1.conda', target_full_path='/home/conda/feedstock_root/build_artifacts/pkg_cache/libgcc-ng-14.2.0-h69a702a_1.conda'>
2024-10-17T16:40:19.9221795Z     CacheUrlAction<url='https://conda.anaconda.org/conda-forge/linux-64/libstdcxx-ng-14.2.0-h4852527_1.conda', target_full_path='/home/conda/feedstock_root/build_artifacts/pkg_cache/libstdcxx-ng-14.2.0-h4852527_1.conda'>
2024-10-17T16:40:19.9223009Z     CacheUrlAction<url='https://conda.anaconda.org/conda-forge/linux-64/binutils_linux-64-2.43-h4852527_1.conda', target_full_path='/home/conda/feedstock_root/build_artifacts/pkg_cache/binutils_linux-64-2.43-h4852527_1.conda'>
2024-10-17T16:40:19.9224047Z     CacheUrlAction<url='https://conda.anaconda.org/conda-forge/linux-64/python_abi-3.13-5_cp313.conda', target_full_path='/home/conda/feedstock_root/build_artifacts/pkg_cache/python_abi-3.13-5_cp313.conda'>
2024-10-17T16:40:19.9224671Z   extract_actions:
2024-10-17T16:40:19.9225548Z     ExtractPackageAction<source_full_path='/home/conda/feedstock_root/build_artifacts/pkg_cache/gcc_impl_linux-64-13.3.0-hfea6d02_1.conda', target_full_path='/home/conda/feedstock_root/build_artifacts/pkg_cache/gcc_impl_linux-64-13.3.0-hfea6d02_1'>
2024-10-17T16:40:19.9226789Z     ExtractPackageAction<source_full_path='/home/conda/feedstock_root/build_artifacts/pkg_cache/python-3.13.0-h9ebbce0_100_cp313.conda', target_full_path='/home/conda/feedstock_root/build_artifacts/pkg_cache/python-3.13.0-h9ebbce0_100_cp313'>
2024-10-17T16:40:19.9234589Z     ExtractPackageAction<source_full_path='/home/conda/feedstock_root/build_artifacts/pkg_cache/binutils_impl_linux-64-2.43-h4bf12b8_1.conda', target_full_path='/home/conda/feedstock_root/build_artifacts/pkg_cache/binutils_impl_linux-64-2.43-h4bf12b8_1'>
2024-10-17T16:40:19.9236199Z     ExtractPackageAction<source_full_path='/home/conda/feedstock_root/build_artifacts/pkg_cache/libsanitizer-13.3.0-heb74ff8_1.conda', target_full_path='/home/conda/feedstock_root/build_artifacts/pkg_cache/libsanitizer-13.3.0-heb74ff8_1'>
2024-10-17T16:40:19.9237515Z     ExtractPackageAction<source_full_path='/home/conda/feedstock_root/build_artifacts/pkg_cache/libstdcxx-14.2.0-hc0a3c3a_1.conda', target_full_path='/home/conda/feedstock_root/build_artifacts/pkg_cache/libstdcxx-14.2.0-hc0a3c3a_1'>
2024-10-17T16:40:19.9238680Z     ExtractPackageAction<source_full_path='/home/conda/feedstock_root/build_artifacts/pkg_cache/libgcc-devel_linux-64-13.3.0-h84ea5a7_101.conda', target_full_path='/home/conda/feedstock_root/build_artifacts/pkg_cache/libgcc-devel_linux-64-13.3.0-h84ea5a7_101'>
2024-10-17T16:40:19.9239715Z     ExtractPackageAction<source_full_path='/home/conda/feedstock_root/build_artifacts/pkg_cache/kernel-headers_linux-64-3.10.0-he073ed8_17.conda', target_full_path='/home/conda/feedstock_root/build_artifacts/pkg_cache/kernel-headers_linux-64-3.10.0-he073ed8_17'>
2024-10-17T16:40:19.9240790Z     ExtractPackageAction<source_full_path='/home/conda/feedstock_root/build_artifacts/pkg_cache/libmpdec-4.0.0-h4bc722e_0.conda', target_full_path='/home/conda/feedstock_root/build_artifacts/pkg_cache/libmpdec-4.0.0-h4bc722e_0'>
2024-10-17T16:40:19.9241658Z     ExtractPackageAction<source_full_path='/home/conda/feedstock_root/build_artifacts/pkg_cache/libgcc-ng-14.2.0-h69a702a_1.conda', target_full_path='/home/conda/feedstock_root/build_artifacts/pkg_cache/libgcc-ng-14.2.0-h69a702a_1'>
2024-10-17T16:40:19.9242589Z     ExtractPackageAction<source_full_path='/home/conda/feedstock_root/build_artifacts/pkg_cache/libstdcxx-ng-14.2.0-h4852527_1.conda', target_full_path='/home/conda/feedstock_root/build_artifacts/pkg_cache/libstdcxx-ng-14.2.0-h4852527_1'>
2024-10-17T16:40:19.9243488Z     ExtractPackageAction<source_full_path='/home/conda/feedstock_root/build_artifacts/pkg_cache/binutils_linux-64-2.43-h4852527_1.conda', target_full_path='/home/conda/feedstock_root/build_artifacts/pkg_cache/binutils_linux-64-2.43-h4852527_1'>
2024-10-17T16:40:19.9244360Z     ExtractPackageAction<source_full_path='/home/conda/feedstock_root/build_artifacts/pkg_cache/python_abi-3.13-5_cp313.conda', target_full_path='/home/conda/feedstock_root/build_artifacts/pkg_cache/python_abi-3.13-5_cp313'>

Note how the Cache and Extract actions are not as many as the link_precs. That means some of them are found in the cache already.

That said, I noticed that the --debug logs had a few warnings about not being able to hardlink from the cache into the target location in build_artifacts (because that's mounted, so different filesystem), so there's a bit of IO overhead there. That's nothing new (current released strategy suffers from the same issue) so I think we are fine. We could study in a different PR whether it makes sense to not mount the feedstock but copy internally when we are only doing CI. Locally we would indeed mount so folks can debug post-mortems.

@jaimergp jaimergp marked this pull request as ready for review October 17, 2024 21:17
@jaimergp jaimergp requested a review from a team as a code owner October 17, 2024 21:17
@beckermr beckermr merged commit 1eadcb8 into conda-forge:main Oct 17, 2024
2 checks passed
Comment on lines +38 to +39
mv /opt/conda/conda-meta/history /opt/conda/conda-meta/history.$(date +%Y-%m-%d-%H-%M-%S)
echo > /opt/conda/conda-meta/history
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this needed?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants