Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generate conda binaries and publish them in a robotology conda channel #620

Closed
traversaro opened this issue Feb 9, 2021 · 0 comments · Fixed by #652
Closed

Generate conda binaries and publish them in a robotology conda channel #620

traversaro opened this issue Feb 9, 2021 · 0 comments · Fixed by #652

Comments

@traversaro
Copy link
Member

As #477 is now fixed and the documentation on how to build the robotology-superbuild from source with conda-forge dependencies is available for the interested user at https://github.com/robotology/robotology-superbuild/blob/master/doc/conda-forge.md, I think we can now track in this issue the work to automatically provide conda binaries for the packages in the robotology-superbuild, especially because some PIs extressed interest in this activity @lornat75 @DanielePucci .

Goals

The main objective of this are:

  • Speedup the CI process for projects that depend on a lot of projects that are contained in the robotology-superbuild, without the need to manually maintain CI caches
  • Speedup the time spent in creation of Docker images, also for use in cloud system such as GitPod or GitHub Codespaces
  • Simplify the installation process of robotology software, especially for MATLAB/Simulink or Python users without a lot of experience of CMake/C++ tools, or users on platforms in which a C++ compiler is not easily installable
  • Simplify installation of robotology software in cloud-based systems such as Binder or Google Colab
  • Simplify generation of Windows installers using conda Constructor, to avoid the error-prone current process of compiling every time from source all the dependencies with vcpkg (the CI https://github.com/robotology/robotology-superbuild-dependencies-vcpkg at the moment takes 9/10 hours, so debugging any problem is quite complicated)
  • Simplify quick installation of multiple version of the libraries side-by-side for debugging problems (my actual use case : ) and the reason I work on this )

Strategy

The basic idea is to setup a conda channel that builds on top of conda-forge, and contains the packages installed by the robotology-superbuild, without requiring for package maintainers any additional step beside adding their project to the robotology-superbuild. This idea of a "conda channel" that builds on top of conda-forge has been already successfully exploited in several scientific domains:

Prototype

Most of the logic for generating the conda recipes and build them can be already found in the experimental branch https://github.com/traversaro/robotology-superbuild/tree/master, in particular a description of the recipe generation process (that is not meant for downstream users, but just for CI) can be found in https://github.com/traversaro/robotology-superbuild/blob/master/doc/conda-recipe-generation.md while the GitHub Action that actually compiles the binary can be found in https://github.com/traversaro/robotology-superbuild/blob/conda-recipe-generation/.github/workflows/generate-conda-packages.yaml .

Note some maintainers of packages contained in the robotology-superbuild may be interested to directly submit their packages to conda-forge, for better visibility or to make it easier for their user to install them. For those packages and their dependencies, the recipe generation machinery can be configured to avoid generating the packages that are already in conda-forge, and instead use the one that are in conda-forge. This can be easily done by setting the <pkg>_CONDA_PKG_NAME and <pkg>_CONDA_PKG_CONDA_FORCE_OVERRIDE variables, as done in https://github.com/traversaro/robotology-superbuild/blob/959717e26570f2d0ad42dfcf5dc0f57dcb68aa1c/conda/cmake/CondaGenerationOptions.cmake#L21 for osqp:

# If a package is already available in conda-forge, use it instead of generating a recipe for it
set(osqp_CONDA_PKG_NAME libosqp)
set(osqp_CONDA_PKG_CONDA_FORGE_OVERRIDE ON)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant