Improve conda setup #38728

tobiasdiez · 2024-09-28T04:41:13Z

A few small quality of life improvements to the conda setup.
Notably the environment files under src are now moved to the root, as the old env files there were non-functional and only confused users. Relately, removed the outdated,not working and untested instructions to use conda solely to provide the system packages for sage-the-distro.

A few other improvements along the way:

Make conda in devcontainer working again by forcing mamba v1 (v2 was released a few days ago and breaks a few things related to the lock files)
Force usage of conda-forge everywhere (mixing of channels is no longer supported)

📝 Checklist

The title is concise and informative.
The description explains in detail what this PR is about.
I have linked a relevant issue or discussion.
I have created tests covering the changes.
I have updated the documentation and checked the documentation preview.

⌛ Dependencies

github-actions · 2024-09-28T05:26:12Z

Documentation preview for this PR (built with commit 6a669b4; changes) is ready! 🎉
This preview will update shortly after each push to this PR.

kwankyu · 2024-09-28T23:30:55Z

Is it not possible to put all the environment files into some directory, like .environment? They make the sage root directory look cluttered...

tobiasdiez · 2024-09-29T08:32:54Z

Is it not possible to put all the environment files into some directory, like .environment? They make the sage root directory look cluttered...

Conda-lock has the possibility to create "multi-platform" lock files. That would reduce the number of lock files. I can look into this afterwards.

kwankyu · 2024-09-29T10:56:06Z

Nice. Thanks.

kwankyu · 2024-10-10T04:43:50Z

How am I supposed to use the default devcontainer using conda?

(base) @kwankyu ➜ /workspaces/sage (develop) $ ./sage
Traceback (most recent call last):
  File "/workspaces/sage/src/bin/sage-ipython", line 9, in <module>
    from sage.misc.banner import banner
ModuleNotFoundError: No module named 'sage'

(base) @kwankyu ➜ /workspaces/sage (develop) $ conda activate sage-dev
(sage-dev) @kwankyu ➜ /workspaces/sage (develop) $ sage
bash: sage: command not found
(sage-dev) @kwankyu ➜ /workspaces/sage (develop) $ ./sage
Traceback (most recent call last):
  File "/workspaces/sage/src/bin/sage-ipython", line 9, in <module>
    from sage.misc.banner import banner
ModuleNotFoundError: No module named 'sage'

kwankyu · 2024-10-10T05:29:55Z

Actually it seems that codespaces failed in the setup stage:

tobiasdiez · 2024-10-10T05:52:48Z

Thanks for testing. I can reproduce the issue. It fails with:

Building wheels for collected packages: sagemath-standard
  Created temporary directory: /tmp/pip-wheel-zedyx3_w
  Destination directory: /tmp/pip-wheel-zedyx3_w
  Building editable for sagemath-standard (pyproject.toml): started
  Building editable for sagemath-standard (pyproject.toml): finished with status 'error'
Failed to build sagemath-standard
Exception information:
Traceback (most recent call last):
  File "/opt/conda/envs/sage-dev/lib/python3.11/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
    status = run_func(*args)
             ^^^^^^^^^^^^^^^
  File "/opt/conda/envs/sage-dev/lib/python3.11/site-packages/pip/_internal/cli/req_command.py", line 245, in wrapper
    return func(self, options, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/sage-dev/lib/python3.11/site-packages/pip/_internal/commands/install.py", line 429, in run
    raise InstallationError(
pip._internal.exceptions.InstallationError: Could not build wheels for sagemath-standard, which is required to install pyproject.toml-based projects

Not very informative sadly. Does it print more for you?

For me it works when I call pip install --no-build-isolation -v -v -e ./src manually in the terminal (after activating sage-dev). Could you give this a try?

kwankyu · 2024-10-10T06:00:21Z

As you see in the screenshot, the devcontainer seems to fail somehow before even getting to building sagemath-standard...

tobiasdiez · 2024-10-10T06:03:29Z

try to start the codespace with more space: 2-core • 8GB RAM • 32GB • 14.11 GB works for me

kwankyu · 2024-10-10T06:08:03Z

try to start the codespace with more space: 2-core • 8GB RAM • 32GB • 14.11 GB works for me

2-core • 8GB RAM • 32GB is the default codespaces machine type for me. I don't know what that 14.11 GB is.

Trying again.

kwankyu · 2024-10-10T06:23:37Z

It failed exactly at the same step, as shown in the above screenshot.

tobiasdiez · 2024-10-10T06:36:02Z

Try to use https://docs.github.com/en/codespaces/customizing-your-codespace/changing-the-machine-type-for-your-codespace#changing-the-machine-type with 64GB. The 14GB is the amount of space it uses for me and you can see this at https://github.com/codespaces.

kwankyu · 2024-10-10T06:51:59Z

It seems that there is no machine type with 64GB disk space available for me. I am on free tier.

You are using 14GB. The default machine already has 32GB. Hence the disk space should not be a problem...

tobiasdiez · 2024-10-10T08:14:51Z

I agree, it's strange that you run out of space. In the background of the screenshot you shared, you also see that the codespace actually was created successful - just afterwards it runs out of space.

But maybe we leave this for another PR? The only change I made to the codespace setup was to fix the conda initialization, which is also working for you (sage-dev is correctly initialized in your codespace, just sage failed to build).

kwankyu · 2024-10-10T08:47:35Z

I agree, it's strange that you run out of space. In the background of the screenshot you shared, you also see that the codespace actually was created successful - just afterwards it runs out of space.

It seems really a space problem. There indeed appeared log messages (strangely above the "success" message) like:

...
  [Errno 28] No space left on device: '/workspaces/sage/src/sage/rings/polynomial/polynomial_rational_flint.cpp'
...

But maybe we leave this for another PR? The only change I made to the codespace setup was to fix the conda initialization, which is also working for you (sage-dev is correctly initialized in your codespace, just sage failed to build).

Are you using a bigger machine, say with 64 GB disk space?

It is really problematic if the default 32 GB machine could not cope with the conda install, since this is the default devcontainer.json...

dimpase · 2024-10-10T11:52:53Z

by the way, why is there no environment-3.12-linux.yml ?

dimpase

I've checked on a local Linux conda install with Python 3.11, running

conda env create --file environment-3.11-linux.yml --name sage-build
mamba activate sage-build
pip install --no-build-isolation -v -v -e ./src

the resulting ./sage seems to work.

Before building I rebased over the latest beta, 10.5.beta6

kwankyu · 2024-10-10T16:18:06Z

src/doc/en/installation/conda.rst

-                  --prefix=$CONDA_PREFIX
-    $ make
-
-
 .. _sec-installation-conda-develop:


I am not sure about removing this section. It seems that installing sage from source within active conda environment is a supported mode. I think the files build/pkgs/*/distros/conda.txt are there to support this mode of installing sage from source.

That this is broken now is not a sufficient reason to drop it. If we are officially to drop it, we should do it properly (including a discussion in sage-devel and removing all files build/pkgs/*/distros/conda.txt).

So I suggest to update this section instead of removing it, for now.

@saraedum - as a Sage conda maintainer, any opinion?

I am not sure about removing this section. It seems that installing sage from source within active conda environment is a supported mode.

The supported mode is to install all dependencies with conda; what is not supported (in the sense that no one checks that it works or provides help if it doesn't) is to build the python dependencies using sage-the-distro, but install all non-python dependencies using conda. I also don't see any point in supporting such a mixed install.

I think the files build/pkgs/*/distros/conda.txt are there to support this mode of installing sage from source.

No, these files are only there to generate the conda environment files.

... I also don't see any point in supporting such a mixed install.

I don't know... The mixed install has been in sage for years, even in broken and unsupported state. Its complete removal may need to be discussed in sage-devel if experts here have different opinions. I am not an expert.

Let's wait for @saraedum's opinion.

I think the files build/pkgs/*/distros/conda.txt are there to support this mode of installing sage from source.

No, these files are only there to generate the conda environment files.

OK. So we should keep those files anyway.

On the other hand, those files are also used for the mixed install. No?

Building Sage in this way is the only way I've been able to successfully build Sage in a way that gives me a Jupyter kernel at the end of it. I think it's important that people are able to build Sage using conda in a way that gives them all Sage features (including a working Jupyter kernel). I would be against removing this unless we have another documented way of building Sage using conda such that you get a Jupyter kernel at the end of it.

Following the instructions "Using conda to provide all dependencies for the Sage library" doesn't give me a Jupyter kernel (or at least not one that is found by following these instructions). Building from source without conda was giving me a bunch of dependency issues and I never got it to work (I'm on Fedora), which is the entire reason I use conda to build Sage.

These instructions are not needed. The sagemath jupyter kernel is automatically installed in the same conda environment once you pip installed sagemath:

$ jupyter kernelspec list Available kernels: python3 /opt/conda/envs/sage-dev-basic/share/jupyter/kernels/python3 sagemath /opt/conda/envs/sage-dev-basic/share/jupyter/kernels/sagemath

If you would want to install the kernel into a different env, we would need to add an install command accepting a prefix similar to https://ipython.readthedocs.io/en/latest/install/kernel_install.html#kernels-for-different-environments, or you use the path from jupyter kernelspec list to install it manually in a different venv.

Whether it's conda or not, one normally wants the kernel installed into the same (v)env as used for the build.
Such an installation is cheap, anyway, and can always be done, whenever jupyter is present.

So a general feature to make it work ought to work in conda, too.

It's actually a general feature: sage always installs the kernel specs for the same venv, see https://github.com/sagemath/sage/blob/develop/src/sage_setup/command/sage_install.py

These instructions are not needed. The sagemath jupyter kernel is automatically installed in the same conda environment once you pip installed sagemath:

$ jupyter kernelspec list Available kernels: python3 /opt/conda/envs/sage-dev-basic/share/jupyter/kernels/python3 sagemath /opt/conda/envs/sage-dev-basic/share/jupyter/kernels/sagemath

If you would want to install the kernel into a different env, we would need to add an install command accepting a prefix similar to https://ipython.readthedocs.io/en/latest/install/kernel_install.html#kernels-for-different-environments, or you use the path from jupyter kernelspec list to install it manually in a different venv.

I think manually using the path from jupyter kernelspec list is acceptable. But in order to get the kernel like you described here I needed to remove the --editable flag from the pip install command. I think that should be documented.

I could reproduce the problem with editable install. I've now added the information about the juptyer kernel in the docs.

Thanks for bringing these issues up!

I could reproduce the problem with editable install. I've now added the information about the juptyer kernel in the docs.

Thanks for bringing these issues up!

The explanation looks good, thanks for addressing the feedback!

kwankyu · 2024-10-10T16:22:31Z

I will leave review to Dima.

tobiasdiez · 2024-10-11T02:18:02Z

It is really problematic if the default 32 GB machine could not cope with the conda install, since this is the default devcontainer.json...

I agree. But the conda dependencies only take 4-5GB or so. Not sure how much the OS in the codespace takes, but shouldn't be more than 5GB. So the problematic part is that in order to build sage one needs >= 10-15GB. This is also about the same size where we hit issues with the build github actions a year ago...so not really a new problem.

by the way, why is there no environment-3.12-linux.yml ?

Will be added in #36431.

I've checked on a local Linux conda install with Python 3.11, running ... the resulting ./sage seems to work.

Thanks for testing!

kwankyu · 2024-10-11T03:24:29Z

...This is also about the same size where we hit issues with the build github actions a year ago...so not really a new problem.

Perhaps not new.

If the conda install does not work with the default 32 GB machine, we should not set it as the default devcontainer for developers who want to use codespaces for sage development. The PR #38789 adds a devcontainer that is ready to build sage from source. We may need to consider to change the default devcontainer to it...

tobiasdiez · 2024-10-22T10:18:37Z

I'm setting this now to positive review based on the positive review and feedback by @dimpase and @vincentmacri (thanks!), since this is urgently needed to fix the ci-issues that will be introduced by #38804.

We can always readd the outdated instructions that I've removed from the docs in a later PR.

dimpase · 2024-10-22T11:51:43Z

fine with me to give it a positive review

Improve conda setup

de60b5a

github-actions bot added the s: needs review label Sep 28, 2024

tobiasdiez requested a review from dimpase September 28, 2024 04:46

Merge branch 'develop' into conda-improv

d53776a

kwankyu mentioned this pull request Oct 8, 2024

Fix codespaces devcontainer #38787

Closed

5 tasks

dimpase approved these changes Oct 10, 2024

View reviewed changes

kwankyu reviewed Oct 10, 2024

View reviewed changes

tobiasdiez mentioned this pull request Oct 19, 2024

GAP 4.13.1 #38804

Open

tobiasdiez added 2 commits October 20, 2024 14:33

Clarify how to use Jupyter with conda

109cb91

Merge branch 'develop' into conda-improv

6a669b4

tobiasdiez added s: positive review and removed s: needs review labels Oct 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve conda setup #38728

Improve conda setup #38728

tobiasdiez commented Sep 28, 2024

github-actions bot commented Sep 28, 2024 •

edited

Loading

kwankyu commented Sep 28, 2024 •

edited

Loading

tobiasdiez commented Sep 29, 2024

kwankyu commented Sep 29, 2024

kwankyu commented Oct 10, 2024

kwankyu commented Oct 10, 2024

tobiasdiez commented Oct 10, 2024

kwankyu commented Oct 10, 2024

tobiasdiez commented Oct 10, 2024

kwankyu commented Oct 10, 2024

kwankyu commented Oct 10, 2024

tobiasdiez commented Oct 10, 2024

kwankyu commented Oct 10, 2024

tobiasdiez commented Oct 10, 2024

kwankyu commented Oct 10, 2024 •

edited

Loading

dimpase commented Oct 10, 2024

dimpase left a comment •

edited

Loading

kwankyu Oct 10, 2024

dimpase Oct 10, 2024

tobiasdiez Oct 11, 2024 •

edited

Loading

kwankyu Oct 11, 2024

vincentmacri Oct 18, 2024

dimpase Oct 19, 2024

tobiasdiez Oct 19, 2024

vincentmacri Oct 19, 2024

tobiasdiez Oct 20, 2024

vincentmacri Oct 21, 2024

kwankyu commented Oct 10, 2024

tobiasdiez commented Oct 11, 2024

kwankyu commented Oct 11, 2024

tobiasdiez commented Oct 22, 2024

dimpase commented Oct 22, 2024

Improve conda setup #38728

Are you sure you want to change the base?

Improve conda setup #38728

Conversation

tobiasdiez commented Sep 28, 2024

📝 Checklist

⌛ Dependencies

github-actions bot commented Sep 28, 2024 • edited Loading

kwankyu commented Sep 28, 2024 • edited Loading

tobiasdiez commented Sep 29, 2024

kwankyu commented Sep 29, 2024

kwankyu commented Oct 10, 2024

kwankyu commented Oct 10, 2024

tobiasdiez commented Oct 10, 2024

kwankyu commented Oct 10, 2024

tobiasdiez commented Oct 10, 2024

kwankyu commented Oct 10, 2024

kwankyu commented Oct 10, 2024

tobiasdiez commented Oct 10, 2024

kwankyu commented Oct 10, 2024

tobiasdiez commented Oct 10, 2024

kwankyu commented Oct 10, 2024 • edited Loading

dimpase commented Oct 10, 2024

dimpase left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tobiasdiez Oct 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kwankyu commented Oct 10, 2024

tobiasdiez commented Oct 11, 2024

kwankyu commented Oct 11, 2024

tobiasdiez commented Oct 22, 2024

dimpase commented Oct 22, 2024

github-actions bot commented Sep 28, 2024 •

edited

Loading

kwankyu commented Sep 28, 2024 •

edited

Loading

kwankyu commented Oct 10, 2024 •

edited

Loading

dimpase left a comment •

edited

Loading

tobiasdiez Oct 11, 2024 •

edited

Loading