Should we endorse Pixi?

[maresb]

While I have not yet had the chance to transition to Pixi in any of my projects, I'm convinced that this is the future of lockfiles in the Conda ecosystem. Moreover, they have a full-time team moving things forwards.

I think that conda-lock will still play a role with supporting a large number of existing projects since the effort of retooling can be hard to justify. But I'd hope that new projects would look primarily at Pixi.

Dually, I'd be very curious to hear from people who have tried both, which things does conda-lock do that are are not yet adequately supported by Pixi?

[synapticarbors]

I've just started experimenting with pixi and there are definitely a lot of things to like. The biggest issue that I've seen so far is just the difference in workflow/philosophy from conda. Pixi is directory-based, whereas conda/mamba has a global registry of environments. With conda I use the same environment for several projects, and will also do stuff like work in a scratch directory to fire up some jupyter-lab instance to do some quick analysis, or just open up a terminal, run mamba activate <env> and then work from there.

Pixi requires you to cd into a known location and run pixi shell or pixi run, or make use of the output of pixi shell-hook, which again has to live somewhere. So this isn't necessarily about features related to conda-lock, but is more a question about whether pixi itself is a full drop-in replacement for all conda workflows, and about adoption. There's been some discussion of pixi supporting a global registry of environments (prefix-dev/pixi#188), but this difference in workflows is an opinionated decision on the part of the pixi developers and isn't likely to change at least in the short term.

I think one thing to ask is if there is a world where you could use pixi to generate lock files and those lock files would be usable directly from conda/mamba to build environments. Then pixi could be used in place of conda-lock, without the requirement of adopting the full pixi workflow.

[maresb]

Thanks @synapticarbors for the feedback, it's really valuable, especially since I haven't found the time to investigate these details myself!

Regarding your question, someone asked something very similar here and received and answer.

In order to get the x-link, the answer directs here: prefix-dev/pixi#800

[maresb]

I'm trying to develop some ideas for how to proceed with this, while being honest about my lack of time for maintenance.

README and migration guide

I propose adding a notice to the README, something like:

The conda-lock maintainers recommend using pixi for new projects. Pixi is evolving at a much faster pace and provides better support, so we also recommend considering migrating existing projects when feasible.

We might also consider some sort of migration guide separate from the README and make roughly the following points:

• pixi is based on projects rather than environments
• pixi is an all-in-one project management tool analogous to Poetry
• Groups of dependency specifications are called "features". Features are combined into "environments".
• Pixi "environments" are stored within the project directory (like PDM) and are normal Conda environments.
• In Python projects, even when using pyproject.toml in place of pixi.toml, the pixi dependencies are independent of the project.dependencies from the pyproject.toml and thus currently require repeating the dependencies from project.dependencies in the pixi.toml's dependencies or pypi-dependencies sections.
• (And maybe some others?)

Missing features

Then there's the question of what functionality is missing in the gap between pixi and conda-lock?

Some things that come to mind:

• There's no way to conda-lock install a pixi.lock to a global environment outside the project directory, though there are many proposals floating around for such functionality. discussion.
• Conda-lock currently parses sources from pyproject.toml, meta.yaml, and environment.yaml files, while pixi doesn't. discussion
• (Any other functionalities that people are missing?)

What should we do?

It's plausible that this one or both of these features arrive soon in pixi. On the other hand, given what we currently have in conda-lock, I wonder if we could quickly provide solutions ourselves.

• Given all the code we have already written, maybe it would be easy for us to parse pixi.lock files and render them to the explicit format that can be installed using your Conda command of choice. I think this would make most sense as a separate project cannibalized from conda-lock code. This idea originates from @bollwyvl. (As for accessing the lockfiles, after a brief attempt at searching for formal schema I asked.)

• As for the dependency source files, I was thinking it would be helpful to have a pre-commit style dependency updater. This could be a Python CLI that takes in a bunch of -f dependency-source arguments like conda-lock currently does, and then modifies a pixi.toml to ensure the dependencies are correctly reflected in the pixi.toml. We already have the function make_lock_spec for turning files into dependency specifications, and the specifications are even in the form of a LockSpecification Pydantic model.

I'm very curious what people think, and if anyone's willing to tackle any of this?

[synapticarbors]

Just as a note, I recently wrote a prototype tool to render a pixi.lock to an explicit conda environment spec file that can be installed by conda/mamba:
https://github.com/synapticarbors/pixi2ces

I've been talking to some of the pixi devs about developing it into the start of the pixi export command. I just submitted a PR to add a to_spec_string to the ExplicitEnvironmentSpec struct in rattler, which will then be used downstream in pixi.

[bollwyvl]

endorse

I think in general, having a call-out for Alternatives would be sufficient, probably with ❌/✅ feature table. For example, conda-lock will bootstrap on most platforms given only pip but the same cannot be said for a number of other tools (including pixi).

As to integrating directly with pixi... as a solver, as long as it doesn't provide @EXPLICIT or the same output as --dry-run --json, it's not going to be very useful. If it had those, being able to say conda-lock --pixi would be grand, but would be an entirely separate code path, and would almost certainly entail building more tmp folders, and need a bunch more switches.

I don't know if re-implementing the logic for its feature system makes any sense to maintain longterm.

pixi2ces

This looks great as inspiration, and pixi/rattler providing this as a first-party, rather than adding another link in the toolchain, will be welcome! What we don't need today (or in the near future) are more file formats in more versions parseable only/partially by some tools at some versions. @EXPLICIT is terrible (md5 gah), but works everywhere.

Otherwise: pixi is trying to do an awful lot. A pixi project can make for good process but doesn't yet offer a compelling product for the knowledge trapped in the manifest file. This is what truly kills me about heavy process tools such as mvn, etc. as it focuses on the build as the product, which creates true nightmares downstream. I don't know if a pixi-constructor will have some plan for that, e.g. actually embedding a pixi runtime in the installer, but it would probably feel pretty bad as an end user that didn't sign up for the complexity and just wants, likely, one, well-documented CLI command/desktop shortcut to work really well.

I have found the wheel-related stuff to be underwhelming (and the magic it does vs pyproject.toml), so end up using pixi.toml and pip install (usually -e, but sometimes worse things) inside an environment. But then I never used the pip features in conda-lock, or its YAML lockfile format, either. pixi has gotten me onboard to checking in their YAML (or will it be TOML?), but I treat it as entirely opaque, and it would be best to rely on the documented CLI behavior. Sadly the CLI outputs does not yet have a JSON schema, which increases integration risk.

cannibalized

Before adding anything here, or copy pasting any code into new packages, perhaps these morsels could be fully consumed. conda-lock is still a very useful tool (not a mindset), and a lot of product pipelines depend on it. However, I doubt any of them use all of the optional features, but they are always paying for them. I'd see splitting most of them into much more focused, standalone packages (e.g. conda-lock-git, and not even bothering with [extras]), and tying them together with entry_points, leaving conda-lock-core as a python-only library with absolutely minimal dependencies.

[maresb]

Thanks so much @bollwyvl for your input!!! I'm struggling a bit to unpack everything. I'd really appreciate it if you could help me understand your viewpoint.

bootstrap on most platforms given only pip
...
leaving conda-lock-core as a python-only library

Why do you prefer Python-only code? Is this a security/auditability thing?

being able to say conda-lock --pixi would be grand, but would be an entirely separate code path, and would almost certainly entail building more tmp folders, and need a bunch more switches

In my head I've actually been envisioning conda-lock-pixi as a separate package, or maybe even a rust app, that basically manages the tmp folders, lets pixi do its thing, and transforms the result into something useful. I think pixi would do most of the heavy lifting, so as long as pixi is behaving properly I'd hope the conda-lock-pixi side would be fairly straightforward. Do you see this as problematic?

I'm confused by the "Otherwise: ..." paragraph about conda-constructor functionality. It seems like you wish there were a CLI interface for extracting info, but I don't understand how this relates to constructor.

pixi has gotten me onboard to checking in their YAML (or will it be TOML?), but I treat it as entirely opaque, and it would be best to rely on the documented CLI behavior. Sadly the CLI outputs does not yet have a JSON schema, which increases integration risk.

I've been thinking that py-rattler may be a good way to interface with the underlying data structures from Python.

perhaps these morsels could be fully consumed

Which morsels, and what specifically do you mean by "consumed"?

I'd see splitting most of them into much more focused, standalone packages (e.g. conda-lock-git, and not even bothering with [extras]), and tying them together with entry_points, leaving conda-lock-core as a python-only library with absolutely minimal dependencies.

I really love this idea, and I wish I had the time to do it myself. Here I get what you're saying, so I can explain it for readers who are missing context. This is basically a complaint that conda-lock has some heavyweight dependency requirements that support features only used by some. For example, we have gitpython which some time ago had a security advisory that caused lots of grief. Also, the Poetry dependency constantly causes a lot of grief because Poetry is designed to be a standalone tool that operates in its own environment, but we're entangled with its internals making it a pretty big effort to maintain #637.

[bollwyvl]

prefer Python-only

The doorhandle counts. The recommended install option (curl | bash) is a nasty "modern" smell. Starting from a "dumb" box that only has a system python, pip install conda-lock can bootstrap up to something that works.

Otherwise

conda-lock is a tool that takes in files, runs some subprocess, and puts out files. It doesn't claim to do much more than that (though, as lauded above, it can actually install environments as a byproduct of having a $CONDA_EXE).

pixi (and related) are an ethos. As with other heavy workflow tools, a team's product can end up being pixi run get-value, as opposed to pixi run build-shippable-thing-what-provides-value. For a lot of toolchains, smaller Unix-ish tools that do one thing, but over agreed-upon disk artifacts, will remain attractive.

If pixi grew an installer builder (after growing a .conda package builder), one could imagine it actually shipping white-labelled pixi run get-value aliased as my-cool-thing. Which would be slick. But they would probably not have the properties of a well-understood conda environment.

"consumed"?

Move everything other than bog-standard, no-magic-comments, multiple environment.yml parsing into contrib/ packages. Initially, at least, release them all with == pins on conda_lock. If they later graduate into their own repos and version lines, with claimed and tested compatibility, all's the better.

Within this repo, have a folder structure like...

pyproject.toml
src/
  conda_lock/
contrib/
  conda-lock-cli/
    pyproject.toml
  conda-lock-git/
    pyproject.toml
  conda-lock-poetry/
    pyproject.toml
  conda-lock-pixi/
    pyproject.toml
  conda-lock-foobarbaz/
    pyproject.toml

The top-level conda-lock package can have [extras] that depend on the other ones (I guess), but for the most part, they are much more loosely coupled.

[jayqi]

As someone who just spent some time today experimenting with a migration from conda-lock to Pixi for a project, I just wanted to give some feedback that my biggest motivation was because Pixi does minimal updates to lockfiles. conda-lock not supporting this (#370) is one of the biggest pain points I have in using it.

[maresb]

Yes @jayqi, I agree that the minimal updates are very nice.

Can you provide any feedback regarding the actual migration process? For example, did you encounter any annoyances when transitioning from an environment-based workflow to a project-based workflow? It'd be great to be able to provide a migration guide.

[zmbc]

The recommended install option (curl | bash) is a nasty "modern" smell.

It's worth noting that pixi is just a binary. The script at https://pixi.sh/install.sh downloads it and puts it on the path. That's it. You can easily do that manually from their releases page.

It's probably a matter of opinion but I am totally fine with pixi or similar being the "root" that installs Python+pip instead of the other way around.

Also, the concern about controlling where environments are stored that @synapticarbors raised is, I think, completely resolved by pixi's detached-environments feature.

Personally I would be in favor of a notice like @maresb proposed.

[maresb]

Thanks a lot @zmbc for the input!

I was looking at py-rattler and it seems to me like most if not all the things we'd need to reimplement conda-lock are already there and wouldn't even require a pixi binary.

[bollwyvl]

I'll preface all of my comments with: any new-start project I've started recently is based on pixi. It's not yet suitable for larger projects I work on due to some of the limitations pointed out above.

totally fine with pixi or similar being the "root" that installs Python+pip

Kind of the whole point of using a tool like conda-lock is to be able to capture the full provenance chain of a solution, which, indeed, even ensureconda will break if used, but self-update tools make this almost intractable.

For some more context raised on their repo: prefix-dev/pixi#1765

detached-envs

... still creates an unpredictable environment directory prefix, which then, clearly, can't then be moved.

wouldn't even require a pixi binary

But then would conda-lock grow a hard dependency on py-rattler? Again, it seems like if this isn't made more modular, the situation will just get worse for a self-proclaimed "lightweight" tool:

> mamba create --dry-run -n foo python | wc
     53     177    2885
> mamba create --dry-run -n foo "conda-lock=0.*" | wc
     79     332    5029
> mamba create --dry-run -n foo "conda-lock=1.*" | wc
    119     572    8947
> mamba create --dry-run -n foo conda-lock | wc
    120     584    9057

[maresb]

I just generated a pixi.toml for a large-ish conda-lock-based project. The code I used can be found in #664.

This could be considered a first step towards enabling pixi as a backend.

[wholtz]

I am finally giving pixi a try. Thus far I've hit 2 pain points:

1. I'm working on a M2 Macbook but developing an environment that will be used on linux-amd64. With conda-lock I can lock for any platform, but pixi requires that your current platform is one of the platforms you are locking for. I'm pulling packages from bioconda that are not available for ARM. Therefore, in order to lock on my Macbook with pixi, I need to do the locking within a docker container where I use the --platform switch. Which brings me to my second pain point
2. Private pipy repos are annoying to access with pixi. Because I am locking within a docker container, the pixi's keyring based authentication isn't going to work. Which means I need to use a .netrc file. The ergonomics of this are challenging. I can mount a .netrc file into the container I'm locking it, but my password (token) is not long lived, thus I need to update the .netrc file regularly. I'd much prefer an environmental variable based authentication option.

[maresb]

Thanks a lot @wholtz for the feedback! I really appreciate it.

The Prefix team is extremely responsive, and I'm sure they'd also appreciate your feedback. (You can usually get an immediate response on Discord, if you have that.)

[pavelzw]

1. is not true. if you remove osx-arm64 from your platforms, you can just as well solve the environment from your macbook. you get a warning when running pixi install but that's it

❯ pixi install
 WARN Not installing dependency on current platform: (osx-arm64) as it is not part of this project's supported platforms.
✔ The default environment has been installed.
❯ pixi list --platform linux-aarch64
Package                Version    Build               Size        Kind   Source
_openmp_mutex          4.5        2_gnu               23.2 KiB    conda  _openmp_mutex-4.5-2_gnu.tar.bz2
bzip2                  1.0.8      h68df207_7          185.4 KiB   conda  bzip2-1.0.8-h68df207_7.conda
ca-certificates        2024.8.30  hcefe29a_0          155.4 KiB   conda  ca-certificates-2024.8.30-hcefe29a_0.conda
ld_impl_linux-aarch64  2.43       h80caac9_2          681.9 KiB   conda  ld_impl_linux-aarch64-2.43-h80caac9_2.conda
libffi                 3.4.2      h3557bc0_5          58.1 KiB    conda  libffi-3.4.2-h3557bc0_5.tar.bz2
libgcc                 14.2.0     he277a41_1          522.7 KiB   conda  libgcc-14.2.0-he277a41_1.conda
libgcc-ng              14.2.0     he9431aa_1          52.8 KiB    conda  libgcc-ng-14.2.0-he9431aa_1.conda
libgomp                14.2.0     he277a41_1          452.7 KiB   conda  libgomp-14.2.0-he277a41_1.conda
libnsl                 2.0.1      h31becfc_0          33.7 KiB    conda  libnsl-2.0.1-h31becfc_0.conda
libsqlite              3.47.0     hc4a20ef_0          1017.7 KiB  conda  libsqlite-3.47.0-hc4a20ef_0.conda
libuuid                2.38.1     hb4cce97_0          34.9 KiB    conda  libuuid-2.38.1-hb4cce97_0.conda
libxcrypt              4.4.36     h31becfc_1          111.6 KiB   conda  libxcrypt-4.4.36-h31becfc_1.conda
libzlib                1.3.1      h86ecc28_2          65.1 KiB    conda  libzlib-1.3.1-h86ecc28_2.conda
ncurses                6.5        hcccb83c_1          902.8 KiB   conda  ncurses-6.5-hcccb83c_1.conda
openssl                3.3.2      h86ecc28_0          3.3 MiB     conda  openssl-3.3.2-h86ecc28_0.conda
python                 3.10.15    hbf90c55_2_cpython  12.4 MiB    conda  python-3.10.15-hbf90c55_2_cpython.conda
readline               8.2        h8fc344f_1          287.2 KiB   conda  readline-8.2-h8fc344f_1.conda
tk                     8.6.13     h194ca79_0          3.2 MiB     conda  tk-8.6.13-h194ca79_0.conda
tzdata                 2024b      hc8b5060_0          119.5 KiB   conda  tzdata-2024b-hc8b5060_0.conda
xz                     5.2.6      h9cdd2b7_0          430.2 KiB   conda  xz-5.2.6-h9cdd2b7_0.tar.bz2

[wholtz]

@pavelzw - I believe it is true if you have both conda and pypi dependencies:

$ pipx install
 WARN Not installing dependency on current platform: (osx-arm64) as it is not part of this project's supported platforms.
  × Cannot update pypi dependencies without first installing a conda prefix that includes python.

And the environment does includes python v3.12 via conda-forge.

[tdejager]

With regards to the first point @wholtz you are right! We need to have a python available for any sdists that needs to be built. The best idea to get around this is currently: prefix-dev/pixi#1340 which would be a pretty big feature, but also pretty awesome I think.

With regards to authentication we are kinda tied here to what uv supports, that's why there is a PR open to add custom middleware so then we can add any authentication however we like it: astral-sh/uv#3502. For example, to add an environment variable support we would need to find the way that that best matches the uv authentication methods which are listed here for different registries: https://docs.astral.sh/uv/guides/integration/alternative-indexes/. Myself, I don't really like an EXTRA_INDEX_URL as it makes the manifest file not reproducible.

We could think of allowing environment variables in the toml files something along the lines of ${env:MY_KEY} an allow then you can use the HTTP basic authentication scheme with uv, if the registry allows it.

In the mean time, wouldn't installing keyring with the backend that you need be an option inside the container?