Conda lock files #5827

edmundmiller · 2024-06-18T19:47:55Z

Follow up to #4080. This will add conda lock files whenever the environment.yml gets updated, and then pass that to wave.

edmundmiller · 2024-08-08T16:14:18Z

Status update:

This PR doesn't really do anything for us until we adopt wave.

Main issue was the GitHub action can't commit the lock file back for whatever permissions issue.

pinin4fjords · 2024-08-09T09:00:46Z

This is pretty frickin' awesome, lock files have been on my xmas list for ages.

My only questions surround how these lock files will actually get used. As-is. the conda and container directives are unchanged for this module, so behaviour in use will be unchanged.

We spoke on Slack about how I thought (and others disagreed) that we should also change the container directive to use what Wave spits back. I also wonder how we could make it so that people deploying via Conda could get a static environment built off the lock file?

https://github.com/seqeralabs/wave-cli?tab=readme-ov-file#build-a-container-by-using-a-conda-lock-file

Couldn't get it to commit

https://github.com/nf-core/modules/blob/90b1d1bd7f9474e173abd9e4082dedb559c7542a/.github/workflows/fix-linting.yml#L62-L71 Co-authored-by: mashehu <mashehu@users.noreply.github.com>

modules/nf-core/bowtie2/align/main.nf

pinin4fjords · 2024-08-12T11:06:49Z

@adamrtalbot raises the excellent point that lock files will be architecture-specific. So maybe what we need is actually a conda-lock-amd64.txt, conda-lock-arm64.txt? Then workflows would need to be run with architecture-specific configurations referencing these?

adamrtalbot · 2024-08-12T11:08:14Z

@adamrtalbot raises the excellent point that lock files will be architecture-specific. So maybe what we need is actually a conda-lock-amd64.txt, conda-lock-arm64.txt? Then workflows would need to be run with architecture-specific configurations referencing these?

Can you embed multiple architectures in a lock file? The docs kinda say you can: https://github.com/conda/conda-lock?tab=readme-ov-file#platform-specification

pinin4fjords · 2024-08-12T11:16:28Z

Can you embed multiple architectures in a lock file? The docs kinda say you can: https://github.com/conda/conda-lock?tab=readme-ov-file#platform-specification

YAAAAS!

 conda-lock -f environment.yml -p osx-64 -p linux-64
Spec hash already locked for ['osx-arm64']. Skipping solve.
Locking dependencies for ['linux-64', 'osx-64']...
INFO:conda_lock.conda_solver:linux-64 using specs ['bioconda::multiqc=1.23']
...

Good man. We would probably want to put them in the environment.yml, given that cross-platform compatibility will vary.

pinin4fjords · 2024-08-12T11:54:50Z

.github/workflows/wave.yml

+      - name: Run conda-lock
+        run: |
+          rm --force "${{steps.lockfile-name.outputs.result}}"
+          conda-lock lock $MAMBA --file "${{ matrix.files }}" --kind lock --platform linux-64 --lockfile "${{steps.lockfile-name.outputs.result}}"


So, I tested why the lock file didn't work, and I think it's because of the default unified lock files. To get a lock file compatible with Nextflow (which just does like this), you have to render it to a platform specific one:

conda-lock render -p osx-64 Rendering lockfile(s) for osx-64... - Install lock using : conda create --name YOURENV --file conda-osx-64.lock

We can also just do:

conda-lock --kind explicit -f environment.yml

... which automatically renders to all the specified platforms.

In any case, for nextflow as-is, we will need platform-specific lockfiles.

Awesome! I didn't really want to have 4+ lock files floating around. I didn't know this was a possibility, great find!

I'm afraid we're stuck with those 4 lock files, unless we do that rendering dynamically somehow.

pinin4fjords · 2024-08-12T11:56:08Z

@adamrtalbot raises the excellent point that lock files will be architecture-specific. So maybe what we need is actually a conda-lock-amd64.txt, conda-lock-arm64.txt? Then workflows would need to be run with architecture-specific configurations referencing these?

Can you embed multiple architectures in a lock file? The docs kinda say you can: https://github.com/conda/conda-lock?tab=readme-ov-file#platform-specification

@adamrtalbot unfortunately I just discovered that the reason @edmundmiller 's new conda declaration didn't work is probably because of unified configs. So absent a fix to Nextflow we'll be needing platform-specific ones.

Edit: Paolo nixed my idea for supporting unified lock files. So we are definitely limited to the platform-specific ones.

pinin4fjords

So basically I think we'll need to do like this

pinin4fjords · 2024-08-12T12:17:04Z

.github/workflows/wave.yml

+      - name: Run conda-lock
+        run: |
+          rm --force "${{steps.lockfile-name.outputs.result}}"
+          conda-lock lock $MAMBA --file "${{ matrix.files }}" --kind lock --platform linux-64 --lockfile "${{steps.lockfile-name.outputs.result}}"


Suggested change

conda-lock lock $MAMBA --file "${{ matrix.files }}" --kind lock --platform linux-64 --lockfile "${{steps.lockfile-name.outputs.result}}"

conda-lock lock $MAMBA --file "${{ matrix.files }}" --kind explicit --platform linux-64 --lockfile "${{steps.lockfile-name.outputs.result}}"

mv conda-linux-64.lock conda-lock-linux-64.yml

Nextflow needs it to be a .yml, but use of --kind explicit means we get .lock files, so we'll need to do some file renaming.

Platforms should probably be set in the environment.ymls actually.

Can we just do...

Suggested change

conda-lock lock $MAMBA --file "${{ matrix.files }}" --kind lock --platform linux-64 --lockfile "${{steps.lockfile-name.outputs.result}}"

conda-lock lock $MAMBA --file "${{ matrix.files }}" --kind lock --platform linux-64 --lockfile "${{steps.lockfile-name.outputs.result}}".yml

Doesn't look like that works 😞

The explicit thing is important. It makes you a lock file for each platform which is consumable by conda env create, and therefore by Nextflow.

But by doing explicit you lose control over the naming of the multiple outputs, and Nextflow needs .yml, hence the rename hack I suggested.

pinin4fjords · 2024-08-12T12:17:13Z

.github/workflows/wave.yml

+          git config push.default upstream
+          git add .
+          git status
+          git commit -m "[automated] autogenerated conda-lock file"


Suggested change

git commit -m "[automated] autogenerated conda-lock file"

git commit -m "[automated] autogenerated conda-lock files"

It should just be a single commit per lock file I think?

One lock file per platform

pinin4fjords · 2024-08-12T12:21:46Z

modules/nf-core/bowtie2/align/main.nf

@@ -2,7 +2,7 @@ process BOWTIE2_ALIGN {
    tag "$meta.id"
    label 'process_high'

-    conda "${moduleDir}/environment.yml"
+    conda "${moduleDir}/conda-lock.yml"


Suggested change

conda "${moduleDir}/conda-lock.yml"

conda "${moduleDir}/conda-lock-linux-64.yml"

(I'm obviously not proposing we actually change modules for platform specificity, just illustrating).

I think tracking them at this point might keep us sane though 😅

Because the lock files will be platform specific they can't go directly on the modules. I think we'd need platform-specific configs at the workflow level that referenced all the lockfiles.

pinin4fjords · 2024-08-12T12:23:31Z

.prettierignore

@@ -15,3 +15,4 @@ __pycache__
 *.pyo
 *.pyc
 .github/renovate.json5
+**/conda-lock.yml


Suggested change

**/conda-lock.yml

**/conda-lock*.yml

edmundmiller self-assigned this Jun 18, 2024

edmundmiller requested a review from pinin4fjords June 18, 2024 19:48

edmundmiller added this to the Wave/Seqera Containers Migration milestone Jun 18, 2024

edmundmiller mentioned this pull request Jun 18, 2024

Batch module updates - Fall 2024 #5828

Open

ewels linked an issue Jun 19, 2024 that may be closed by this pull request

Use Conda lock files #5835

Open

ewels changed the base branch from master to batch_update_staging June 19, 2024 13:15

edmundmiller mentioned this pull request Jul 16, 2024

Create frozen Conda environments for modules nf-core/tools#2193

Open

edmundmiller force-pushed the lock-files branch from 440bca1 to 70acdbe Compare August 8, 2024 16:55

edmundmiller changed the base branch from batch_update_staging to master August 8, 2024 17:14

edmundmiller force-pushed the lock-files branch 5 times, most recently from 13df3e3 to a1bb7d5 Compare August 8, 2024 17:57

edmundmiller marked this pull request as ready for review August 8, 2024 18:05

edmundmiller requested review from JoseEspinosa and drpatelh as code owners August 8, 2024 18:05

edmundmiller requested a review from mashehu August 8, 2024 18:05

edmundmiller and others added 10 commits August 9, 2024 13:22

ci(#2193): Add condalock file generation

dbad13e

https://github.com/seqeralabs/wave-cli?tab=readme-ov-file#build-a-container-by-using-a-conda-lock-file

ci: Add write permissions

6ea762d

ci: Add gh pr checkout

b740f85

ci: Try a pre-canned action

95ba6cc

ci: Cut out conda-lock-refresh

07a250e

Couldn't get it to commit

ci: Use plain GITHUB_TOKEN secret

b3ef764

ci: Skip committing

98cbce3

ci: Use Commit step from prettier linting

253a398

https://github.com/nf-core/modules/blob/90b1d1bd7f9474e173abd9e4082dedb559c7542a/.github/workflows/fix-linting.yml#L62-L71 Co-authored-by: mashehu <mashehu@users.noreply.github.com>

ci: Try actions/checkout#719

f085cc1

ci: Clean up commit message

fe828fa

edmundmiller and others added 7 commits August 9, 2024 13:22

ci: Output lock file to the directory

8433c87

chore: Update an environment file for testing

a9f4d26

[automated] autogenerated conda-lock file

22c4985

ci: Clean up the matrix to avoid a race condition

3afecb6

style: Ignore conda-lock for prettier

c6b7ae6

ci: Allow no conda-lock updates

251b8ca

build: Try a conda-lock file

de2fdf7

edmundmiller force-pushed the lock-files branch from bfc0973 to de2fdf7 Compare August 9, 2024 18:23

pinin4fjords reviewed Aug 12, 2024

View reviewed changes

modules/nf-core/bowtie2/align/main.nf Show resolved Hide resolved

pinin4fjords mentioned this pull request Aug 12, 2024

Enable use of conda-lock outputs in conda declaration nextflow-io/nextflow#5219

Closed

pinin4fjords reviewed Aug 12, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Conda lock files #5827

Conda lock files #5827

edmundmiller commented Jun 18, 2024 •

edited

Loading

edmundmiller commented Aug 8, 2024

pinin4fjords commented Aug 9, 2024

pinin4fjords commented Aug 12, 2024

adamrtalbot commented Aug 12, 2024

pinin4fjords commented Aug 12, 2024 •

edited

Loading

pinin4fjords Aug 12, 2024 •

edited

Loading

edmundmiller Aug 12, 2024

pinin4fjords Aug 13, 2024

pinin4fjords commented Aug 12, 2024 •

edited

Loading

pinin4fjords left a comment

pinin4fjords Aug 12, 2024

edmundmiller Aug 12, 2024

edmundmiller Aug 12, 2024

pinin4fjords Aug 13, 2024

pinin4fjords Aug 12, 2024

edmundmiller Aug 12, 2024

pinin4fjords Aug 13, 2024

pinin4fjords Aug 12, 2024

pinin4fjords Aug 12, 2024

edmundmiller Aug 12, 2024

pinin4fjords Aug 13, 2024

pinin4fjords Aug 12, 2024

	conda-lock lock $MAMBA --file "${{ matrix.files }}" --kind lock --platform linux-64 --lockfile "${{steps.lockfile-name.outputs.result}}"
	conda-lock lock $MAMBA --file "${{ matrix.files }}" --kind explicit --platform linux-64 --lockfile "${{steps.lockfile-name.outputs.result}}"
	mv conda-linux-64.lock conda-lock-linux-64.yml

	git commit -m "[automated] autogenerated conda-lock file"
	git commit -m "[automated] autogenerated conda-lock files"

	conda "${moduleDir}/conda-lock.yml"
	conda "${moduleDir}/conda-lock-linux-64.yml"

Conda lock files #5827

Are you sure you want to change the base?

Conda lock files #5827

Conversation

edmundmiller commented Jun 18, 2024 • edited Loading

edmundmiller commented Aug 8, 2024

pinin4fjords commented Aug 9, 2024

pinin4fjords commented Aug 12, 2024

adamrtalbot commented Aug 12, 2024

pinin4fjords commented Aug 12, 2024 • edited Loading

pinin4fjords Aug 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pinin4fjords commented Aug 12, 2024 • edited Loading

pinin4fjords left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edmundmiller commented Jun 18, 2024 •

edited

Loading

pinin4fjords commented Aug 12, 2024 •

edited

Loading

pinin4fjords Aug 12, 2024 •

edited

Loading

pinin4fjords commented Aug 12, 2024 •

edited

Loading