Skip to content

Commit

Permalink
Merge branch 'dev' into modules-template-indentation
Browse files Browse the repository at this point in the history
  • Loading branch information
awgymer authored Mar 19, 2024
2 parents b16f670 + b9699d2 commit ac27026
Show file tree
Hide file tree
Showing 14 changed files with 75 additions and 57 deletions.
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
repos:
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.3.2
rev: v0.3.3
hooks:
- id: ruff # linter
args: [--fix, --exit-non-zero-on-fix] # sort imports and fix
Expand Down
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,17 @@
- Update templates to use nf-core/setup-nextflow v2
- Link to troubleshooting docs when pipeline fails ([#2845](https://github.com/nf-core/tools/pull/2845))
- Add fallback to `download_pipeline.yml` in case the pipeline does not support stub runs ([#2846](https://github.com/nf-core/tools/pull/2846))
- Set topic variable correctly in the mastodon announcement ([#2848](https://github.com/nf-core/tools/pull/2848))
- Add a cleanup action to `download_pipeline.yml` to fix failures caused by inadequate storage space on the runner ([#2849](https://github.com/nf-core/tools/pull/2849))

### Linting

- Only match assignments of params in `main.nf` and not references like `params.aligner == <something>` ([#2833](https://github.com/nf-core/tools/pull/2833))

### Download

- Replace `--tower` with `--platform`. The former will remain for backwards compatability for now but will be removed in a future release.

### Components

### General
Expand All @@ -29,6 +34,8 @@
- Optimize layers in dockerfile ([#2842](https://github.com/nf-core/tools/pull/2842))
- Update gitpod/workspace-base Docker digest to 1e133e5 ([#2843](https://github.com/nf-core/tools/pull/2843))
- Update python:3.11-slim Docker digest to a2eb07f ([#2847](https://github.com/nf-core/tools/pull/2847))
- Strip out mention of "Nextflow Tower" and replace with "Seqera Platform" wherever possible
- Update pre-commit hook astral-sh/ruff-pre-commit to v0.3.3 ([#2850](https://github.com/nf-core/tools/pull/2850))

## [v2.13.1 - Tin Puppy Patch](https://github.com/nf-core/tools/releases/tag/2.13) - [2024-02-29]

Expand Down
File renamed without changes.
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -376,7 +376,7 @@ The pipeline files are automatically updated (`params.custom_config_base` is set
So using `-profile <NAME>` should work if available within [nf-core/configs](https://github.com/nf-core/configs).

> [!WARNING]
> This option is not available when downloading a pipeline for use with [Nextflow Tower](#adapting-downloads-to-nextflow-tower) because the application manages all configurations separately.
> This option is not available when downloading a pipeline for use with [Seqera Platform](#adapting-downloads-to-seqera-platform) because the application manages all configurations separately.
### Downloading Apptainer containers

Expand Down Expand Up @@ -428,14 +428,14 @@ Note that compressing many GBs of binary files can be slow, so specifying `--com

If the download speeds are much slower than your internet connection is capable of, you can set `--parallel-downloads` to a large number to download loads of images at once.

### Adapting downloads to Nextflow Tower
### Adapting downloads to Seqera Platform

[seqeralabs® Nextflow Tower](https://cloud.tower.nf/) provides a graphical user interface to oversee pipeline runs, gather statistics and configure compute resources. While pipelines added to _Tower_ are preferably hosted at a Git service, providing them as disconnected, self-reliant repositories is also possible for premises with restricted network access. Choosing the `--tower` flag will download the pipeline in an appropriate form.
[Seqera Platform](https://seqera.io/platform/) (formerly _"Nextflow Tower"_) provides a graphical user interface to oversee pipeline runs, gather statistics and configure compute resources. While pipelines added to _Seqera Platform_ are preferably hosted at a Git service, providing them as disconnected, self-reliant repositories is also possible for premises with restricted network access. Choosing the `--platform` flag will download the pipeline in an appropriate form.

Subsequently, the `*.git` folder can be moved to it's final destination and linked with a pipeline in _Tower_ using the `file:/` prefix.
Subsequently, the `*.git` folder can be moved to it's final destination and linked with a pipeline in _Seqera Platform_ using the `file:/` prefix.

> [!TIP]
> Also without access to Tower, pipelines downloaded with the `--tower` flag can be run if the _absolute_ path is specified: `nextflow run -r 2.5 file:/path/to/pipelinedownload.git`. Downloads in this format allow you to include multiple revisions of a pipeline in a single file, but require that the revision (e.g. `-r 2.5`) is always explicitly specified.
> Also without access to Seqera Platform, pipelines downloaded with the `--platform` flag can be run if the _absolute_ path is specified: `nextflow run -r 2.5 file:/path/to/pipelinedownload.git`. Downloads in this format allow you to include multiple revisions of a pipeline in a single file, but require that the revision (e.g. `-r 2.5`) is always explicitly specified.
## Pipeline software licences

Expand Down
15 changes: 12 additions & 3 deletions nf_core/__main__.py
Original file line number Diff line number Diff line change
Expand Up @@ -358,19 +358,27 @@ def create_params_file(pipeline, revision, output, force, show_hidden):
help="Archive compression type",
)
@click.option("-f", "--force", is_flag=True, default=False, help="Overwrite existing files")
# TODO: Remove this in a future release. Deprecated in March 2024.
@click.option(
"-t",
"--tower",
is_flag=True,
default=False,
hidden=True,
help="Download for Seqera Platform. DEPRECATED: Please use --platform instead.",
)
@click.option(
"-t",
"--platform",
is_flag=True,
default=False,
help="Download for Seqera Platform (formerly Nextflow Tower)",
)
@click.option(
"-d",
"--download-configuration",
is_flag=True,
default=False,
help="Include configuration profiles in download. Not available with `--tower`",
help="Include configuration profiles in download. Not available with `--platform`",
)
# -c changed to -s for consistency with other --container arguments, where it is always the first letter of the last word.
# Also -c might be used instead of -d for config in a later release, but reusing params for different options in two subsequent releases might be too error-prone.
Expand Down Expand Up @@ -412,6 +420,7 @@ def download(
compress,
force,
tower,
platform,
download_configuration,
container_system,
container_library,
Expand All @@ -433,7 +442,7 @@ def download(
outdir,
compress,
force,
tower,
tower or platform, # True if either specified
download_configuration,
container_system,
container_library,
Expand Down
48 changes: 25 additions & 23 deletions nf_core/download.py
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ class DownloadWorkflow:
pipeline (str): A nf-core pipeline name.
revision (List[str]): The workflow revision to download, like `1.0`. Defaults to None.
container (bool): Flag, if the Singularity container should be downloaded as well. Defaults to False.
tower (bool): Flag, to customize the download for Nextflow Tower (convert to git bare repo). Defaults to False.
platform (bool): Flag, to customize the download for Seqera Platform (convert to git bare repo). Defaults to False.
outdir (str): Path to the local download directory. Defaults to None.
"""

Expand All @@ -101,7 +101,7 @@ def __init__(
outdir=None,
compress_type=None,
force=False,
tower=False,
platform=False,
download_configuration=None,
container_system=None,
container_library=None,
Expand All @@ -120,11 +120,11 @@ def __init__(
self.output_filename = None
self.compress_type = compress_type
self.force = force
self.tower = tower
self.platform = platform
# if flag is not specified, do not assume deliberate choice and prompt config inclusion interactively.
# this implies that non-interactive "no" choice is only possible implicitly (e.g. with --tower or if prompt is suppressed by !stderr.is_interactive).
# this implies that non-interactive "no" choice is only possible implicitly (e.g. with --platform or if prompt is suppressed by !stderr.is_interactive).
# only alternative would have been to make it a parameter with argument, e.g. -d="yes" or -d="no".
self.include_configs = True if download_configuration else False if bool(tower) else None
self.include_configs = True if download_configuration else False if bool(platform) else None
# Specifying a cache index or container library implies that containers should be downloaded.
self.container_system = "singularity" if container_cache_index or bool(container_library) else container_system
# Manually specified container library (registry)
Expand Down Expand Up @@ -165,8 +165,8 @@ def download_workflow(self):
)
self.prompt_revision()
self.get_revision_hash()
# Inclusion of configs is unnecessary for Tower.
if not self.tower and self.include_configs is None:
# Inclusion of configs is unnecessary for Seqera Platform.
if not self.platform and self.include_configs is None:
self.prompt_config_inclusion()
# If a remote cache is specified, it is safe to assume images should be downloaded.
if not self.container_cache_utilisation == "remote":
Expand All @@ -177,7 +177,7 @@ def download_workflow(self):
self.prompt_singularity_cachedir_utilization()
self.prompt_singularity_cachedir_remote()
# Nothing meaningful to compress here.
if not self.tower:
if not self.platform:
self.prompt_compression_type()
except AssertionError as e:
raise DownloadError(e) from e
Expand All @@ -196,7 +196,7 @@ def download_workflow(self):
)

# Set an output filename now that we have the outdir
if self.tower:
if self.platform:
self.output_filename = f"{self.outdir}.git"
summary_log.append(f"Output file: '{self.output_filename}'")
elif self.compress_type is not None:
Expand All @@ -205,11 +205,11 @@ def download_workflow(self):
else:
summary_log.append(f"Output directory: '{self.outdir}'")

if not self.tower:
if not self.platform:
# Only show entry, if option was prompted.
summary_log.append(f"Include default institutional configuration: '{self.include_configs}'")
else:
summary_log.append(f"Enabled for seqeralabs® Nextflow Tower: '{self.tower}'")
summary_log.append(f"Enabled for Seqera Platform: '{self.platform}'")

# Check that the outdir doesn't already exist
if os.path.exists(self.outdir):
Expand All @@ -233,8 +233,8 @@ def download_workflow(self):
log.info("Saving '{}'\n {}".format(self.pipeline, "\n ".join(summary_log)))

# Perform the actual download
if self.tower:
self.download_workflow_tower()
if self.platform:
self.download_workflow_platform()
else:
self.download_workflow_static()

Expand Down Expand Up @@ -273,7 +273,7 @@ def download_workflow_static(self):
log.info("Compressing output into archive")
self.compress_download()

def download_workflow_tower(self, location=None):
def download_workflow_platform(self, location=None):
"""Create a bare-cloned git repository of the workflow, so it can be launched with `tw launch` as file:/ pipeline"""

log.info("Collecting workflow from GitHub")
Expand All @@ -289,7 +289,7 @@ def download_workflow_tower(self, location=None):
# Remove tags for those revisions that had not been selected
self.workflow_repo.tidy_tags_and_branches()

# create a bare clone of the modified repository needed for Tower
# create a bare clone of the modified repository needed for Seqera Platform
self.workflow_repo.bare_clone(os.path.join(self.outdir, self.output_filename))

# extract the required containers
Expand All @@ -306,9 +306,11 @@ def download_workflow_tower(self, location=None):
except OSError as e:
raise DownloadError(f"[red]{e}[/]") from e

# Justify why compression is skipped for Tower downloads (Prompt is not shown, but CLI argument could have been set)
# Justify why compression is skipped for Seqera Platform downloads (Prompt is not shown, but CLI argument could have been set)
if self.compress_type is not None:
log.info("Compression choice is ignored for Tower downloads since nothing can be reasonably compressed.")
log.info(
"Compression choice is ignored for Seqera Platform downloads since nothing can be reasonably compressed."
)

def prompt_pipeline_name(self):
"""Prompt for the pipeline name if not set with a flag"""
Expand All @@ -321,13 +323,13 @@ def prompt_revision(self):
"""
Prompt for pipeline revision / branch
Prompt user for revision tag if '--revision' was not set
If --tower is specified, allow to select multiple revisions
If --platform is specified, allow to select multiple revisions
Also the static download allows for multiple revisions, but
we do not prompt this option interactively.
"""
if not bool(self.revision):
(choice, tag_set) = nf_core.utils.prompt_pipeline_release_branch(
self.wf_revisions, self.wf_branches, multiple=self.tower
self.wf_revisions, self.wf_branches, multiple=self.platform
)
"""
The checkbox() prompt unfortunately does not support passing a Validator,
Expand Down Expand Up @@ -384,7 +386,7 @@ def get_revision_hash(self):
else:
self.outdir = f"{self.pipeline.replace('/', '-').lower()}_{self.revision[0]}"

if not self.tower:
if not self.platform:
for revision, wf_sha in self.wf_sha.items():
# Set the download URL and return - only applicable for classic downloads
self.wf_download_url = {
Expand All @@ -406,7 +408,7 @@ def prompt_config_inclusion(self):
def prompt_container_download(self):
"""Prompt whether to download container images or not"""

if self.container_system is None and stderr.is_interactive and not self.tower:
if self.container_system is None and stderr.is_interactive and not self.platform:
stderr.print("\nIn addition to the pipeline code, this tool can download software containers.")
self.container_system = questionary.select(
"Download software container images:",
Expand Down Expand Up @@ -1621,7 +1623,7 @@ def setup_local_repo(self, remote, location=None, in_cache=True):
def tidy_tags_and_branches(self):
"""
Function to delete all tags and branches that are not of interest to the downloader.
This allows a clutter-free experience in Tower. The untagged commits are evidently still available.
This allows a clutter-free experience in Seqera Platform. The untagged commits are evidently still available.
However, due to local caching, the downloader might also want access to revisions that had been deleted before.
In that case, don't bother with re-adding the tags and rather download anew from Github.
Expand Down Expand Up @@ -1655,7 +1657,7 @@ def tidy_tags_and_branches(self):
if self.repo.head.is_detached:
self.repo.head.reset(index=True, working_tree=True)

# no branch exists, but one is required for Tower's UI to display revisions correctly). Thus, "latest" will be created.
# no branch exists, but one is required for Seqera Platform's UI to display revisions correctly). Thus, "latest" will be created.
if not bool(self.repo.heads):
if self.repo.is_valid_object("latest"):
# "latest" exists as tag but not as branch
Expand Down
4 changes: 2 additions & 2 deletions nf_core/lint/actions_awsfulltest.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ def actions_awsfulltest(self):
.. note:: You can manually trigger the AWS tests by going to the `Actions` tab on the pipeline GitHub repository and selecting the
`nf-core AWS full size tests` workflow on the left.
.. tip:: For tests on full data prior to release, `Nextflow Tower <https://tower.nf>`_ launch feature can be employed.
.. tip:: For tests on full data prior to release, `Seqera Platform <https://seqera.io/platform/>`_ launch feature can be employed.
The ``.github/workflows/awsfulltest.yml`` file is tested for the following:
Expand Down Expand Up @@ -52,7 +52,7 @@ def actions_awsfulltest(self):

# Warn if `-profile test` is still unchanged
try:
steps = wf["jobs"]["run-tower"]["steps"]
steps = wf["jobs"]["run-platform"]["steps"]
if not any(aws_profile in step["run"] for step in steps if "run" in step.keys()):
raise AssertionError()
except (AssertionError, KeyError, TypeError):
Expand Down
10 changes: 5 additions & 5 deletions nf_core/pipeline-template/.github/workflows/awsfulltest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,12 @@ on:
types: [published]
workflow_dispatch:
jobs:
run-tower:
run-platform:
name: Run AWS full tests
if: github.repository == '{{ name }}'
runs-on: ubuntu-latest
steps:
- name: Launch workflow via tower
- name: Launch workflow via Seqera Platform
uses: seqeralabs/action-tower-launch@v2
# TODO nf-core: You can customise AWS full pipeline tests as required
# Add full size test data (but still relatively small datasets for few samples)
Expand All @@ -33,7 +33,7 @@ jobs:

- uses: actions/upload-artifact@v4
with:
name: Tower debug log file
name: Seqera Platform debug log file
path: |
tower_action_*.log
tower_action_*.json
seqera_platform_action_*.log
seqera_platform_action_*.json
12 changes: 6 additions & 6 deletions nf_core/pipeline-template/.github/workflows/awstest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,13 @@ name: nf-core AWS test
on:
workflow_dispatch:
jobs:
run-tower:
run-platform:
name: Run AWS tests
if: github.repository == '{{ name }}'
runs-on: ubuntu-latest
steps:
# Launch workflow using Tower CLI tool action {%- raw %}
- name: Launch workflow via tower
# Launch workflow using Seqera Platform CLI tool action {%- raw %}
- name: Launch workflow via Seqera Platform
uses: seqeralabs/action-tower-launch@v2
with:
workspace_id: ${{ secrets.TOWER_WORKSPACE_ID }}
Expand All @@ -27,7 +27,7 @@ jobs:

- uses: actions/upload-artifact@v4
with:
name: Tower debug log file
name: Seqera Platform debug log file
path: |
tower_action_*.log
tower_action_*.json
seqera_platform_action_*.log
seqera_platform_action_*.json
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ jobs:
- name: get topics and convert to hashtags
id: get_topics
run: |
curl -s https://nf-co.re/pipelines.json | jq -r '.remote_workflows[] | select(.full_name == "${{ github.repository }}") | .topics[]' | awk '{print "#"$0}' | tr '\n' ' ' >> $GITHUB_OUTPUT
echo "topics=$(curl -s https://nf-co.re/pipelines.json | jq -r '.remote_workflows[] | select(.full_name == "${{ github.repository }}") | .topics[]' | awk '{print "#"$0}' | tr '\n' ' ')" >> $GITHUB_OUTPUT
- uses: rzr/fediverse-action@master
with:
Expand All @@ -25,7 +25,7 @@ jobs:
Please see the changelog: ${{ github.event.release.html_url }}
${{ steps.get_topics.outputs.GITHUB_OUTPUT }} #nfcore #openscience #nextflow #bioinformatics
${{ steps.get_topics.outputs.topics }} #nfcore #openscience #nextflow #bioinformatics
send-tweet:
runs-on: ubuntu-latest
Expand Down
Loading

0 comments on commit ac27026

Please sign in to comment.