Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add SLURM support #215

Open
widdowquinn opened this issue Aug 3, 2020 · 10 comments
Open

Add SLURM support #215

widdowquinn opened this issue Aug 3, 2020 · 10 comments
Assignees
Labels
enhancement something we'd like pyani to do that it doesn't already HIGH PRIORITY high priority issue interface issues related to how the user tells pyani to do something
Milestone

Comments

@widdowquinn
Copy link
Owner

Summary:

pyani currently only supports SGE/OGE-like systems, but SLURM is very popular. It would be useful to support SLURM scheduling.

@widdowquinn widdowquinn added enhancement something we'd like pyani to do that it doesn't already interface issues related to how the user tells pyani to do something labels Aug 3, 2020
@widdowquinn widdowquinn added this to the 0.3.1 milestone Aug 3, 2020
@widdowquinn widdowquinn self-assigned this Aug 3, 2020
@TSL-RamKrishna
Copy link

Hi
I have been working on getting SLURM support for pyani and it seems to be working now.
I have forked pyani and worked on this repo : https://github.com/TeamMacLean/pyani
Basically, i made a copy of run_sge.py and renamed as run_slurm.py and made changes to the job submission commands.
I have also renamed some command option like sgegroupsize to groupsize.

@widdowquinn
Copy link
Owner Author

Many thanks @TSL-RamKrishna - that's great!

As it happens I've just got access to a SLURM cluster, so I can actually try this out.

@TSL-RamKrishna
Copy link

Thanks. Let me know when you try that out. I have push updated the repo today as well.

@widdowquinn widdowquinn linked a pull request Feb 3, 2021 that will close this issue
@widdowquinn widdowquinn modified the milestones: 0.3.1, 0.3.0 May 13, 2021
@widdowquinn
Copy link
Owner Author

The changes have now been pulled into pr_236 to be brought in line with existing tests, CLI expectations, and for further development.

@all-contributors please add @TSL-RamKrishna for code, ideas

@allcontributors
Copy link
Contributor

@widdowquinn

I've put up a pull request to add @TSL-RamKrishna! 🎉

@peterjc
Copy link
Collaborator

peterjc commented Oct 19, 2021

Is the branch https://github.com/widdowquinn/pyani/tree/pr_236 still in progress and the best bet if I urgently needed to run pyANI under SLURM? Is it worth making that into a new pull request to garner feedback on?

@baileythegreen
Copy link
Contributor

Tagging @widdowquinn to make sure he sees this sooner.

@widdowquinn
Copy link
Owner Author

widdowquinn commented Oct 19, 2021

It is still in progress.

The current status is that the pr_236 branch will run on SLURM. However, we need to refactor how we aggregate jobs within pyani because SLURM counts each task within a batch as a single job so that - for example - a 200 genome comparison batched into 10k lumps thus counts as 40,000 jobs and not 4 (x10,000) jobs.

The usual setting of the maximum number of submittable jobs on a SLURM cluster is not much higher than this, which greatly limits the potential for scaling on a SLURM cluster.

The changes required to how we handle jobs in the backend are significant and probably best handled in concert with other changes we have planned. It's a much bigger job than just a drop-in replacement for the old SGE code.

@peterjc
Copy link
Collaborator

peterjc commented Oct 19, 2021

Thanks. I suspected the SLURM batching was still an issue from our discussion elsewhere. I'm hoping to use this on up to 500 genomes, so would hit this problem :(

@widdowquinn
Copy link
Owner Author

Aye - I need to make progress on this for my own stuff, but this time of year is unpleasantly busy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement something we'd like pyani to do that it doesn't already HIGH PRIORITY high priority issue interface issues related to how the user tells pyani to do something
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants