Add more tools from Galaxy Australia TPV config

[cat-bro]

Some tools that have been added to AU's TPV config in the last year or so

[bgruening]

Thanks a lot @cat-bro!

I'm not sure how to review those properly. I hope our cgroups2 setup is soon back and we can contribute numbers.

I tried to flag a few tools that IMHO would waste a lot of resources in the happy-case and where a size-based rule might be good to have.

Thanks again!

[bgruening]

That looks like way too much for the general case. The tool is not doing much afaik.
Maybe we should include here a rule?

[cat-bro]

Thanks very much for reviewing! This rarely uses above the default 3.8GB and only when the inputs are large, so I'll add a rule here.

[bgruening]

This looks also very generous. Most tools will not need more than a few 100 MB for normal files sizes.

[cat-bro]

For needle this needs more than 3.8GB fairly regularly for smallish inputs so I'll target this tool ID on its own. As a general rule it will only need extra memory for large input sizes.

[bgruening]

We have this in place.

  toolshed.g2.bx.psu.edu/repos/galaxyp/eggnog_mapper/eggnog_mapper/.*:
    cores: 8
    mem: 24
    env:
      EGGNOG_DBMEM: "--dbmem"
    params:
      singularity_run_extra_arguments: "--env EGGNOG_DBMEM=--dbmem"
    scheduling:
      require:
        - singularity

[cat-bro]

Thanks @bgruening ! That's a separate tool ID and I'll try this out on Australia. eggnog_mapper was very popular a year ago and is still used regularly. Is there a difference between {params: {singularity_run_extra_arguments: "--env EGGNOG_DBMEM=--dbmem"}} and {env: {SINGULARITYENV_EGGNOG_DBMEM: dbmem}}? We have been using environment variables prefixed with SINGULARITYENV_ and right now I can't remember where I got that from.

[cat-bro]

From the eggnog mapper readme Admins can choose to set the environment variable ``EGGNOG_DBMEM=--dbmem`` which will copy the complete EggNOG annotation DB into memory which is usually much faster than using multiple cores (but needs approx. 37GB of RAM).

[bgruening]

bedtools are used a lot and most of the time 4GB is enough as it seems?

Maybe a rule here?

[cat-bro]

This varies a lot between tools. genomecoveragebed regularly exceeds 3.8 with no input, intersectbed is known to need lots. I might leave this out for now and come back with tool-by-tool estimates.

[bgruening]

That looks strange, but its just a gut feeling :(

[cat-bro]

This one is also hard to generalise. For inputs less that 0.8GB we might be safe with 24GB but it's a steep increase after that. We have jobs running out of memory on Galaxy Australia with 4GB of input - there must be other variables at play. Jobs that try to use too much memory are killed, so we're better off with the local rule in this case.

[bgruening]

Thanks @cat-bro