JupyLab Tool Issues

[nomadscientist]

JupyLab/Jupyter tool issues collected from 2024 Bioinfo Bootcamp preparation and trying to get these tutorials to work after updates:

Please see notes here about get/put issues & installation issues & kernel switching
galaxyproject/training-material#5297

Possible ideal future = have a single cell Jupyter (that auto-updates with any main updates to the tool) but has 'slow' packages pre-installed...

[nomadscientist]

@wee-snufkin
@FoxHin5431

[bgruening]

Can you please test this again with a fresh jupyter notebook?

[nomadscientist]

OK -

1. We need to calculate how much resource each step in the notebook takes, can you point me to a screenshot or FAQ on how to do this?
2. One key time-suck in this tutorial is they have to install salmon (wget -nv https://github.com/COMBINE-lab/salmon/releases/download/v1.10.0/salmon-1.10.0_linux_x86_64.tar.gz) and unzip it (tar -xvzf salmon-1.10.0_linux_x86_64.tar.gz) and also this (conda install -y -c conda-forge -c bioconda atlas-gene-annotation-manipulation) which takes a while.

Can these be just inherent in a Jupyter Notebook to avoid these long steps? We'll keep the code in a comments box for users to apply to other scenarios, like GoogleCollab etc., but it would speed up usage a lot if we didn't have to install these every time.

[FoxHin5431]

Hi Wendi,
Would something like this work, I only did it for the bash parts of the alevin tutorial - as each kernel needs a different way to do this. Unless there is a system behind the notebook I can't access it. I wrote some code to do it, logging the CPU and memory times and then some code to compare the start and finish times of the cells run to the collected CPU and memory times.

memory_usage.log

task_duration_data.xlsx

*I don't trust cell 9 times - the browser kept crashing, not jupyters fault.

Best

Mark

log files

cell_log.log
cpu_usage.log

[nomadscientist]

Hmm, am I understanding this correctly that if we manage to remove the GTF annotation step (or, perhaps, run it but only on 1 chromosome or something similar) then we will substantially reduce the resources? I'm surprised, as I thought Alevin was the killer.
Regardless, the memory used plot - is this what's kicking over the 4GB allotment per notebook? Although it's expressed as a % so I am not sure if I'm interpreting it correctly. Can you express those plots as raw data Mark?

[wee-snufkin]

2. One key time-suck in this tutorial is they have to install salmon (wget -nv https://github.com/COMBINE-lab/salmon/releases/download/v1.10.0/salmon-1.10.0_linux_x86_64.tar.gz) and unzip it (tar -xvzf salmon-1.10.0_linux_x86_64.tar.gz) and also this (conda install -y -c conda-forge -c bioconda atlas-gene-annotation-manipulation) which takes a while.

Just to clarify - Salmon can be also installed using conda. In the tutorial the alternative way of downloading and unziping was shown because at the time of writing the tutorial it worked better and quicker compared to conda installation.

Can these be just inherent in a Jupyter Notebook to avoid these long steps? We'll keep the code in a comments box for users to apply to other scenarios, like GoogleCollab etc., but it would speed up usage a lot if we didn't have to install these every time.

But having salmon and dropletutils pre-installed (as well as atlas-gene-annotation-manipulation and tximeta)in Jupyter Notebook would make running this tutorial much easier.

[wee-snufkin]

Hmm, am I understanding this correctly that if we manage to remove the GTF annotation step (or, perhaps, run it but only on 1 chromosome or something similar) then we will substantially reduce the resources? I'm surprised, as I thought Alevin was the killer. Regardless, the memory used plot - is this what's kicking over the 4GB allotment per notebook? Although it's expressed as a % so I am not sure if I'm interpreting it correctly. Can you express those plots as raw data Mark?

Isn't Alevin the killer, having crashed Mark's browser multiple times? It rather doesn't happen in Google Colab

[FoxHin5431]

Something like this:

Looks like cell 7 goes almost over 5gb

[FoxHin5431]

Hmm, am I understanding this correctly that if we manage to remove the GTF annotation step (or, perhaps, run it but only on 1 chromosome or something similar) then we will substantially reduce the resources? I'm surprised, as I thought Alevin was the killer. Regardless, the memory used plot - is this what's kicking over the 4GB allotment per notebook? Although it's expressed as a % so I am not sure if I'm interpreting it correctly. Can you express those plots as raw data Mark?

Isn't Alevin the killer, having crashed Mark's browser multiple times? It rather doesn't happen in Google Colab

I was running a lot of browser tabs at the same time but it could have been jupyter - will try it again in the morning - I saved my version of the pipeline with the monitoring - so should be able to just set it up again.

[bgruening]

In the information site of every Galaxy dataset you will also find something like that:

The "Max memory usage recorded" would be interesting.

[FoxHin5431]

Hi,
Here are the max memory and duration attached to the tasks.

start	finish	max_usage_gb	duration	cell_task
10/10/2024 13:29	10/10/2024 13:30	4.81	5 seconds	gtf2annotation
10/10/2024 13:30	10/10/2024 13:30	4.81	5 seconds	gtf2annotation
10/10/2024 13:30	10/10/2024 13:30	4.81	5 seconds	gtf2annotation
10/10/2024 13:30	10/10/2024 13:31	4.82	5 seconds	gtf2annotation
10/10/2024 13:31	10/10/2024 13:33	5.05	2.0 minutes	gtf2annotation