Development: Batching

Batching

A key issue is the number of batches into which the work is divided. When the compute cluster is busy, many small jobs can waste a lot of time in queuing.

• 50 jobs in 50 batches for 3.30.70.1470 (which has 45 starting clusters), see 3.30.70.1470.progress.log
• 94 jobs in 89 batches for 1.10.1390.10 (which has 189 starting clusters), see 1.10.1390.10.progress.log
• 452 jobs in 353 batches for 3.20.20.120 (which has 614 starting clusters), see 3.40.50.970.progress.log
• 763 jobs in 343 batches for 3.40.50.970 (which has 1384 starting clusters), see 3.20.20.120.progress.log
• 8352 jobs in 541 batches for 3.40.50.620 (which has 7764 starting clusters), see 3.40.50.620.progress.log

Commands to get these sorts of numbers:

grep -c 'submit' 3.40.50.620.progress.log
grep -Po '\d+\s+submitted' 3.40.50.620.progress.log | awk '{print $1}' | sort -g | uniq -c | awk '{print $1, "*", $2}' | tr '\n' '+' | sed 's/+$//g' | xargs -I VAR perl -e 'print "".( VAR )."\n"; '