Skip to content

Releases: 4dn-dcic/tibanna

0.3.7

22 Jul 19:19
001c584
Compare
Choose a tag to compare
  • Tibanna unicorn is now independent of dcicutils.
  • Unicorn not throwing error because it was set to pass it to the last step of pony is now fixed.

0.3.6

12 Jun 17:57
ce3957c
Compare
Choose a tag to compare
  • Major changes
    • Use of dcicutils instead of Submit4dn
    • CWL v1 support

0.3.5

15 Feb 03:40
c5f6b2e
Compare
Choose a tag to compare
  • postrunjson now patched to ffmeta
  • rerun, rerun_many and kill_all added as invoke methods
  • launch_utils added
  • framework for metadata run_status update when error happens at 2nd/3rd step.
  • an issue with secondary input file handling is fixed
  • some issues with pseudo workflow run got fixed

0.3.4

02 Feb 14:17
Compare
Choose a tag to compare
  • Now handling pseudo-workflows (when output_files are provided, tibanna creates a pseudo workflow_run object and links it to the output files, without actually running a workflow)

0.3.3

23 Jan 19:53
Compare
Choose a tag to compare
  • removed SBG
  • now takes experiments or source_experiments field from input file metadata and adds to the output processed file metadata
  • now reports total space used for input/temp/output files to postrun.json and step function output.
  • fixed an issue that the update_ffmeta_awsf step does not pass everything.

0.3.2

17 Jan 18:31
199c077
Compare
Choose a tag to compare
  • A couple of changes have been made for nice handling of Pairsqc.
    • secondary input file handling
    • the last step of awsem (update_ffmeta_awsf) is updated to have integrated qc parser and updater to handle both fastqc and pairsqc

v0.3.1

12 Dec 17:55
50f2925
Compare
Choose a tag to compare
  • Tibanna branch (url) can be specified for awsf functions inside an input json file.
  • Custom fields in CWL are now accepted.

v0.3

06 Dec 22:08
Compare
Choose a tag to compare

This release is nearly completely different from the previous release.

  • AWS step functions are used as an upstream scheduler.
  • The older awsf component is now working as a lambda function which generates an autonomous instance called awsem.
  • Command-line launch uses invoke.
  • Jupyter notebooks are used for connecting between steps and communicating with 4DN fourfront. (This functionality is not meant for general public)

v0.2

19 Sep 19:43
Compare
Choose a tag to compare

Version v0.2 has been tested.

Changes made:
  • The whole process is now automated. The script create_json.py will create a json file, upload to s3, launch an instance that executes the workflow, sends logs to s3 and self-terminates. It uses the .tibanna.config file for storing relatively invariant parameters (e.g. reference bucket name, AMI, keyname, etc) and takes input files, parameters, etc as command-line options. It also adds the launched instance information to a job_list file to enable easier tracking.
  • minor change in the json format. (the previous format may not work especially for a multi-layer cwl)

v0.1

17 Sep 02:22
Compare
Choose a tag to compare

v0.1 is tested on aws.
Changes made:

  • Scripts/cwls are pulled from github instead of S3
  • intermittent logging to S3 (not in 'real time', but still better than nothing)
  • Large EBS (>2.5TB) can be mounted. EBS is now mounted to a nonroot directory so that large EBS volumes can be attached (2.5TB EBS can't be attached to root).
  • Different EBS types are handled and especially io1 type with 5000 IOPS is set to default (very fast data transfer between S3 and EBS)
  • Several directory handling issues related to cwl/docker (that came with the new nonroot EBS) were resolved
  • run_workflow.(JOBID).sh is now automatically created by create_run_workflow.sh
  • Minor format change in input json. (previous one doens't work anymore)