Skip to content
This repository has been archived by the owner on May 8, 2019. It is now read-only.

Data provenence #31

Open
tonyyzy opened this issue Feb 4, 2019 · 3 comments
Open

Data provenence #31

tonyyzy opened this issue Feb 4, 2019 · 3 comments
Labels
Good to have Non-essential but good to have
Milestone
End of Project

Comments

@tonyyzy
Copy link
Owner

tonyyzy commented Feb 4, 2019

Increase data reproducibility by logging stdout, docker images version.
Look into cwl-runner provenence options
@tonyyzy tonyyzy added the Good to have Non-essential but good to have label Feb 4, 2019
@tonyyzy tonyyzy added this to the End of Project milestone Feb 4, 2019
@tonyyzy
Copy link
Owner Author

tonyyzy commented Feb 10, 2019

cwl provenance is a really good concept and matches our ideas (store a copy of cwl files for tools and workflows)
Tested with our test data, works fine. Also works with --parallel options.
HOWEVER, an error was encountered when testing on Plasmodium data, seems like a javascript error but shouldn't. Without --provenance tag it works find with plasmodium. Change subject_name doesn't work.

zy2414@node023:/data/rnaseq/test/plas$ cwl-runner --provenance=./prov --outdir=./workflow1_results2 ../../RNASeq/workflows/docker/star_samtools_stringtie_prepDE_DESeq2_four_inputs.cwl ./star_samtools_stringtie_prepDE_DESeq2.yml
/project/soft/linux64/anaconda/Anaconda3-2018.12-Linux-x86_64/bin/cwl-runner 1.0.20180820141117
[provenance] Temporary research object: /tmp/tmp7hnt9_ib
Resolved '../../RNASeq/workflows/docker/star_samtools_stringtie_prepDE_DESeq2_four_inputs.cwl' to 'file:///data/rnaseq/RNASeq/workflows/docker/star_samtools_stringtie_prepDE_DESeq2_four_inputs.cwl'
[provenance] Added packed workflow: workflow/packed.cwl
[provenance] Added data file /tmp/tmp7hnt9_ib/data/9b/9bb021d92a1bb7478d90ce7aed6cdec43e344bf8
[provenance] Relative path for data file data/9b/9bb021d92a1bb7478d90ce7aed6cdec43e344bf8
[provenance] Added data file /tmp/tmp7hnt9_ib/data/4e/4ea1d963d87e556fb381ce6cbd2acbc1f4c96966
[provenance] Relative path for data file data/4e/4ea1d963d87e556fb381ce6cbd2acbc1f4c96966
[provenance] Added data file /tmp/tmp7hnt9_ib/data/ef/eff2f2cd3bab93fa537d67ee80ebe593ac0f2d2e
[provenance] Relative path for data file data/ef/eff2f2cd3bab93fa537d67ee80ebe593ac0f2d2e
[provenance] Added data file /tmp/tmp7hnt9_ib/data/f5/f51250ebd5a05d93c38f9cb3adb163c9933caabe
[provenance] Relative path for data file data/f5/f51250ebd5a05d93c38f9cb3adb163c9933caabe
[provenance] Added data file /tmp/tmp7hnt9_ib/data/43/43554b7a5dbca1f9b21c071fb75dcc289c2ce93d
[provenance] Relative path for data file data/43/43554b7a5dbca1f9b21c071fb75dcc289c2ce93d
[provenance] Added data file /tmp/tmp7hnt9_ib/data/70/702ee12acad42938f62a77517b9b9e7cddb1ba16
[provenance] Relative path for data file data/70/702ee12acad42938f62a77517b9b9e7cddb1ba16
[provenance] Added data file /tmp/tmp7hnt9_ib/data/8f/8f417ab11490cb3352513fd26d1fcdc893f9832b
[provenance] Relative path for data file data/8f/8f417ab11490cb3352513fd26d1fcdc893f9832b
[provenance] Added data file /tmp/tmp7hnt9_ib/data/e3/e3dfed0d4ec3517bd44364b435ea94c2f8c692fd
[provenance] Relative path for data file data/e3/e3dfed0d4ec3517bd44364b435ea94c2f8c692fd
[provenance] Added data file /tmp/tmp7hnt9_ib/data/cd/cdf97ffaa124261fadb4d28f37518c7c71c1fab3
[provenance] Relative path for data file data/cd/cdf97ffaa124261fadb4d28f37518c7c71c1fab3
[provenance] Added data file /tmp/tmp7hnt9_ib/data/e9/e90892944e5d3bff5f7336bec92bd772d720d1a3
[provenance] Relative path for data file data/e9/e90892944e5d3bff5f7336bec92bd772d720d1a3
[provenance] Added data file /tmp/tmp7hnt9_ib/data/3e/3edd683d9737e637cfa225c84257bb31cd7e1367
[provenance] Relative path for data file data/3e/3edd683d9737e637cfa225c84257bb31cd7e1367
[provenance] Added data file /tmp/tmp7hnt9_ib/data/35/35c66bf776766acdfb30d5930f8137c99e91113c
[provenance] Relative path for data file data/35/35c66bf776766acdfb30d5930f8137c99e91113c
[provenance] Generated customised job file: workflow/primary-job.json
[provenance] Added data file /tmp/tmp7hnt9_ib/data/b4/b444ac06613fc8d63795be9ad0beaf55011936ac
[provenance] Relative path for data file data/b4/b444ac06613fc8d63795be9ad0beaf55011936ac
[provenance] Added data file /tmp/tmp7hnt9_ib/data/10/109f4b3c50d7b0df729d299bc6f8e9ef9066971f
[provenance] Relative path for data file data/10/109f4b3c50d7b0df729d299bc6f8e9ef9066971f
[provenance] Added data file /tmp/tmp7hnt9_ib/data/3e/3ebfa301dc59196f18593c45e519287a23297589
[provenance] Relative path for data file data/3e/3ebfa301dc59196f18593c45e519287a23297589
[provenance] Added data file /tmp/tmp7hnt9_ib/data/1f/1ff2b3704aede04eecb51e50ca698efd50a1379b
[provenance] Relative path for data file data/1f/1ff2b3704aede04eecb51e50ca698efd50a1379b
[workflow ] start
[workflow ] starting step star_readmap_3
[step star_readmap_3] start
Exception on step 'star_readmap_3'
[step star_readmap_3] Cannot make job: ../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: Expression evaluation error:
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: Expecting value: line 1 column 1 (char 0)
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: script was:
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 99                 "class": "File",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 100                 "location": "file:///data/rnaseq/GenomeIndex/Plasmodium_Yoelii/STARIndex/sjdbInfo.txt",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 101                 "basename": "sjdbInfo.txt",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 102                 "size": 6,
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 103                 "nameroot": "sjdbInfo",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 104                 "nameext": ".txt",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 105                 "path": "/var/lib/cwl/stg409e96a3-4611-4f22-94ee-2c416a04fe60/STARIndex/sjdbInfo.txt",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 106                 "dirname": "/var/lib/cwl/stg409e96a3-4611-4f22-94ee-2c416a04fe60/STARIndex"
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 107             },
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 108             {
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 109                 "class": "File",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 110                 "location": "file:///data/rnaseq/GenomeIndex/Plasmodium_Yoelii/STARIndex/sjdbList.out.tab",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 111                 "basename": "sjdbList.out.tab",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 112                 "size": 0,
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 113                 "nameroot": "sjdbList.out",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 114                 "nameext": ".tab",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 115                 "path": "/var/lib/cwl/stg409e96a3-4611-4f22-94ee-2c416a04fe60/STARIndex/sjdbList.out.tab",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 116                 "dirname": "/var/lib/cwl/stg409e96a3-4611-4f22-94ee-2c416a04fe60/STARIndex"
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 117             },
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 118             {
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 119                 "class": "File",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 120                 "location": "file:///data/rnaseq/GenomeIndex/Plasmodium_Yoelii/STARIndex/genomeParameters.txt",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 121                 "basename": "genomeParameters.txt",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 122                 "size": 747,
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 123                 "nameroot": "genomeParameters",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 124                 "nameext": ".txt",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 125                 "path": "/var/lib/cwl/stg409e96a3-4611-4f22-94ee-2c416a04fe60/STARIndex/genomeParameters.txt",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 126                 "dirname": "/var/lib/cwl/stg409e96a3-4611-4f22-94ee-2c416a04fe60/STARIndex"
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 127             },
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 128             {
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 129                 "class": "File",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 130                 "location": "file:///data/rnaseq/GenomeIndex/Plasmodium_Yoelii/STARIndex/Genome",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 131                 "basename": "Genome",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 132                 "size": 69730304,
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 133                 "nameroot": "Genome",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 134                 "nameext": "",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 135                 "path": "/var/lib/cwl/stg409e96a3-4611-4f22-94ee-2c416a04fe60/STARIndex/Genome",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 136                 "dirname": "/var/lib/cwl/stg409e96a3-4611-4f22-94ee-2c416a04fe60/STARIndex"
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 137             },
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 138             {
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 139                 "class": "File",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 140                 "location": "file:///data/rnaseq/GenomeIndex/Plasmodium_Yoelii/STARIndex/SA",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 141                 "basename": "SA",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 142                 "size": 176797792,
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 143                 "nameroot": "SA",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 144                 "nameext": "",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 145                 "path": "/var/lib/cwl/stg409e96a3-4611-4f22-94ee-2c416a04fe60/STARIndex/SA",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 146                 "dirname": "/var/lib/cwl/stg409e96a3-4611-4f22-94ee-2c416a04fe60/STARIndex"
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 147             },
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 148             {
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 149                 "class": "File",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 150                 "location": "file:///data/rnaseq/GenomeIndex/Plasmodium_Yoelii/STARIndex/SAindex",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 151                 "basename": "SAindex",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 152                 "size": 1565873619,
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 153                 "nameroot": "SAindex",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 154                 "nameext": "",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 155                 "path": "/var/lib/cwl/stg409e96a3-4611-4f22-94ee-2c416a04fe60/STARIndex/SAindex",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 156                 "dirname": "/var/lib/cwl/stg409e96a3-4611-4f22-94ee-2c416a04fe60/STARIndex"
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 157             }
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 158         ],
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 159         "path": "/var/lib/cwl/stg409e96a3-4611-4f22-94ee-2c416a04fe60/STARIndex",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 160         "dirname": "/var/lib/cwl/stg409e96a3-4611-4f22-94ee-2c416a04fe60"
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 161     },
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 162     "outFileNamePrefix": "test3",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 163     "readFilesIn": [
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 164         {
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 165             "class": "File",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 166             "location": "file:///data/rnaseq/test/plas/SRR3584110_1.fastq",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 167             "size": 7215178831,
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 168             "basename": "SRR3584110_1.fastq",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 169             "nameroot": "SRR3584110_1",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 170             "nameext": ".fastq",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 171             "path": "/var/lib/cwl/stg4c4fe571-d618-4219-a484-6604bb72267b/SRR3584110_1.fastq",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 172             "dirname": "/var/lib/cwl/stg4c4fe571-d618-4219-a484-6604bb72267b"
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 173         },
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 174         {
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 175             "class": "File",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 176             "location": "file:///data/rnaseq/test/plas/SRR3584110_2.fastq",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 177             "size": 7215178831,
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 178             "basename": "SRR3584110_2.fastq",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 179             "nameroot": "SRR3584110_2",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 180             "nameext": ".fastq",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 181             "path": "/var/lib/cwl/stg833312ef-c155-4531-b43e-ce7ef1cc4514/SRR3584110_2.fastq",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 182             "dirname": "/var/lib/cwl/stg833312ef-c155-4531-b43e-ce7ef1cc4514"
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 183         }
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 184     ],
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 185     "threads": 24,
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 186     "XSTag": "intronMotif"
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 187 };
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 188 var self = null;
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 189 var runtime = {
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 190     "cores": 1,
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 191     "ram": 1024,
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 192     "tmpdirSize": 1024,
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 193     "outdirSize": 1024,
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 194     "tmpdir": "/tmp",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 195     "outdir": "/var/spool/cwl"
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 196 };
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 197 (function(){return (("mkdir " + inputs.outFileNamePrefix + " && cd " + inputs.outFileNamePrefix + " && STAR "));})()
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: stdout was: ''
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: stderr was: ''
[workflow ] completed permanentFail
[provenance] added provenance: [<QualifiedName: provenance:primary.cwlprov.xml>, <QualifiedName: provenance:primary.cwlprov.provn>, <QualifiedName: provenance:primary.cwlprov.json>, <QualifiedName: provenance:primary.cwlprov.ttl>, <QualifiedName: provenance:primary.cwlprov.nt>, <QualifiedName: provenance:primary.cwlprov.jsonld>]
{
    "DESeq2_out": null,
    "prepDE_out": null,
    "samtools_out": null,
    "star_readmap_out": null,
    "stringtie_out": null
}
Final process status is permanentFail
[provenance] Finalizing Research Object
[provenance] Generated bagit metadata: /tmp/tmp7hnt9_ib
[provenance] Deleting existing /data/rnaseq/test/plas/prov
[provenance] Research Object saved to /data/rnaseq/test/plas/prov

@tonyyzy
Copy link
Owner Author

tonyyzy commented Feb 15, 2019

cwl provenance is a really good concept and matches our ideas (store a copy of cwl files for tools and workflows)
Tested with our test data, works fine. Also works with --parallel options.
HOWEVER, an error was encountered when testing on Plasmodium data, seems like a javascript error but shouldn't. Without --provenance tag it works find with plasmodium. Change subject_name doesn't work.

zy2414@node023:/data/rnaseq/test/plas$ cwl-runner --provenance=./prov --outdir=./workflow1_results2 ../../RNASeq/workflows/docker/star_samtools_stringtie_prepDE_DESeq2_four_inputs.cwl ./star_samtools_stringtie_prepDE_DESeq2.yml
/project/soft/linux64/anaconda/Anaconda3-2018.12-Linux-x86_64/bin/cwl-runner 1.0.20180820141117
[provenance] Temporary research object: /tmp/tmp7hnt9_ib
Resolved '../../RNASeq/workflows/docker/star_samtools_stringtie_prepDE_DESeq2_four_inputs.cwl' to 'file:///data/rnaseq/RNASeq/workflows/docker/star_samtools_stringtie_prepDE_DESeq2_four_inputs.cwl'
[provenance] Added packed workflow: workflow/packed.cwl
[provenance] Added data file /tmp/tmp7hnt9_ib/data/9b/9bb021d92a1bb7478d90ce7aed6cdec43e344bf8
[provenance] Relative path for data file data/9b/9bb021d92a1bb7478d90ce7aed6cdec43e344bf8
[provenance] Added data file /tmp/tmp7hnt9_ib/data/4e/4ea1d963d87e556fb381ce6cbd2acbc1f4c96966
[provenance] Relative path for data file data/4e/4ea1d963d87e556fb381ce6cbd2acbc1f4c96966
[provenance] Added data file /tmp/tmp7hnt9_ib/data/ef/eff2f2cd3bab93fa537d67ee80ebe593ac0f2d2e
[provenance] Relative path for data file data/ef/eff2f2cd3bab93fa537d67ee80ebe593ac0f2d2e
[provenance] Added data file /tmp/tmp7hnt9_ib/data/f5/f51250ebd5a05d93c38f9cb3adb163c9933caabe
[provenance] Relative path for data file data/f5/f51250ebd5a05d93c38f9cb3adb163c9933caabe
[provenance] Added data file /tmp/tmp7hnt9_ib/data/43/43554b7a5dbca1f9b21c071fb75dcc289c2ce93d
[provenance] Relative path for data file data/43/43554b7a5dbca1f9b21c071fb75dcc289c2ce93d
[provenance] Added data file /tmp/tmp7hnt9_ib/data/70/702ee12acad42938f62a77517b9b9e7cddb1ba16
[provenance] Relative path for data file data/70/702ee12acad42938f62a77517b9b9e7cddb1ba16
[provenance] Added data file /tmp/tmp7hnt9_ib/data/8f/8f417ab11490cb3352513fd26d1fcdc893f9832b
[provenance] Relative path for data file data/8f/8f417ab11490cb3352513fd26d1fcdc893f9832b
[provenance] Added data file /tmp/tmp7hnt9_ib/data/e3/e3dfed0d4ec3517bd44364b435ea94c2f8c692fd
[provenance] Relative path for data file data/e3/e3dfed0d4ec3517bd44364b435ea94c2f8c692fd
[provenance] Added data file /tmp/tmp7hnt9_ib/data/cd/cdf97ffaa124261fadb4d28f37518c7c71c1fab3
[provenance] Relative path for data file data/cd/cdf97ffaa124261fadb4d28f37518c7c71c1fab3
[provenance] Added data file /tmp/tmp7hnt9_ib/data/e9/e90892944e5d3bff5f7336bec92bd772d720d1a3
[provenance] Relative path for data file data/e9/e90892944e5d3bff5f7336bec92bd772d720d1a3
[provenance] Added data file /tmp/tmp7hnt9_ib/data/3e/3edd683d9737e637cfa225c84257bb31cd7e1367
[provenance] Relative path for data file data/3e/3edd683d9737e637cfa225c84257bb31cd7e1367
[provenance] Added data file /tmp/tmp7hnt9_ib/data/35/35c66bf776766acdfb30d5930f8137c99e91113c
[provenance] Relative path for data file data/35/35c66bf776766acdfb30d5930f8137c99e91113c
[provenance] Generated customised job file: workflow/primary-job.json
[provenance] Added data file /tmp/tmp7hnt9_ib/data/b4/b444ac06613fc8d63795be9ad0beaf55011936ac
[provenance] Relative path for data file data/b4/b444ac06613fc8d63795be9ad0beaf55011936ac
[provenance] Added data file /tmp/tmp7hnt9_ib/data/10/109f4b3c50d7b0df729d299bc6f8e9ef9066971f
[provenance] Relative path for data file data/10/109f4b3c50d7b0df729d299bc6f8e9ef9066971f
[provenance] Added data file /tmp/tmp7hnt9_ib/data/3e/3ebfa301dc59196f18593c45e519287a23297589
[provenance] Relative path for data file data/3e/3ebfa301dc59196f18593c45e519287a23297589
[provenance] Added data file /tmp/tmp7hnt9_ib/data/1f/1ff2b3704aede04eecb51e50ca698efd50a1379b
[provenance] Relative path for data file data/1f/1ff2b3704aede04eecb51e50ca698efd50a1379b
[workflow ] start
[workflow ] starting step star_readmap_3
[step star_readmap_3] start
Exception on step 'star_readmap_3'
[step star_readmap_3] Cannot make job: ../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: Expression evaluation error:
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: Expecting value: line 1 column 1 (char 0)
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: script was:
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 99                 "class": "File",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 100                 "location": "file:///data/rnaseq/GenomeIndex/Plasmodium_Yoelii/STARIndex/sjdbInfo.txt",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 101                 "basename": "sjdbInfo.txt",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 102                 "size": 6,
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 103                 "nameroot": "sjdbInfo",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 104                 "nameext": ".txt",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 105                 "path": "/var/lib/cwl/stg409e96a3-4611-4f22-94ee-2c416a04fe60/STARIndex/sjdbInfo.txt",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 106                 "dirname": "/var/lib/cwl/stg409e96a3-4611-4f22-94ee-2c416a04fe60/STARIndex"
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 107             },
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 108             {
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 109                 "class": "File",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 110                 "location": "file:///data/rnaseq/GenomeIndex/Plasmodium_Yoelii/STARIndex/sjdbList.out.tab",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 111                 "basename": "sjdbList.out.tab",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 112                 "size": 0,
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 113                 "nameroot": "sjdbList.out",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 114                 "nameext": ".tab",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 115                 "path": "/var/lib/cwl/stg409e96a3-4611-4f22-94ee-2c416a04fe60/STARIndex/sjdbList.out.tab",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 116                 "dirname": "/var/lib/cwl/stg409e96a3-4611-4f22-94ee-2c416a04fe60/STARIndex"
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 117             },
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 118             {
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 119                 "class": "File",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 120                 "location": "file:///data/rnaseq/GenomeIndex/Plasmodium_Yoelii/STARIndex/genomeParameters.txt",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 121                 "basename": "genomeParameters.txt",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 122                 "size": 747,
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 123                 "nameroot": "genomeParameters",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 124                 "nameext": ".txt",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 125                 "path": "/var/lib/cwl/stg409e96a3-4611-4f22-94ee-2c416a04fe60/STARIndex/genomeParameters.txt",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 126                 "dirname": "/var/lib/cwl/stg409e96a3-4611-4f22-94ee-2c416a04fe60/STARIndex"
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 127             },
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 128             {
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 129                 "class": "File",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 130                 "location": "file:///data/rnaseq/GenomeIndex/Plasmodium_Yoelii/STARIndex/Genome",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 131                 "basename": "Genome",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 132                 "size": 69730304,
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 133                 "nameroot": "Genome",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 134                 "nameext": "",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 135                 "path": "/var/lib/cwl/stg409e96a3-4611-4f22-94ee-2c416a04fe60/STARIndex/Genome",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 136                 "dirname": "/var/lib/cwl/stg409e96a3-4611-4f22-94ee-2c416a04fe60/STARIndex"
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 137             },
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 138             {
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 139                 "class": "File",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 140                 "location": "file:///data/rnaseq/GenomeIndex/Plasmodium_Yoelii/STARIndex/SA",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 141                 "basename": "SA",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 142                 "size": 176797792,
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 143                 "nameroot": "SA",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 144                 "nameext": "",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 145                 "path": "/var/lib/cwl/stg409e96a3-4611-4f22-94ee-2c416a04fe60/STARIndex/SA",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 146                 "dirname": "/var/lib/cwl/stg409e96a3-4611-4f22-94ee-2c416a04fe60/STARIndex"
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 147             },
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 148             {
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 149                 "class": "File",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 150                 "location": "file:///data/rnaseq/GenomeIndex/Plasmodium_Yoelii/STARIndex/SAindex",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 151                 "basename": "SAindex",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 152                 "size": 1565873619,
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 153                 "nameroot": "SAindex",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 154                 "nameext": "",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 155                 "path": "/var/lib/cwl/stg409e96a3-4611-4f22-94ee-2c416a04fe60/STARIndex/SAindex",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 156                 "dirname": "/var/lib/cwl/stg409e96a3-4611-4f22-94ee-2c416a04fe60/STARIndex"
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 157             }
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 158         ],
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 159         "path": "/var/lib/cwl/stg409e96a3-4611-4f22-94ee-2c416a04fe60/STARIndex",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 160         "dirname": "/var/lib/cwl/stg409e96a3-4611-4f22-94ee-2c416a04fe60"
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 161     },
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 162     "outFileNamePrefix": "test3",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 163     "readFilesIn": [
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 164         {
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 165             "class": "File",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 166             "location": "file:///data/rnaseq/test/plas/SRR3584110_1.fastq",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 167             "size": 7215178831,
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 168             "basename": "SRR3584110_1.fastq",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 169             "nameroot": "SRR3584110_1",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 170             "nameext": ".fastq",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 171             "path": "/var/lib/cwl/stg4c4fe571-d618-4219-a484-6604bb72267b/SRR3584110_1.fastq",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 172             "dirname": "/var/lib/cwl/stg4c4fe571-d618-4219-a484-6604bb72267b"
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 173         },
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 174         {
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 175             "class": "File",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 176             "location": "file:///data/rnaseq/test/plas/SRR3584110_2.fastq",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 177             "size": 7215178831,
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 178             "basename": "SRR3584110_2.fastq",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 179             "nameroot": "SRR3584110_2",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 180             "nameext": ".fastq",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 181             "path": "/var/lib/cwl/stg833312ef-c155-4531-b43e-ce7ef1cc4514/SRR3584110_2.fastq",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 182             "dirname": "/var/lib/cwl/stg833312ef-c155-4531-b43e-ce7ef1cc4514"
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 183         }
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 184     ],
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 185     "threads": 24,
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 186     "XSTag": "intronMotif"
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 187 };
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 188 var self = null;
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 189 var runtime = {
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 190     "cores": 1,
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 191     "ram": 1024,
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 192     "tmpdirSize": 1024,
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 193     "outdirSize": 1024,
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 194     "tmpdir": "/tmp",
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 195     "outdir": "/var/spool/cwl"
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 196 };
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: 197 (function(){return (("mkdir " + inputs.outFileNamePrefix + " && cd " + inputs.outFileNamePrefix + " && STAR "));})()
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: stdout was: ''
../../RNASeq/cwl-tools/docker/STAR_readmap.cwl:12:5: stderr was: ''
[workflow ] completed permanentFail
[provenance] added provenance: [<QualifiedName: provenance:primary.cwlprov.xml>, <QualifiedName: provenance:primary.cwlprov.provn>, <QualifiedName: provenance:primary.cwlprov.json>, <QualifiedName: provenance:primary.cwlprov.ttl>, <QualifiedName: provenance:primary.cwlprov.nt>, <QualifiedName: provenance:primary.cwlprov.jsonld>]
{
    "DESeq2_out": null,
    "prepDE_out": null,
    "samtools_out": null,
    "star_readmap_out": null,
    "stringtie_out": null
}
Final process status is permanentFail
[provenance] Finalizing Research Object
[provenance] Generated bagit metadata: /tmp/tmp7hnt9_ib
[provenance] Deleting existing /data/rnaseq/test/plas/prov
[provenance] Research Object saved to /data/rnaseq/test/plas/prov

Ok, this issue is to do with running jobs in parallel..node.js times out frequently

@tonyyzy
Copy link
Owner Author

tonyyzy commented Feb 15, 2019

With provenance, a research object can be generated but what can we get out of it? other than docker images id, workflow logs, packed workflow and snapshots of cwl tools

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Good to have Non-essential but good to have
Projects
None yet
Milestone
End of Project
Development

No branches or pull requests
1 participant
@tonyyzy