Save a JSON status file

[jywarren]

In the Ruby version, we output a status.json file to track status, with the following parameters:

https://github.com/publiclab/mapknitter-exporter-sinatra/blob/b065d1de02730347255d1df4c55c9368ca53ec60/app.rb#L119-L137

Perhaps we should show status in a similar way here!

image-sequencer-app/src/api/v2/index.js

Lines 73 to 82 in 9afad91

	if (upload) {
	fs.writeFileSync(path.join(__dirname, `../../../temp/export${pid}.html`), html);
	mapknitterBucket.upload(path.join(__dirname, `../../../temp/export${pid}.html`), {
	gzip: true
	}).then(() => {
	mapknitterBucket
	.file(`export${pid}.html`)
	.makePublic();
	fs.unlinkSync(path.join(__dirname, `../../../temp/export${pid}.html`));
	});

[tech4GT]

Hi @jywarren Sorry for the absence, I was on vacation! 😃
I am starting this now! I can't seem to find the other thread where we discussed this, so just wanted to check in with you!
This file goes in the same bucket as the final export right?

[jywarren]

That's right, it does!

…

On Fri, Jan 10, 2020 at 7:34 AM Varun Gupta ***@***.***> wrote: Hi @jywarren <https://github.com/jywarren> Sorry for the absence, I was on vacation! 😃 I am starting this now! I can't seem to find the other thread where we discussed this, so just wanted to check in with you! This file goes in the same bucket as the final export right? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#21?email_source=notifications&email_token=AAAF6J6XORKUDYAKFBYNKKTQ5BSZNA5CNFSM4H7JOYOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEITYV7I#issuecomment-573016829>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAF6J2GL4MOKZGPF4CXZ6LQ5BSZNANCNFSM4H7JOYOA> .

[tech4GT]

@jywarren I have been thinking about the sever crashing on big exports, do you think it is possible that the server does not have the disk storage to store the export before uploading? I mean once the export is done we store the export as a file and this file is then uploaded to the cloud, following which we delete the file stored locally. Is it possible that the server does not have the disk storage to store a very large file? I mean we have been focussed on RAM all this time, if the disk space turns out to be the issue it would be kinda funny! :D

[tech4GT]

Alright! I have the basic status json mechanism in place! All the fields are not yet being filled but it works for the export!

[jywarren]

Amazing! My integration work at mapknitter has also been progressing so this is great. There's a new example file in here too: publiclab/Leaflet.DistortableImage#485 (Though it needs the images to be online so it doesn't work locally) The disk space issue seems plausible. How can we check or test this?

…

On Sat, Jan 11, 2020, 3:53 AM Varun Gupta ***@***.***> wrote: Alright! I have the basic status json mechanism in place! All the fields are not yet being filled but it works for the export! [image: Screen Shot 2020-01-11 at 2 15 38 PM] <https://user-images.githubusercontent.com/25617855/72201746-369c7100-347d-11ea-9011-6e2a03feb3f4.png> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#21?email_source=notifications&email_token=AAAF6J77PIWFHOOT63LA453Q5GBV3A5CNFSM4H7JOYOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIV5AQQ#issuecomment-573296706>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAF6J7I7RIAT763AYRTYVDQ5GBV3ANCNFSM4H7JOYOA> .

[tech4GT]

Awesome! I'll try to research a bit about the disk space issue, or worst case we write code to store the output of all the steps in the bucket and then delete them as the compositing process is completed.

[jywarren]

Super! Maybe there's a way to catch and handle errors at the container level?

…

On Sat, Jan 11, 2020, 11:13 AM Varun Gupta ***@***.***> wrote: Awesome! I'll try to research a bit about the disk space issue, or worst case we write code to store the output of all the steps in the bucket and then delete them as the compositing process is completed. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#21?email_source=notifications&email_token=AAAF6J4UWIQ32VKAX5FOJG3Q5HV3XA5CNFSM4H7JOYOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIWFGXY#issuecomment-573330271>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAF6J57JO3TALK44OHFBPLQ5HV3XANCNFSM4H7JOYOA> .

[tech4GT]

Hmm, there should be. I'm not really sure at this moment. Let me get back to you with a little more research!

[tech4GT]

Oh! I just noticed something! The server restart occurs in the warping phase itself. So the disk issue should not be at play here. The status json has 4 phases: starting, warping, compositing, uploading, uploaded. The export which failed is stuck at warping.