Implement a removing worker

[derpeter]

As we always fail to clean up when an event is done we need a worker that services the "removing" state of the tracker. This should be a second worker running on the same host as the publishing worker and and also share some code.

• Service removing state
• Make sure to only remove master encoding files when all subformats are released
  • We may need / want tracker support for this
• Clean up encoded / tmp / capture / on the releasing host
• Clean up on the encoder machines / storage is out of scope for now

@a-tze @jjeising RFC

[jjeising]

Seems like a good idea! On the second thought, I see some open questions:

Clean up encoded / tmp / capture

On the releasing host encoded could be cleaned after a successful upload. tmp should probably be kept clean by the crs-scripts, and when it is cleaned, it should be scrubbed on a per project basis, not by individual ticket. Otherwise there would have to be a naming convention (e.g. folder with ticket id).

As far as cleaning capture I have major doubts this can be done on a per ticket basis. Even if we would identify the used segments these could also be used by other tickets and segments without usage would still remain.

I think this would be far more practicable (and less complicated) as a script triggered by a webhook on a per project basis. This could be triggered when archiving a project. Although I see this would be an issue for long running projects (for periodicals).

[derpeter]

All folders mentioned are on the releasing host. For $reasons we have a capture dir there to.
I'm not sure why but tmp is not cleaned afterwards but it isnt.

But we could start with only cleaning /encoded/ and see where we go from there.
The main question is, how to figure when all subtickets are done.

Another option would be of course to enable the worker only when the trackerfahrer is sure ....

[a-tze]

Not cleaning up the tmp folder is intentional. If only metadata changes, files from tmp will be tagged again and end up in encoded dir.
This may be implemented in another way but I'm not going to touch this to save some minutes of manual work per month.

[saerdnaer]

Related Issue: #22