Implement log rotation

[garethbowen]

We need to implement some sort of log rotation, compression, and/or archival for all logs on the medic-os Docker images. This needs to be done for all logs (api, sentinel, couchdb, etc) to ensure we don't run out of space.

This needs to be done with thought to maintain ease of use, for example, the current log must always have the same name and recent logs (~ last 24 hours) must be easy to access (not zipped or backed up off server).

NB: alpha.dev medic-api log is currently 174MB.

cc @browndav @henokgetachew

[ghost]

This should be currently performed by logrotate(1), and the naming issues should be gone now that gardener is gone. I suspect it might just not be switched on for the services that were migrated out of gardener due to an oversight; I'll take a look now and push a PR.

Do you have any strong preference on retention/turnover parameters other than the 24 hour turnover (e.g. additional size-based turnover)? We should be able to set these in one place (/etc/logrotate.conf) container-wide.

[garethbowen]

It'd be good to have a size-based turnover as well just in case something goes wrong and blows out the logs in a hurry. Ideally this would be set very high so it's only used as a safety net and logs are otherwise in 24 hours slices.

[ghost]

In the future this can certainly be more DRY, but this gets the job done and we can move/consolidate/link the small duplicated per-package scripts at a later point.

[ghost]

The DRY-ness issue is filed at medic/medic-os#10 and should be straightforward to do all at once.

[henokgetachew]

LGTM. Also +1 for filing the DRY-ness issue.

[ngaruko]

Tested on alpha.dev. medic-sentinel log size looks pretty small (140kb), but medic-api log size looks a bit larger but definitely way smaller than the 174 MB.

/srv/storage/medic-api/logs# du -sh 26M

/srv/storage/medic-sentinel/logs# du -sh 140K

[garethbowen]

@ngaruko The important thing isn't the size of the logs but that the logs split when they get big/old enough. You should see more than a single log file once the limit is hit.

[ngaruko]

Thanks @garethbowen. Do we know what that limit is for both sentinel and api?

[garethbowen]

@henokgetachew @browndav What's the expected rotation strategy?

[ghost]

The rotation strategy is currently still logrotate, and the base configuration can be found here.

The missing log rotation for Horticulturalist-managed services is fixed in git and in the Docker 3.0.0-pre.01 on ECR, but unfortunately 3.0.0-pre.01 picked up a version of Horticulturalist that had bugs. As a result, the 3.0.0-pre.01 image isn't incredibly useful as-is.

We're building a 3.0.0-pre.02 with the latest version of Horticulturalist tomorrow morning, and should have it on ECR before end of day tomorrow.

When it's pushed to ECR and tested, we'll comment here and update the instructions in medic-infrastructure so that everyone's able to pull the new image and launch it on Docker CE (locally) or Docker Machine (on AWS).