Checking Logs

Openshift Logs

Possible scenario: An error alert comes in from our CHES application showing:

CHES (9f0fbe-prod) Error Alert - ches - HTTP response status code 500, hostname: ches-app-master-67-x2tsp, namespace: 9f0fbe-prod, agent: Java/11.0.8, method: POST, path: /api/v1/email

To investigate the cause of this message we can check logs on OpenShift. Logging process:

• CHES node application container ('app') on pod ches-app-master-67-x2tsp logged a record showing the error.
• The log was also forwarded to the Platform-level Elasticsearch index and is visible on the Kibana dashboard linked to the OpenSHift console
• The original log record was also sent to the logging sidecar, which itself forwarded the log to our error alert service (Fluentd)

The first place to get more info about the error could be to open the platform-level Kibana dashboard.

• on OpenShift console, go to the pod (Applications > Pods > ches-app-master-67-x2tsp)
• click on 'Logs` tab
• click on the 'view archive' link on right hand side of screen (to open Kibana dashbaord)
• in Kibana dsashboard you need to click on 'Discover' from the left hand nav.
• you will want to add some filters. click 'add filter' link below query bar. for example: message is not kube and 'kubernetes.container_name' is app
• You can narrow down the time-frame by clicking the date range by clicking the time frame shown in the top right hand corner above the query bar. We know the error happened around 12:00 am on Sep 6th.