logs both in Grafana and through Jenkins config - breaks displayed logs (colors?)

[ArieLevs]

Jenkins and plugins versions report

Environment

plugin version 3.1320.v2eededb_d909e

What Operating System are you using (both controller, and any agents involved in the problem)?

Jenkins 2.452.2
Jenkins in kubernetes, default agent is jenkins/inbound-agent:jdk17

Reproduction steps

1. follow Storing Jenkins Pipeline Logs in Loki
2. configure the Store pipeline logs In Loki and visualize logs both in Grafana and through Jenkins option
3. logs are shown in Jenkins UI and in Grafana, which is OK
4. logs are missing colors, links are not clickable (from Jenkins UI), seems that anything that has special characters no longer appears in logs

Expected Results

logs should keep appearing as usual both in Grafana and Jenkins UI

Actual Results

logs are missing colors, links are not clickable (from Jenkins UI), seems that anything that has special characters no longer appears in logs

for example, printed error from tools like tflint, flake8, etc no longer appear to the user

Anything else?

No response

Are you interested in contributing a fix?

No response

[cyrille-leclerc]

We probably know the limitation but I would like to double check please: can you provide a description of the color highlighting and hyperlinks that you are looking for, ideally with an annotated screenshot of pipeline logs with the formatting you are looking for.

The cause is likely to be that Elasticsearch, the first OTel logs backend we supported, didn't support ANSI color invisible chars so we had to extract the rich text formatting annotations from the plain text.

See :

• https://github.com/jenkinsci/opentelemetry-plugin/blob/3.1320.v2eededb_d909e/src/main/java/io/jenkins/plugins/opentelemetry/job/log/OtelLogOutputStream.java#L44

[ArieLevs]

Hi @cyrille-leclerc sure,

here is a screenshot using regular console logs with no otel integration

then, when enabling Store pipeline logs In Loki and visualize logs both in Grafana and through Jenkins logs look like:

• please note - possible another bug?, after setting this option I only see logs in Grafana and not in UI 😕
  see this log line

  Caught exception evaluating: it.writeLogTo(offset,output) in /job/xxxx/job/xxxx/view/change-requests/job/PR-148/4/console. Reason: com.jayway.jsonpath.PathNotFoundException: Expected to find an object with property ['data'] in path $ but found 'java.lang.String'. This is not a json object according to the JsonProvider: 'com.jayway.jsonpath.spi.json.JsonSmartJsonProvider'.
  

logs in Grafana are (were exactly the same in UI)

[cyrille-leclerc]

Update: dealing with logs formatting in logs backends that support ANSI color encoding is more complex than anticipated because Jenkins log formatting is no ANSI color encoding or is more than just ANSI color encoding.
I'll continue to explore this.

@ArieLevs can you please open a bug about the issue

Caught exception evaluating: it.writeLogTo(offset,output) in /job/xxxx/job/xxxx/view/change-requests/job/PR-148/4/console. Reason: com.jayway.jsonpath.PathNotFoundException: Expected to find an object with property ['data'] in path $ but found 'java.lang.String'. This is not a json object according to the JsonProvider: 'com.jayway.jsonpath.spi.json.JsonSmartJsonProvider'.

It seems to be specific to your Loki version. Do you use self managed Loki? Which version please?

[kuisathaverat]

It was a designed decision to store logs without the format flags; if you want to keep that format in the Jenkins console view, you can use the mirror option. In this way you send the logs to Loky and keep a local copy in the original way so the format is not lost.

[ArieLevs]

Update: dealing with logs formatting in logs backends that support ANSI color encoding is more complex than anticipated because Jenkins log formatting is no ANSI color encoding or is more than just ANSI color encoding. I'll continue to explore this.

@ArieLevs can you please open a bug about the issue

Caught exception evaluating: it.writeLogTo(offset,output) in /job/xxxx/job/xxxx/view/change-requests/job/PR-148/4/console. Reason: com.jayway.jsonpath.PathNotFoundException: Expected to find an object with property ['data'] in path $ but found 'java.lang.String'. This is not a json object according to the JsonProvider: 'com.jayway.jsonpath.spi.json.JsonSmartJsonProvider'.

It seems to be specific to your Loki version. Do you use self managed Loki? Which version please?

Done #946
Yes using self managed Loki 3.0.0 (k8s)

[ArieLevs]

It was a designed decision to store logs without the format flags; if you want to keep that format in the Jenkins console view, you can use the mirror option. In this way you send the logs to Loky and keep a local copy in the original way so the format is not lost.

this actually worked very well, the only thing is I can no long see a "link" to Grafana, similar to how its used with Store pipeline logs In Loki and visualize logs both in Grafana and through Jenkins.
to get the desired logs, I had to randomly search a string that appeared in my console logs, then get the trace_id and only then filter by that.

I'll mark issue as closed as all is actually working as expected, it just feels like end uses of Jenkins will never use this feature, as they don't have a direct link from Jenkins <-> Grafana this way. (but this is still good enough for our infra team to investigate Jenkins and get insights)