logs do not displayed in UI when Loki and visualize logs both in Grafana and through Jenkins

[ArieLevs]

Jenkins and plugins versions report

Environment

plugin version 3.1320.v2eededb_d909e

What Operating System are you using (both controller, and any agents involved in the problem)?

Jenkins 2.452.2
Jenkins in kubernetes, default agent is jenkins/inbound-agent:jdk17

Reproduction steps

1. configure Jenkins otel plugin to Store pipeline logs In Loki and visualize logs both in Grafana and through Jenkins
2. then configure Jenkins otel plugin to Don't store pipeline logs in Loki
3. configure again for Store pipeline logs In Loki and visualize logs both in Grafana and through Jenkins

Expected Results

logs should be visible in Grafana + Loki again

Actual Results

logs are displayed in Grafana only, UI console logs no longer visible with error

Caught exception evaluating: it.writeLogTo(offset,output) in /job/xxxx/job/xxxx/view/change-requests/job/PR-148/4/console. 
Reason: com.jayway.jsonpath.PathNotFoundException: 
Expected to find an object with property ['data'] in path $ 
but found 'java.lang.String'. 

This is not a json object according to the JsonProvider: 'com.jayway.jsonpath.spi.json.JsonSmartJsonProvider'.

Anything else?

No response

Are you interested in contributing a fix?

No response

[cyrille-leclerc]

@ArieLevs Please give details on the Loki instance you are accessing (version...)

[ArieLevs]

@cyrille-leclerc sorry for the delay, Loki is a V3 version, installed to k8s using loki helm chart from https://grafana.github.io/helm-charts chart version is 6.6.5
app version is 3.0.0, deployment mode is Distributed