bug: fluentbit.io/parser annotation not working properly

[alternaivan]

Describe the issue

We have a fluent-operator installed on the cluster in the fluent namespace, with namespace-provided configuration (filters, parsers, and output).

We have been using the kubernetes filter with the k8sLoggingParser: true option set in the default namespace, so we can specify a custom parser on the resource level, in the pod annotation, as described here.

We configured and applied the custom parsers with the namespaced Parser CRD on the default namespace.

When specifying the annotation on the pod level, the log field (which was not parsed via Kubernetes filter), doesn't get parsed by the parser specified via annotation. I've also tried using the parser name and the parser name with hashed namespaced suffix in the end. Both of those options didn't work.

For comparisson, I've also enabled the k8sLoggingExclude option, and the exclusion annotation. This seems to work as expected.

To Reproduce

Create a namespaced Parser and add an annotation on the pod level that specifies that parser.

Expected behavior

Log field is properly parsed via the custom parser specified in the annotation.

Your Environment

- Fluent Operator version: 3.2.0

How did you install fluent operator?

Via helm.

Additional context

No response

[cw-Guo]

just to confirm, do you turned on the merge_log option for kubernetes filter?

To perform processing of the log key, it's mandatory to enable the Merge_Log configuration property in this filter, then the following processing order will be done

see https://docs.fluentbit.io/manual/pipeline/filters/kubernetes#processing-the-log-value

[alternaivan]

Hi @cw-Guo,

just to confirm, do you turned on the merge_log option for kubernetes filter?

Yes.