[BUG] Dashboard fails with error 502 Bad Gateway when using OIDC-based authn with Keycloak

[mcaliandro]

What is the bug?
The Dashboard application crashes with error 502 Bad Gateway when an user authenticated & authorized by Keycloak tries to access Opensearch.

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Go to Dashboard URL, i.e., https://opensearch.example.com/
2. Click on 'Log in with single sign-on'
3. Wait a couple of seconds
4. Error 502 Bad Gateway is shown in a plain HTML page

What is the expected behavior?
An authenticated & authorized user should see the application's homepage, and navigate through the Dashboards' menus and sections.

What is your host/environment?

• Kubernetes v1.25
• Opensearch K8s Operator v2.4.0
• Opensearch v2.7.0
• Opensearch Dashboards v2.7.0

Do you have any screenshots?

Do you have any additional context?
Logs from Dashboard pod

{"type":"log","@timestamp":"2023-08-23T15:22:21Z","tags":["info","plugins","securityDashboards"],"pid":453,"message":"Error: unexpected end of file\n    at Zlib.zlibOnError [as onerror] (zlib.js:187:17)\n    at processChunkSync (zlib.js:451:12)\n    at zlibBufferSync (zlib.js:175:12)\n    at Object.inflateSync (zlib.js:786:14)\n    at inflateValue (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/utils/compression.ts:25:43)\n    at getExtraAuthStorageValue (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/session/cookie_splitter.ts:48:15)\n    at OpenIdAuthentication.getExtraAuthStorageValue (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/openid/openid_auth.ts:293:20)\n    at OpenIdAuthentication.buildAuthHeaderFromCookie (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/openid/openid_auth.ts:308:44)\n    at MultipleAuthentication.buildAuthHeaderFromCookie (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/multiple/multi_auth.ts:181:50)\n    at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:163:42\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:95:5)\n    at Object.interceptAuth [as authenticate] (/usr/share/opensearch-dashboards/src/core/server/http/lifecycle/auth.js:112:22)\n    at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n    at module.exports.internals.Auth._authenticate (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/auth.js:273:30)\n    at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32) {\n  errno: -5,\n  code: 'Z_BUF_ERROR'\n}"}
{"type":"log","@timestamp":"2023-08-23T15:22:21Z","tags":["debug","opensearch","opendistro_security","query"],"pid":453,"message":"401\nGET /_plugins/_security/authinfo\n"}
{"type":"log","@timestamp":"2023-08-23T15:22:21Z","tags":["error","plugins","securityDashboards"],"pid":453,"message":"Failed to resolve user tenant: Error: Failed authentication: Authentication Exception"}
{"type":"response","@timestamp":"2023-08-23T15:22:20Z","tags":[],"pid":453,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"host":"<opensearch-dashboard-url>","x-request-id":"31e8b2144848a84d81c45b3228a9854f","x-real-ip":"<client-ipaddr>","x-forwarded-for":"<client-ipaddr>1","x-forwarded-host":"<opensearch-dashboard-url>","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-scheme":"https","x-scheme":"https","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","dnt":"1","upgrade-insecure-requests":"1","sec-fetch-dest":"document","sec-fetch-mode":"navigate","sec-fetch-site":"none","sec-fetch-user":"?1"},"remoteAddress":"10.42.2.219","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0"},"res":{"statusCode":302,"responseTime":951,"contentLength":9},"message":"GET / 302 951ms - 9.0B"}
{"type":"response","@timestamp":"2023-08-23T15:22:21Z","tags":[],"pid":453,"method":"get","statusCode":200,"req":{"url":"/app/login","method":"get","headers":{"host":"<opensearch-dashboard-url>","x-request-id":"a04ce13246ed28ce94e457fd15825111","x-real-ip":"<client-ipaddr>","x-forwarded-for":"<client-ipaddr>","x-forwarded-host":"<opensearch-dashboard-url>","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-scheme":"https","x-scheme":"https","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","dnt":"1","upgrade-insecure-requests":"1","sec-fetch-dest":"document","sec-fetch-mode":"navigate","sec-fetch-site":"none","sec-fetch-user":"?1"},"remoteAddress":"10.42.2.219","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0"},"res":{"statusCode":200,"responseTime":96,"contentLength":9},"message":"GET /app/login 200 96ms - 9.0B"}
{"type":"log","@timestamp":"2023-08-23T15:22:21Z","tags":["debug","http","server","OpenSearchDashboards","cookie-session-storage"],"pid":453,"message":"Error: Unauthorized"}
{"type":"log","@timestamp":"2023-08-23T15:22:27Z","tags":["debug","http","server","OpenSearchDashboards","cookie-session-storage"],"pid":453,"message":"Error: Unauthorized"}
{"type":"response","@timestamp":"2023-08-23T15:22:27Z","tags":[],"pid":453,"method":"get","statusCode":401,"req":{"url":"/api/v1/restapiinfo","method":"get","headers":{"host":"<opensearch-dashboard-url>","x-request-id":"1303f354be8246a84e57cb2c9da8c5ea","x-real-ip":"<client-ipaddr>","x-forwarded-for":"<client-ipaddr>","x-forwarded-host":"<opensearch-dashboard-url>","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-scheme":"https","x-scheme":"https","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","referer":"https://<opensearch-dashboard-url>/app/login?","content-type":"application/json","osd-version":"2.7.0","osd-xsrf":"osd-fetch","dnt":"1","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin"},"remoteAddress":"10.42.2.219","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","referer":"https://<opensearch-dashboard-url>/app/login?"},"res":{"statusCode":401,"responseTime":7,"contentLength":9},"message":"GET /api/v1/restapiinfo 401 7ms - 9.0B"}
{"type":"log","@timestamp":"2023-08-23T15:22:27Z","tags":["debug","http","server","OpenSearchDashboards","cookie-session-storage"],"pid":453,"message":"Error: Unauthorized"}
{"type":"response","@timestamp":"2023-08-23T15:22:27Z","tags":[],"pid":453,"method":"get","statusCode":401,"req":{"url":"/api/v1/configuration/account","method":"get","headers":{"host":"<opensearch-dashboard-url>","x-request-id":"74ba3c6c69e4635edef572e43286a0dc","x-real-ip":"<client-ipaddr>","x-forwarded-for":"<client-ipaddr>","x-forwarded-host":"<opensearch-dashboard-url>","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-scheme":"https","x-scheme":"https","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","referer":"https://<opensearch-dashboard-url>/app/login?","content-type":"application/json","osd-version":"2.7.0","osd-xsrf":"osd-fetch","dnt":"1","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin"},"remoteAddress":"10.42.2.219","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","referer":"https://<opensearch-dashboard-url>/app/login?"},"res":{"statusCode":401,"responseTime":11,"contentLength":9},"message":"GET /api/v1/configuration/account 401 11ms - 9.0B"}
{"type":"response","@timestamp":"2023-08-23T15:22:27Z","tags":[],"pid":453,"method":"get","statusCode":200,"req":{"url":"/ui/fonts/inter_ui/Inter-UI-Regular.woff2","method":"get","headers":{"host":"<opensearch-dashboard-url>","x-request-id":"02474602e978893c4c1cb9ba4757279f","x-real-ip":"<client-ipaddr>","x-forwarded-for":"<client-ipaddr>","x-forwarded-host":"<opensearch-dashboard-url>","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-scheme":"https","x-scheme":"https","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","accept":"application/font-woff2;q=1.0,application/font-woff;q=0.9,*/*;q=0.8","accept-language":"en-US,en;q=0.5","accept-encoding":"identity","dnt":"1","referer":"https://<opensearch-dashboard-url>/app/login?","sec-fetch-dest":"font","sec-fetch-mode":"cors","sec-fetch-site":"same-origin"},"remoteAddress":"10.42.2.219","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","referer":"https://<opensearch-dashboard-url>/app/login?"},"res":{"statusCode":200,"responseTime":10,"contentLength":9},"message":"GET /ui/fonts/inter_ui/Inter-UI-Regular.woff2 200 10ms - 9.0B"}
{"type":"log","@timestamp":"2023-08-23T15:22:27Z","tags":["debug","http","server","OpenSearchDashboards","cookie-session-storage"],"pid":453,"message":"Error: Unauthorized"}
{"type":"response","@timestamp":"2023-08-23T15:22:27Z","tags":[],"pid":453,"method":"get","statusCode":401,"req":{"url":"/api/v1/auth/dashboardsinfo","method":"get","headers":{"host":"<opensearch-dashboard-url>","x-request-id":"be35376a0d887498f9d5489655d82bd9","x-real-ip":"<client-ipaddr>","x-forwarded-for":"<client-ipaddr>","x-forwarded-host":"<opensearch-dashboard-url>","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-scheme":"https","x-scheme":"https","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","referer":"https://<opensearch-dashboard-url>/app/login?","content-type":"application/json","osd-version":"2.7.0","osd-xsrf":"osd-fetch","dnt":"1","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin"},"remoteAddress":"10.42.2.219","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","referer":"https://<opensearch-dashboard-url>/app/login?"},"res":{"statusCode":401,"responseTime":4,"contentLength":9},"message":"GET /api/v1/auth/dashboardsinfo 401 4ms - 9.0B"}
{"type":"response","@timestamp":"2023-08-23T15:22:27Z","tags":[],"pid":453,"method":"post","statusCode":200,"req":{"url":"/api/core/capabilities","method":"post","headers":{"host":"<opensearch-dashboard-url>","x-request-id":"697ccde49f00404b213e4e6f83b6fd5c","x-real-ip":"<client-ipaddr>","x-forwarded-for":"<client-ipaddr>","x-forwarded-host":"<opensearch-dashboard-url>","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-scheme":"https","x-scheme":"https","content-length":"668","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","referer":"https://<opensearch-dashboard-url>/app/login?","content-type":"application/json","osd-version":"2.7.0","osd-xsrf":"osd-fetch","origin":"https://<opensearch-dashboard-url>","dnt":"1","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin"},"remoteAddress":"10.42.2.219","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","referer":"https://<opensearch-dashboard-url>/app/login?"},"res":{"statusCode":200,"responseTime":20,"contentLength":9},"message":"POST /api/core/capabilities 200 20ms - 9.0B"}
{"type":"log","@timestamp":"2023-08-23T15:22:27Z","tags":["debug","http","server","OpenSearchDashboards","cookie-session-storage"],"pid":453,"message":"Error: Unauthorized"}
{"type":"response","@timestamp":"2023-08-23T15:22:27Z","tags":[],"pid":453,"method":"get","statusCode":401,"req":{"url":"/api/v1/auth/type","method":"get","headers":{"host":"<opensearch-dashboard-url>":"3295e240b72e8b42054827c32e6aa077","x-real-ip":"<client-ipaddr>","x-forwarded-for":"<client-ipaddr>","x-forwarded-host":"<opensearch-dashboard-url>","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-scheme":"https","x-scheme":"https","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","referer":"https://<opensearch-dashboard-url>/app/login?","content-type":"application/json","osd-version":"2.7.0","osd-xsrf":"osd-fetch","dnt":"1","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin"},"remoteAddress":"10.42.2.219","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","referer":"https://<opensearch-dashboard-url>/app/login?"},"res":{"statusCode":401,"responseTime":5,"contentLength":9},"message":"GET /api/v1/auth/type 401 5ms - 9.0B"}
{"type":"log","@timestamp":"2023-08-23T15:22:27Z","tags":["debug","http","server","OpenSearchDashboards","cookie-session-storage"],"pid":453,"message":"Error: Unauthorized"}
{"type":"response","@timestamp":"2023-08-23T15:22:27Z","tags":[],"pid":453,"method":"get","statusCode":401,"req":{"url":"/api/v1/multitenancy/tenant","method":"get","headers":{"host":"<opensearch-dashboard-url>","x-request-id":"4233165a1ad68dc1e142f7e52ec49fb5","x-real-ip":"<client-ipaddr>","x-forwarded-for":"<client-ipaddr>","x-forwarded-host":"<opensearch-dashboard-url>","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-scheme":"https","x-scheme":"https","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","referer":"https://<opensearch-dashboard-url>/app/login?","content-type":"application/json","osd-version":"2.7.0","osd-xsrf":"osd-fetch","dnt":"1","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin"},"remoteAddress":"10.42.2.219","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","referer":"https://<opensearch-dashboard-url>/app/login?"},"res":{"statusCode":401,"responseTime":4,"contentLength":9},"message":"GET /api/v1/multitenancy/tenant 401 4ms - 9.0B"}
{"type":"log","@timestamp":"2023-08-23T15:22:27Z","tags":["debug","http","server","OpenSearchDashboards","cookie-session-storage"],"pid":453,"message":"Error: Unauthorized"}
{"type":"response","@timestamp":"2023-08-23T15:22:27Z","tags":[],"pid":453,"method":"get","statusCode":401,"req":{"url":"/api/v1/configuration/account","method":"get","headers":{"host":"<opensearch-dashboard-url>","x-request-id":"352d954928c1c715138065349775b8c5","x-real-ip":"<client-ipaddr>","x-forwarded-for":"<client-ipaddr>","x-forwarded-host":"<opensearch-dashboard-url>","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-scheme":"https","x-scheme":"https","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","referer":"https://<opensearch-dashboard-url>/app/login?","content-type":"application/json","osd-version":"2.7.0","osd-xsrf":"osd-fetch","dnt":"1","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin"},"remoteAddress":"10.42.2.219","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","referer":"https://<opensearch-dashboard-url>/app/login?"},"res":{"statusCode":401,"responseTime":6,"contentLength":9},"message":"GET /api/v1/configuration/account 401 6ms - 9.0B"}

[stephen-crawford]

[Triage] This seems like a good issue. Marking triaged since it has a clear issue to be investigated and fixed.

[kannanvr]

We are also facing the similar issue .
Please let us know your analysis

[ldk300]

If you're using ingress-nginx as your ingress controller, this issue can be caused by the ingress you use to access Dashboards. You might want to look into your ingress controller logs for details or just add the following annotation to the ingress and see what happens:
nginx.ingress.kubernetes.io/proxy-buffer-size: "256k"

[maxemontio]

Adding nginx.ingress.kubernetes.io/proxy-buffer-size: "256k" to ingress annotations fixed the same issue, thank you @ldk300

[ajila-123]

Is this issue solved? I am also having similar problems while deploying KPI dashboard using SSL certificate.