labextension in proxy mode doesn't appear to work behind Open Ondemand - check_origin for websockets missing in proxy?

[davidmonro]

What happened:
We are attempting to run dask-jobqueue through Open OnDemand (https://openondemand.org/). This is working, except for the labextension graphs etc. Once the cluster is launched, and the extension activated by inserting '/proxy/8787/status' in the extension search bar, clicking on the buttons produces blank tabs, and errors like this:
[W 2021-02-15 16:25:58.768 ServerApp] 403 GET /node/XXXXX/27554/proxy/8787/individual-task-stream/ws (XXX.XXX.XXX.XXX) 2.03ms referer=None

What you expected to happen:
I expect to get graphs etc instead of blank tabs.

Minimal Complete Verifiable Example:

from dask.distributed import Client,Scheduler
from dask_jobqueue import SLURMCluster

cluster = SLURMCluster(cores=2,memory="4GB")
client = Client(cluster)
cluster.scale(2)

(then activate the extension and click a button)

Anything else we need to know?:
If I start the jupyterlab instance via ssh with port forwarding rather than OnDemand, it works, so it is something to do with running behind the proxy layers.
OnDemand does create a config.py as follows:

c.NotebookApp.ip = '*'
c.NotebookApp.port = 8267
c.NotebookApp.port_retries = 0
c.NotebookApp.password = XXXXXXXX
c.NotebookApp.base_url = '/node/XXXX/8267/'
c.NotebookApp.open_browser = False
c.NotebookApp.allow_origin = '*'
c.NotebookApp.notebook_dir = '/home/ZZZZZ'
c.NotebookApp.disable_check_xsrf = True

Note particularly the allow_origin setting.

The problem appears to be that jupyter_server_proxy doesn't override the WebSocketHandler check_origin method in websocket.py/WebSocketHandlerMixin

If I add a check_origin method to this class which returns True, then everything appears to work.

My guess is that there should be a check_origin method in here very similar to the one in
jupyter_server/base/handlers.py/JupyterHandler but I don't know anything about the architecture of jupyter etc, so I don't know if that is actually the right place.

Environment:

• Dask version: 2021.2.0
• Python version: 3.7.4
• Operating System: Centos 8
• Install method (conda, pip, source): pip

[jacobtomlinson]

Sorry for the delay here. Could you confirm the Bokeh version installed in your HPC environment?

[davidmonro]

Depends which environment; looks like it was 2.2.3 in one environment and 2.3.0 in another.

My debugging indicated that the problem is in jupyter_server_proxy rather than in the labextension itself, but I've not had any luck reporting the issue over there.

Currently we work around it by patching the jupyter_server_proxy, but that is awkward for users building their own environments.

Cheers

David

[jacobtomlinson]

Thanks @davidmonro. Could you share where you raised things on the proxy?

[davidmonro]

I raised it on github and on discourse:

jupyterhub/jupyter-server-proxy#248 and https://discourse.jupyter.org/t/issue-with-dask-labextension-behind-reverse-proxy-looks-like-an-issue-in-jupyterlab-server-proxy/8299/1

Cheers

David