500 errors in CosmicDS

[pkgw]

Reported by @Carifio24 and @patudom.

[W 11:11:49.762 NotebookApp] wwt_kernel_data_relay | incomplete or missing response from kernel | key=pywwt_static0.15.2.final.0 entry=research/ kernel_id=9e00bd6d-6dce-451f-9dbc-2fe254f67dd2 msg_id=eaaeed9b-adc0cfd1d5c6791fb3b8dc0e_1227763_53
[E 11:11:49.762 NotebookApp] {
 "Host": "localhost:8890",
  [...]
 "Referer": "http://localhost:8890/notebooks/cosmicds/stories/hubbles_law/CosmicDS.ipynb",
 }

"primarily when running our app in a Jupyter notebook (as opposed to voila). [W]e get a 500 error, and the window doesn't pop up."

[pkgw]

I haven't reproduced the 500 error but I can cause something to go wrong:

1. Create custom Python environment for testing.
2. Check out cosmicds/cosmicds to e00de24cac83df6439d9f1c4c11425d1ff6d5ea2 (last pre-prekdr commit)
3. Work through pip install -e . with various version tweaks to get everything working
4. jupyter notebook . in cosmicds directory
5. Open cosmicds/stories/hubbles_law/CosmicDS.ipynb
6. Click through intro

I eventually get:

[IPKernelApp] ERROR | No such comm target registered: jupyter.widget.control
[IPKernelApp] WARNING | No such comm: 2297d3b5-1e2c-453b-a0ee-f96089dd9273
[I 12:33:55.570 NotebookApp] 302 GET /api/kernels/wwtkdr/pywwt_static0.15.2.final.0/research/?origin=http%3A%2F%2Flocalhost%3A8888 (127.0.0.1) 0.550000ms
[I 12:34:11.275 NotebookApp] 302 GET /api/kernels/wwtkdr/pywwt_static0.15.2.final.0/research/?origin=http%3A%2F%2Flocalhost%3A8888 (127.0.0.1) 0.380000ms
[W 12:34:11.283 NotebookApp] 404 GET /api/kernels/wwtkdr/pywwt_static0.15.2.final.0/research?origin=http%3A%2F%2Flocalhost%3A8888 (127.0.0.1) 4.840000ms referer=http://localhost:8888/notebooks/cosmicds/stories/hubbles_law/CosmicDS.ipynb

... which obviously isn't what's supposed to be happening. Will investigate.

[pkgw]

OK, this error is due to the cosmicds monkey-patch removed in cosmicds/cosmicds@9545c1c . The message of the commit that added it, cosmicds/cosmicds@d826882, mentions that it's for the Voila baseurl, so if the monkeypatch was active when the code was being run in a plain notebook, that would potentially cause problems. But it would cause what I saw before, not the 500 errors reported above.

@nmearl Would this monkeypatch still be necessary when running pywwt KDR in Voila? We should find a way to fix the pywwt implementation to work correctly in as many circumstances as possible.

@patudom @Carifio24 I'll play around some more but it would definitely be helpful to get a reproduction script.

[Carifio24]

@pkgw So I can sometimes generate a 500 error as follows:

• Complete steps 1-5 that you outlined above
• Run the app once, then restart the app via "Restart & Run All"
• Go to one of the WWT viewers in the introduction (slides 3 or 4)

Occasionally the window doesn't appear, with the body of the iframe being just the text incomplete or missing response from kernel

In the terminal, I get:

[W 18:05:43.050 NotebookApp] wwt_kernel_data_relay | incomplete or missing response from kernel | key=pywwt_static0.15.2.final.0 entry=research/ kernel_id=47c13242-7183-40a5-a6c2-2a97c8d4c7e2 msg_id=5d18a799-4fff484f8de9f466fef2996e_13335_44
[E 18:05:43.051 NotebookApp] {
      "Host": "localhost:8888",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
      "Referer": "http://localhost:8888/notebooks/cosmicds/stories/hubbles_law/CosmicDS.ipynb",
      "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"
    }
[E 18:05:43.051 NotebookApp] 500 GET /wwtkdr/pywwt_static0.15.2.final.0/research/?origin=http%3A%2F%2Flocalhost%3A8888 (127.0.0.1) 30142.010000ms referer=http://localhost:8888/notebooks/cosmicds/stories/hubbles_law/CosmicDS.ipynb

This only happens occasionally. I'll update if I can come up with a better way to reproduce this.

[pkgw]

Thanks, that's helpful. Due to the way the KDR architecture works, it could definitely make sense that a kernel restart needs to be involved to trigger the issue.

[pkgw]

OK, cool, I can reproduce. Do you also get log messages like:

[IPKernelApp] WARNING | Unknown message type: 'wwtkdr_resource_request'
[IPKernelApp] WARNING | No such comm: 017dc4c439ff41a3ad3fdd9edfe8c0ad

when the problem occurs?

[Carifio24]

I do get that message, but at least for me, it pops up together with the 302 -> 404 error.

[pkgw]

For the 302/404 error: after checking out the pre-prekdr commit, you need to edit cosmicds/__init__.py to remove the monkeypatch.

[pkgw]

OK, I think I understand what's going on here.

Basically, in this setup, when the Voila app is present and the WWT app has been loaded at all, and you restart the kernel, the browser needs to re-fetch the WWT app, and the request goes in before the new kernel has been set up to handle KDR requests. I think that this is due to rules about reparenting iframes.

Hypotheses:

1. If you clear the output ("Cell" menu) so that the Voila app goes away, then restart the kernel, things should work.
2. If you open the Voila app, do not navigate to a WWT page, then "Restart and Clear Outputs", then rerun, things should work.
3. If you do the previous item but do navigate to a WWT page and do not subsequently clear the output cell, there will probably be issues, because of the re-fetch triggered upon cleaning up the Voila app. Even if you then navigate away from a WWT page.

So I think that right now, if one makes sure to always clear the output(s) before restarting the kernel, things should work consistently.

But, this is all stemming from the fact that the Python kernel is trying to provide the research app through the KDR. If the pywwt server extension is installed, the research app can be made available directly, without requiring any use of the KDR. Then things should Just Work™ much more reliably. The problem right now is that pywwt-in-the-kernel doesn't (and can't) know whether pywwt-in-the-server is available ... but I think I can figure out a way to allow it to detect that. Or I can at least give you an API to manually activate that mode.

[pkgw]

OK, I think WorldWideTelescope/pywwt#331 should give you a way to address this. Once it's released, if you create all of your WWTJupyterWidget instances with the argument app_url = "/wwtstatic/research/", the problems here should be avoided, I think.

[pkgw]

@Carifio24 @nmearl If one of you gets the chance to test this out with the branched version of pywwt, that would make me more confident that it's a sufficient solution. But based on my testing I think it's OK so I'm happy to go ahead and merge and release to make the new API easily accessible.

edit: Also it would be nice to devise and merge a fix to remove the requirement for the Voila monkeypatch before I release the next version.

[nmearl]

@pkgw Revisiting this issue on a fresh JupyterHub install reveals that, with using app_url="/wwtstatic/research", the embeded iframes in the JupyterWWTWidgets still do not show WWT. It seems that the iframe url is http://localhost:1200/user/admin/wwtstatic/research, but the true WWT url (verified by visiting it directly in a local tab and having it render successfully) is http://localhost:1200/user/admin/user/admin/wwtstatic/research.

Any insight into why that second user/admin is being introduced?

This is with using the latest master of PyWWT (aea3f41) and the latest release of Voila.

[pkgw]

Weird! But also not too surprising, since the handling of the baseurl seems like something that is always going to be very finicky. I'll look into it.

[pkgw]

Hmmm. Well, I don't think that pywwt is doing anything wrong here. The code to set up the URL handling is very straightforward and pretty much verbatim out of the Jupyter docs. In an environment like BinderHub we get correct behavior with regards to the base_url setting.

So I think there must be an issue in JupyterHub, or maybe somehow the interaction of JupyterHub and Voila, that leads to this problem? Maybe the way that the "hub" functionality works, the user's Jupyter Server should think that it has a base_url of /, but it's getting /user/:user instead? If the hub applied the /user/:user prefix at a different layer of the HTTP processing, I think that could cause what you're seeing.

Anyway, lacking a JupyterHub install (and not coming up with anything that looks relevant on Google), I can't make much progress on this. @Carifio24 are you able to run JupyterHub and potentially dig into this?

[nmearl]

@pkgw Interesting. Yeah, it seems like if I override L84 will route_pattern = url_path_join('/', "/wwtstatic/(.*)"), the widgets render properly. The handler path must be getting set additionally somewhere else along the way.