Voila + JupyterLite

[martinRenou]

Problem

Voila + JupyterLite would be amazing.

Proposed Solution

I started looking into implementing an nbconvert exporter that exports a Notebook to a Voila app that spawns a lite-kernel and execute the code in the page. voila-dashboards/voila#1187

So far, I think the exporter only is not enough. We will need to reuse some of the federated-extensions and Addon logic from the JupyterLite CLI.

My idea is to:

• add a Voilite app to the JupyterLite repository
• add a JupyterLite core-Addon that pre-exports all Notebooks (from the contents) to Voilite dashboards using the for-mentioned nbconvert voilite exporter, and add them under the Voilite app directory output
  • the Voilite app will redirect all /voila/render/*.ipynb requests to the pre-rendered voilite dashboards
  • if /voila/render/notebook.ipynb points to a Notebook that was not pre-rendered, we execute nbconvert in the page to render the Notebook, this will be way slower than opening pre-rendered dashboards, but it's a must have anyway.

[bollwyvl]

add a Voilite app to the JupyterLite repository

As long as it has all the machinery to do the rendering in the browser, this is great... mostly because, at present, we don't have a better way for apps to participate in the MIMO-with-shared-webpack-chunks. But notebooks-as-authored-in-lite-as-dashboards is definitely a marquee feature.

/voila/render/*.ipynb

I think we've been trending more towards "branding-neutral" names in URLs (e.g. /repl/ instead of the initial /replite/), so perhaps /dashboard/?

pre-exports all Notebooks

As suggested above, as not everyone site would want this, I think it should likely be opt-in, and have an extra dependency (e.g. jupyterlite[dashboards]) which allow the system to fail gracefully if not present.

[martinRenou]

I think we've been trending more towards "branding-neutral" names in URLs (e.g. /repl/ instead of the initial /replite/), so perhaps /dashboard/?

IMO staying with /voila/render/*.ipynb is better because users will not see a difference between lite and "official" usage: we stay close to the official voila.

As suggested above, as not everyone site would want this, I think it should likely be opt-in, and have an extra dependency (e.g. jupyterlite[dashboards]) which allow the system to fail gracefully if not present.

Maybe through the app config? app could default to ["lab", "retro", "repl"] and be set to ["lab", "retro", "repl", "voila"] optionally?

[bollwyvl]

not everyone site would want this

More specifically, I mean the build-time pre-rendering and dependency on nbconvert, voila, and whatever the new things are.

If it piggy-backs of contents, then it might need a few more switches, which are already complicated. If anything, it would need a switch to disable prerendering for certain globs, or altogether (probably while still leaving the runtime app in place), so more --ignore-dashboards=*.

[martinRenou]

Morning idea:

At build time: Instead of pre-rendering all the Notebooks -> pre-render all the Voila templates with empty cells content.

At runtime: We inject any Notebook cells content (in some JS logic) in the pre-rendered voila template the user is requesting -> no need for nbconvert at runtime.

[martinRenou]

What we want

A Voila app in JupyterLite?

An in-browser Voila "server" that would serve a dashboard upon /voila/render/my_notebook.ipynb

HOW?

idea1: Run nbconvert in the browser to generate the dashboard upon request?

pros:

• Very close to official Voila
• Supports any custom Voila template

cons:

• Need to spawn a Python runtime in the browser to run nbconvert (slow, memory consuming)

idea2: Pre-render all Notebooks into Voila dashboards during the jupyter lite build step?

pros:

• Don't need to run nbconvert in the browser
• Fast to serve dashboards

cons:

• Cannot have an equivalent to voila-preview unless we do run nbconvert in the page in this particular case
• We end up with n_templates * n_notebooks number of notebooks on the server, some of them won't be used
• If the user edits a Notebook, they won't understand why the updates are not visible in the Voila dashboard

idea3: Pre-render all Templates (using an empty Notebook) during the jupyter lite build step, and in the frontend dynamically populate the actual content of the Notebook?

pros:

• (compared to idea2) Not limited to existing Notebooks at build time
• Don't need to run nbconvert in the browser
• Fast to serve dashboards

cons:

• will not work well with templates that need to preprocess the Notebook or depend on the Notebooks cells content at build time (reveal template needs the cells metadata)

idea4: Render Jinja2 templates with a JavaScript library like liquidjs?

Does liquidjs actually support jinja2 templates?

pros:

• does not need a Python runtime
• would work with any template?
• fast!
• can have a nice voila-preview extension

cons:

• need to rewrite some Python logic from Voila in JavaScript

CLI interface?

Generate a voilite dashboard and spawn a static server with:

voilite my_notebook.ipynb

HOW?

idea1: Make a voilite exporter that creates a single HTML file which embeds everything (labextensions, kernel etc)

pros:

• simple

cons:

• does not work well with what's existing already (jupyterlite, jupyterlite-xeus-python)

idea2: Make this cli interface reuse the jupyterlite build logic to bring the labextensions and xeus-python addons.

pros:

• works well with what's existing already (juptyerlite, jupyterlite-xeus-python)

cons:

• more complicated than a single HTML file

[bollwyvl]

Roughly in order of my perceived value, trying to keep the "lite" accurate for both developers and users.

Pre-render all Notebooks

The (optional) pre-render plan is a good first step as the complexity can be contained within the (optional) build addon.

If we pair that with the #165 labextension and serverextension (or even a widget), the user story of..

As a future site owner
In order to build a nice site
Given I have full JupyterLab
 When I change a notebook
  And I press a button
 Then I see a lab or browser tab preview of dashboard
  And I press another button
 Then my site is deployed
  And I can repeat the process

...will be really solid: this was actually the original impetus behind using:

• doit to structure the work of each build in such a way that it would cache everything possible...
  • we lost a little along the way, namely federated extensions get unpacked every re-build, but we can probably fix that up.
• JSON schema all over for config
  • so we can use rjsf to get a decent UI "for free" (ha)
  • still need some extensibility work, e.g. (third-party) addons should be able to layer in their stuff
• traitlets so that much of the work would be evented-by-default
  • if run in-server, we need to re-think the LiteBuildManager a little, probably with a queue system and better status reporting

If we ever got that self-hosted in-browser, that would really be something, and worth an extension that did include the whole build chain, used a kernel, etc. We should not try to do that before 0.1.0 lol.

Render Jinja2 templates with a JavaScript library

As soon as out-of-band pre-rendering works, I think we want to investigate nbconvert-in-js to enable another use case:

As a current lite site user
When I visit the site
  And I work on a notebook
When I press a button
  Then I see my dashboard
  And I can download my notebook
  And submit it 

I think nunjucks is going to get us the closest, as it has jinja2 compatibility as a goal, and can be run asynchronously. We should try to write this in as-standalone-as-possible way (e.g. sans-io, no react, no lumino, etc) as this would actually be really useful in other contexts if it was lightweight enough.

nbconvert in the browser

This kinda could be made to work today... but should not be our go-to... things like --execute won't work (what's a process?) and it would indeed be memory/bandwidth hungry. Much like the jupyverse discussion, I think we would want to leave more hooks in place to allow ideas like these to replace the defaults, but the defaults need to be tuned to keep the lite idea alive: this stuff should feel fast on a million chr*mebooks, and anywhere we can trade some development complexity for megabytes-on-the wire, we should.

I point out some future work there: basically, we would either need to patch, or convince nbconvert upstream to reverse its trend of adding more hard dependencies, and move some things like the jupyter_client dependency into an optional. Perhaps the way to do this for nbconvert-core to become a real thing, which makes fewer assumptions, while the standard nbconvert package pulls in execution.

single HTML file

this is... a lot. I actually got this to work on nbpresent, but it was a kinder, gentler era.... and still relied on CDN. This would, however, be wonderful for an archival document, but only if all the CDN references were resolved, which would require more sophisticated build-time resolution of language dependencies.

#497 goes further and asks for it to be self-modifiable. We do want this at some point. I think this would take some fairly advanced mechanisms to detect all the dynamism, even with a much more robust serviceworker, and would result in a titanic file, as everything (including all that wasm) would be base64 encoded.

If anything, I think we would want to explore this in an out-of-band extension, and not in the core repo... if we did a really good job of documenting our build outputs, it should be possible to take a full jupyter lite archive in and generate an html file, but has a very limited usefulness for interactive computing.

[jtpio]

@martinRenou maybe we could close this issue and continue on the Voici repo now? https://github.com/voila-dashboards/voici