Execute yjs from insight script #11965

4e6 · 2025-01-02T18:45:27Z

Pull Request Description

Allow populating Yjs metadata with values provided by the insight script.

Compile the Insight Script

Run pnpm install

corepack pnpm i

Build app/ydoc-server/polyglot/insight.js bundle with

corepack pnpm run compile

Running

Start the project manager specifying the insight script location with ENSO_JVM_OPTS env variable

ENSO_JVM_OPTS="-Denso.dev.insight=$PWD/app/ydoc-server-polyglot/dist/insight.js"

Every time the insight.js is rebuilt, the engine reloads it automatically.

Debugging (in Chrome Dev Tools)

Add -Dpolyglot.inspect argument:

ENSO_JVM_OPTS="-Denso.dev.insight=$PWD/app/ydoc-server-polyglot/dist/insight.js -Dpolyglot.inspect"

The Engine stops and waits for Chrome Dev Tools to connect to it just like it waits for node.js processes.

Don't Suspend

By default the Chrome Dev Tools integration stops on first executed JavaScript statement. That may not be appropriate in all situations. Add -Dpolyglot.inspect.Suspend=false to the ENSO_JVM_OPTS to disable such behavior. The engine still connects to Chrome Dev Tools, but stops only on breakpoints or when pause is explicitly requested in the debugger.

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

The documentation has been updated, if necessary.
Screenshots/screencasts have been attached, if there are any visual changes. For interactive or animated visual changes, a screencast is preferred.
All code follows the
Scala,
Java,
TypeScript,
and
Rust
style guides. In case you are using a language not listed above, follow the Rust style guide.
Unit tests have been written where possible.
If meaningful changes were made to logic or tests affecting Enso Cloud integration in the libraries,
or the Snowflake database integration, a run of the Extra Tests has been scheduled.
- If applicable, it is suggested to paste a link to a successful run of the Extra Tests.

4e6 · 2025-01-02T18:49:37Z

That's working:

parse the module contents into AST
attach node values provided by the insight as an AST metadata

What's in progress:

synchronize module AST with the ydoc-server. There is an issue with a WebSocket connection that may be caused by the context (mis)configuration

JaroslavTulach

Thanks Dmitry for finding a way to convert insight offsets to ASTs!

JaroslavTulach · 2025-01-03T18:58:44Z

app/ydoc-server-polyglot/src/insight.ts

+  const result = ctx.returnValue(frame)
+  if (result && spanMap) {
+    const key = Ast.nodeKey(ctx.charIndex, ctx.charEndIndex - ctx.charIndex)
+    const asts = spanMap.nodes.get(key)


I see. Whenever we get an insight.on('return', ...) event, we look appropriate asts and update them. I am not sure how fast this is, but it certainly seems good enough for demonstration purposes.

JaroslavTulach · 2025-01-03T19:00:13Z

app/ydoc-server-polyglot/src/insight.ts

+  }
+}, {
+  expressions: true,
+  statements: false,


I believe n = 42 would be a statement - e.g. if you want to observe when "IDE nodes" are being assigned to, you may want to observe statements.

engine/language-server/src/main/scala/org/enso/languageserver/boot/MainModule.scala

engine/common/src/main/java/org/enso/common/ContextFactory.java

JaroslavTulach · 2025-01-31T06:33:32Z

That's working:

* parse the module contents into AST

* attach node values provided by the insight as an AST metadata

What's in progress:

* synchronize module AST with the ydoc-server. There is an issue with a WebSocket connection that may be caused by the context (mis)configuration

What exactly is broken?
How can I use this PR to reproduce the failure?
- what should I execute?
- what should I see then?
can you resolve the merge conflicts, so I can try to play with an update to date working PR?

4e6 · 2025-03-10T09:43:12Z

app/ydoc-server-polyglot/src/insight.ts

+
+const d = new Y.Doc()
+const project = new DistributedProject(d)
+//const provider = attachProvider('ws://[::1]:5976/project', 'index', d)


Uncomment attachProvider method to reproduce the issue

If I comment out the line with 814eb91 I get following errors:

Initializing Insight: {"id":"insight","version":"1.2"} attachProvider ws://[::1]:5976/project index [WARN] [2025-03-14T09:43:00.619] [org.enso.languageserver.protocol.json.JsonConnectionController] Failed to initialize resources java.util.concurrent.CompletionException: org.graalvm.polyglot.PolyglotException: java.net.ConnectException: Connection refused at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315) at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320) at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807) at org.enso.akka.wrapper/akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:49) at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1423) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312) at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843) at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188) Caused by: org.graalvm.polyglot.PolyglotException: java.net.ConnectException: Connection refused at <js>.WebSocket(websocket.js:132) at <js>.setupWS(insight.js:26888) at <js>.connect(insight.js:27164) at <js>.WebsocketProvider(insight.js:27073) at <js>.attachProvider(insight.js:27179) at <js>.poly_enso_eval(insight.js:27213)

I hope that means the WebSocket infrastructure is working fine, it is just misconfigured. What do you think, @4e6?

this one would be resolved by providing ?ls=... argument to the URL

or by adopting Automatically select the only available session #13168

4e6 · 2025-03-10T13:33:26Z

@JaroslavTulach there is an issue with WebSocket connections from inside the insight script. You can reproduce it by trying to establish any connection, i.e.

let ws  = new WebSocket('wss://echo.websocket.org')

JaroslavTulach · 2025-03-13T09:15:10Z

The Chrome Dev Tools debugging shows an exception when creating new socket:

enso$ ./run backend sbt
sbt:enso> runProjectManagerDistribution 
  --env ENSO_JVM_OPTS=-Denso.dev.insight=$PWD/app/ydoc-server-polyglot/dist/insight.js
  --env ENSO_JVM_OPTS=-Dpolyglot.inspect

Concatenate the last three lines to a single sbt line command.

…UG_OPTION

JaroslavTulach · 2025-03-13T11:56:49Z

@JaroslavTulach there is an issue with WebSocket connections from inside the insight script. You can reproduce it by trying to establish any connection,

This is the testing wstest.js to use:

var counter = 0;
function app() {
    let ws  = new WebSocket('wss://echo.websocket.org');
    function sayHi() {
      ws.send(`Hi there! #${counter++}`);
    }
    ws.onopen = function(ev) {
      print("Open: " + ev);
    };
    ws.onmessage = function(ev) {
      print("Msg   : " + Object.getOwnPropertyNames(ev));
      print("  data: " + ev.data);
      setTimeout(sayHi, 3000);
    };
    ws.onclose = function(ev) {
      print("Close: " + Object.getOwnPropertyNames(ev));
    };
    ws.onerror = function(ev) {
      print("Err: " + Object.getOwnPropertyNames(ev));
    };
}

if (typeof insight === "undefined") {
    globalThis.print = console.log;
    if (typeof WebSocket === "undefined") {
        let r = require("ws");
        globalThis.WebSocket = r.WebSocket;
    }
    app();
} else {
    app();
}

the testing script can by used from node (npm install ws and node wstest.js) or as an "Insight Script" in the GUI as ENSO_JVM_OPTS="-Denso.dev.insight=$(pwd)/wstest.js. The script is supposed to send a message to the echo server every three seconds. The script works fine in node js.

I can confirm, the connection gets randomly broken in the GUI. The cause is multithreadedness. GraalVM JavaScript isn't ready to be used from multiple threads. References:

We need to use better Executor to execute callbacks from WebSocket implementation "linearly" and not in parallel.

Update on Apr 11, 2025:

since Share getThreadManager pool among RuntimeServerInstrument and Websocket emulation #12528
the here-in provided script is working properly

…hreaded access

JaroslavTulach · 2025-03-14T03:36:02Z

We need to use better Executor to execute callbacks from WebSocket implementation "linearly" and not in parallel.

Or we need to "recover from parallel Graal.js execution"
e.g. we need threadAccessDeniedHandler as introduced by Allow control of throwDeniedThreadAccess via TruffleContext.threadAccessDeniedHandler oracle/graal#8266
hence Upgrading to Truffle libraries 24.2.0 #12500 - to get that new API
given there are plans to go multi threaded - Improve visualizations feedback via cache observables and safepoints #10525
possibly using JavaScript on multiple threads
we need something like threadAccessDeniedHandler to orchestrate and co-ordinate the access to JavaScript to happen a single thread at a time

…te-yjs-insight

github-actions · 2025-05-23T17:29:50Z

✨ GUI Checks Results

Summary

🧹 Prettier: ❌ Failed
🧰 GUI Checks: ❌ Failed
📚 Storybook: ✅ Deployed

⚠️ Action Required: Please review the failed checks and fix them before merging.

See individual check results for more details.

ℹ️ Chromatic Tests Skipped

Chromatic visual regression tests were skipped for this PR.

To run the tests and deploy Storybook, add the CI: Run Chromatic label to this PR.

Note: We skip tests by default to optimize CI costs. Only run them when your UI changes are ready for review.

👮 Lint GUI Results

Check Results

🧠 Typecheck: ✅ Passed
🧹 Lint: ❌ Failed
🧪 Unit Tests: ✅ Passed

ESLint

3 Warning(s):

`app/ydoc-server-polyglot/src/insight.ts` line `79`

Start Line: 79
End Line: 79
Message: 'metadataJson' is assigned a value but never used. Allowed unused vars must match /^_/u.
From: [@typescript-eslint/no-unused-vars]

`app/ydoc-server-polyglot/src/insight.ts` line `91`

Start Line: 91
End Line: 91
Message: 'frame' is defined but never used. Allowed unused args must match /^_/u.
From: [@typescript-eslint/no-unused-vars]

`app/ydoc-server-polyglot/src/insight.ts` line `121`

Start Line: 121
End Line: 121
Message: 'p' is assigned a value but never used. Allowed unused vars must match /^_/u.
From: [@typescript-eslint/no-unused-vars]

🎭 Playwright Test Results

Summary

⏳ Duration: 5m 25s
✅ Passed: 343 tests
🔴 Failed: 1 tests
⚠️ Flaky: 0 tests

📥 Download Detailed Report

⚠️ Action Required: Please review the failed tests and fix them before merging.

JaroslavTulach · 2025-05-24T04:34:13Z

lib/java/ydoc-polyfill/src/main/resources/org/enso/ydoc/polyfill/web/zlib.js

+        #length;

-        constructor(buffer) {
+        constructor(buffer, offset, length) {


This one is here to support Buffer.from(arrayBuffer[, byteOffset[, length]]). Originally I tried to just extract offset and length from buffer, create a copy of such buffer and pass it into the old constructor. But that doesn't seem to work.

The Buffer.from(arrayBuffer[, byteOffset[, length]]) factory method is supposed to provide live view of the buffer. E.g. further changes are still propagated to the buffer.

As such we are extracting the slice lazily in buffer() getter. With that approach we pass thru the new tests in 8e21525

JaroslavTulach · 2025-05-24T05:30:53Z

lib/java/ydoc-polyfill/src/main/java/org/enso/ydoc/polyfill/WebSocketClient.java

+ *
+ * The {@code enso-engine-*?/enso-*?/component/} represents path to the @{code component} directory
+ * which needs to be adjusted to proper path on each operating system. All lines may need to be
+ * concatenated into a single line.


The WebSocketClient should be a useful utility for testing compatibility of the emulation with real node.js implementation.

…initialization of Enso engine

JaroslavTulach · 2025-05-24T07:10:45Z

lib/java/ydoc-polyfill/src/main/java/org/enso/ydoc/polyfill/WebSocketShell.java

+      System.out.printf("Script %s evaluated to %s\n", code, res);
+      var console = System.console();
+      for (int i = 0; i < 10; i++) {
+        var line = console.readLine("js> ");


This console.readLine seems to work surprisingly well. It supports arrow movements and even provides history of previous entries. Isn't it time to replace JLine with it? Isn't the console actually backed by JLine (repackaged)?

…orking directory in --env specification

JaroslavTulach · 2025-05-27T06:01:11Z

project/DistributionPackage.scala

-      var keyAndValue = args.get(atEnv + 1).split("=", 2)
-      if (jvmOptName == keyAndValue(0)) {
-        prevValue = prevValue + " " + keyAndValue(1)
+      val (key, value) = {


With these changes, it is possible to:

$ sbt sbt:enso> runProjectManagerDistribution --env ENSO_JVM_OPTS=-Denso.dev.insight=$PWD/app/ydoc-server-polyglot/dist/insight.js --env ENSO_JVM_OPTS=-Dpolyglot.inspect

e.g. to influence the environment variable ENSO_JVM_OPTS from sbt shell itself (please concatenate the above example to single sbt command line). No need to restart sbt to change the arguments anymore.

project/DistributionPackage.scala

…Sending message to GUI and stacktrace to the console and log file.

…tion.

...e/runtime-instrument-common/src/main/java/org/enso/interpreter/service/ExecutionService.java

JaroslavTulach · 2025-05-28T11:31:51Z

app/ydoc-server-polyglot/src/insight.ts

+        if (asts) {
+          const resultValue = resultToString(result)
+          for (const ast of asts) {
+            const editAst = syncModule.getVersion(ast)


with Demo: Support displaying ExpressionUpdates found in AST metadata #13179

we should modify this part to ;(ast as any).fields.get('metadata').set('expressionUpdate', update)

e.g. to update the Y.doc server to a value the IDE expects and renders

…te-yjs-insight

4e6 added 7 commits January 2, 2025 18:40

feat: build insight script

2512bb9

feat: initialize polyfill parser

4477b4a

DRAFT: insight parse tree

31c97b9

feat: enso.dev.inspectPort

936bf45

DRAFT: set widget metadata

d9ce757

DRAFT: y-websocket

50266f9

DRAFT: debug websocket

93ba1dc

4e6 added the -language-server label Jan 2, 2025

4e6 self-assigned this Jan 2, 2025

JaroslavTulach reviewed Jan 3, 2025

View reviewed changes

enso-bot bot mentioned this pull request Jan 4, 2025

Text.find slows down with the number of invocations #11859

Closed

This was referenced Jan 9, 2025

Avoid multiple serialize-deserialize cycles when returning results to Language Server clients #5741

Closed

Remove JavaScript from bin/enso executable #12018

Closed

JaroslavTulach mentioned this pull request Jan 20, 2025

Reducing helper methods in Standard.Base.Meta #12031

Merged

3 tasks

JaroslavTulach linked an issue Jan 31, 2025 that may be closed by this pull request

Allow to execute Y.js and Insight together #11477

Closed

4e6 commented Mar 10, 2025

View reviewed changes

4e6 added 2 commits March 10, 2025 13:13

Merge branch 'develop' into wip/db/11477-execute-yjs-insight

7c889c6

update: after merge

2932fac

Combine previous value of ENSO_JVM_OPTS together WithDebugCommand.DEB…

8d17b9e

…UG_OPTION

Re-try callback when JavaScript refuses to execute because of Multi t…

814eb91

…hreaded access

enso-bot bot mentioned this pull request Mar 14, 2025

Allow to execute Y.js and Insight together #11477

Closed

This was referenced Mar 14, 2025

Upgrading to Truffle libraries 24.2.0 #12500

Merged

Make Native Image opt-out #12483

Closed

Merging with latest develop

c8acbb2

Updating to helidon 4.2.2

f383682

JaroslavTulach force-pushed the wip/db/11477-execute-yjs-insight branch from e8558ca to f383682 Compare May 12, 2025 09:35

JaroslavTulach added 4 commits May 12, 2025 14:40

Removing outdated debug configuration

877b227

Merge remote-tracking branch 'origin/develop' into wip/db/11477-execu…

7df699c

…te-yjs-insight

Merge remote-tracking branch 'origin/develop' into wip/db/11477-execu…

e29bfc1

…te-yjs-insight

Create zlib Buffer over ArrayBuffer view

8e21525

JaroslavTulach reviewed May 24, 2025

View reviewed changes

JaroslavTulach linked an issue May 24, 2025 that may be closed by this pull request

Deliver expression updates via Ydoc AST elements #11159

Open

WebSocketClient utility to check compatibility of the polyfill

1c7e750

JaroslavTulach reviewed May 24, 2025

View reviewed changes

JaroslavTulach added 3 commits May 24, 2025 08:13

Delay initialization of insight by ten seconds to not interfere with …

6f0d885

…initialization of Enso engine

Don't clean the session to avoid NPE and get more explainable exception

b10c9c7

Turn into interfactive shell

a44375a

JaroslavTulach reviewed May 24, 2025

View reviewed changes

JaroslavTulach added 3 commits May 26, 2025 17:44

Merging in #13159

efda23a

Allow override (composition) of ENSO_JVM_OPTS env variable

1ed6d4d

Use /home/devel/NetBeansProjects/enso/enso.ide to reference current w…

e8c11ab

…orking directory in --env specification

JaroslavTulach reviewed May 27, 2025

View reviewed changes

JaroslavTulach added 3 commits May 27, 2025 08:18

Cannot stringify frame - the content may be too deep

1ec18f1

--jvm option takes argument in enso, but not in project-manager

b81b815

Allow project-manager --jvm to have an argument

c8e5342

JaroslavTulach reviewed May 27, 2025

View reviewed changes

project/DistributionPackage.scala Outdated Show resolved Hide resolved

JaroslavTulach added 2 commits May 27, 2025 09:11

Unexpected exceptions like IllegalStateException should be reported. …

2be45bf

…Sending message to GUI and stacktrace to the console and log file.

prepareFunctionCall may need Truffle context. Wrapping in submitExecu…

08c21ae

…tion.

JaroslavTulach reviewed May 27, 2025

View reviewed changes

...e/runtime-instrument-common/src/main/java/org/enso/interpreter/service/ExecutionService.java Show resolved Hide resolved

JaroslavTulach mentioned this pull request May 27, 2025

Implement safepoint-aware executor service for a more fine-grained job execution control #11429

Open

JaroslavTulach reviewed May 28, 2025

View reviewed changes

Merge remote-tracking branch 'origin/develop' into wip/db/11477-execu…

8a73292

…te-yjs-insight

JaroslavTulach mentioned this pull request Jun 5, 2025

Consistency of --jvm CLI options: allowing optional JVM path #13225

Merged

3 tasks

Execute yjs from insight script #11965

Are you sure you want to change the base?

Execute yjs from insight script #11965

Uh oh!

Conversation

4e6 commented Jan 2, 2025 • edited by JaroslavTulach Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Description

Compile the Insight Script

Running

Debugging (in Chrome Dev Tools)

Don't Suspend

Checklist

Uh oh!

4e6 commented Jan 2, 2025

That's working:

What's in progress:

Uh oh!

JaroslavTulach left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JaroslavTulach commented Jan 31, 2025

That's working:

What's in progress:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

4e6 commented Mar 10, 2025

Uh oh!

JaroslavTulach commented Mar 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JaroslavTulach commented Mar 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JaroslavTulach commented Mar 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✨ GUI Checks Results

Summary

ℹ️ Chromatic Tests Skipped

👮 Lint GUI Results

Check Results

ESLint

3 Warning(s):

🎭 Playwright Test Results

Summary

Uh oh!

JaroslavTulach May 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JaroslavTulach May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

4e6 commented Jan 2, 2025 •

edited by JaroslavTulach

Loading

JaroslavTulach commented Mar 13, 2025 •

edited

Loading

JaroslavTulach commented Mar 13, 2025 •

edited

Loading

JaroslavTulach commented Mar 14, 2025 •

edited

Loading

github-actions bot commented May 23, 2025 •

edited

Loading

JaroslavTulach May 24, 2025 •

edited

Loading

JaroslavTulach May 27, 2025 •

edited

Loading