Notebook debugging/"run by line"

[roblourens]

The current plan is to support both real debugging and "run by line" in Jupyter. "Run by line" is a simple debug mode that only allows stepping through the code of a single cell. It will be backed by a real debug session but with some debug UI in VS Code disabled.

Basic debugging questions/tasks

• Enable cell breakpoint margin in a notebook
  • Show an explicit button to show the breakpoint margin. The debugger contribution could say that it supports notebooks of some type (similar to the languages property on the debugger contribution), which enables this action.
  • I need to be able to set breakpoints before I run a cell, and I don't want to show the breakpoint margin all of the time, so we need this mode.
    
• How does a debug session start?
  • No launch config
  • We have discussed two concepts: starting a long running debug session, or a "run and debug" button that starts a debug session just for the time a cell is being debugged. Sounds like people generally prefer the second.
  • However, we need to have this "show breakpoint margin" mode, and we could also just combine these into a single "debugging enabled" mode which is pretty much how jupyter lab works. With this enabled, the normal run button hits breakpoints. I think it's more confusing to have a separate button to "run and debug". If I set a breakpoint and want to run the cell, why do I have to go find a different run button? Do I also need "run and debug all" or "run and debug cells below", etc.?
  • I think that ideally, VS Code would manage the experience to start debugging, and would find and invoke the proper DA, but for now, we should have the extension do all of this until we figure out what the experience should actually look like, which would drive vscode/api changes.
  • The extension also can dynamically check whether the kernel supports debugging. If VS Code owns the start debugging experience, this needs to be surfaced.
    • The controller could set a debugAdapter property which tells us the ID of a debug adapter that can debug the active kernel. If it's undefined, debugging is not enabled.
• The work to implement the debug adapter, talk to ipykernel, and jupyter-specific things Native Notebooks - Run by Line vscode-jupyter#5607
• Restore breakpoints to correct cell when URIs change after closing/reopening notebook
  • The prototype repo has a fix for this in the extension, but vscode should do this automatically for all notebooks, maybe by saving the cell index with the breakpoint itself, or just by having some notebook contrib that moves them around.
• Is it bad that the timer runs when paused?
  • Can pause it, but the start/end times in the API will still reflect the paused time. We could just hide this and not persist the duration when pausing in a cell.
• [ ] Variables pane vs debug variables view
• [ ] The debug console vs interactive pane
• [ ] Figure out how debugging across different kernel types would work
  • If dotnet interactive and julia implemented DAP with the same extensions as Jupyter, the same debug adapter could be used - https://jupyter-client.readthedocs.io/en/stable/messaging.html#additions-to-the-dap
  • But then where does the debug adapter extension live, and how does it acquire the connection to the kernel?
  • It is probably easier to have separate debug adapters that live next to the extension code that has a connection to the kernel, we can share code with a library. However it was mentioned that this same problem basically exists with the "restart kernel" command which is pretty much the same across different kernel types but can't be contributed by a single extension without a way to get the kernel connection for other controllers.
  • We are focusing on python and the kernels provided by the vscode-jupyter extension
• [ ] Should we put cells in readonly mode when paused?

Run by line tasks

• Start a debug session when the "run by line" button is clicked
  • The extension will do this with vscode.debug.startDebugging
• Ability to start a debug session in "simple mode" which "run by line" will enable. In "simple mode", the window statusbar will not change color, the debug widget will not appear, and the debug viewlet will not be shown. Everything else can remain the same.
  • How do we start a debug session in "simple mode"? We can make it part of the debug adapter's "capabilities", or a launch config prop read by vscode, or a flag on DebugSessionOptions. I think the first is preferable, but will there be debug UI shown before we get the capabilities?
  • My thinking is that "run by line" will not be a feature that all notebooks get out of the box, but something that an extension can implement using the "simple mode" flag, toolbar contributions, and other basic vscode API, because I am not sure that every notebook type in the world would agree on the details.

https://excalidraw.com/#json=5721711802580992,GeAhnXz10-VKb53A9XmgNA

cc @isidorn @weinand

[weinand]

@roblourens @DavidKutu some comments:

• Breakpoints can be created and edited independent from debug sessions and the typical flow is to add breakpoints to a source file and then start debugging. Since notebooks cells do not show the breakpoint margin by default, I've introduced a "debugging mode" for notebooks. With a toggle button in the editor's header, debugging can be enabled and the breakpoint margin (and all persisted breakpoints) become visible. The breakpoint margin is controlled via the cell's metadata breakpointMargin property. Since "breakpoints" are not part of the simple "Run by line" mode, the toggle button could be repurposed as a toggle for switching between "simple" and "full" debugging mode.
• when notebook cells are rearranged in a notebook editor, persisted cell breakpoints become outdated because their cell URIs still contain the old cell indexes. This code tries to fix this by updating the indexes in the breakpoint URIs. This code is independent from debugging and should really live close to the code which rearranges notebook cells. This problem would not exist if cell URIs would not use the cell's index but some robust ID.

BTW, my Nodebook sample shows how to implement notebook debugging functionality based on existing VS Code debugger extensions. It is simpler than the vscode-simple-jupyter-notebook because it does not rely on the DAP implementation of the xeus kernel and therefore has no DAP forwarding.

[roblourens]

The breakpointMargin metadata is gone and we don't plan to add another way for extensions to control this. Talked more about this today, and we will probably just need some explicit switch like you previously had (or, just show the breakpoint margin all the time).

I don't think we plan to give cells a real ID, but I agree with your points about that problem.

[DonJayamanne]

igure out how debugging across different kernel types would work

I don't think .NET & Julia support debugging in Jupyter today, hence if debugging were to be implemented, i don't think it would be implemented through the jupyter protocol, as both have their corresponding VS Code extensions that support notebooks and do not go through Jupyter (i.e Jupyter protocol isn't involved in execution today in .NET/Julia/R code execution).

@davidanthoff /cc

[davidanthoff]

Yes, that is correct for Julia: there is no debugging support in the IJulia kernel that is used by JupyterLab. But we do have a DAP implementation in the Julia VS Code extension, so my hope would be that we can just use that for the Jupyter notebook scenario that uses the Julia extension to run code.

I think my preference would be to try to get the full debugger working for Julia inside Jupyter notebooks, rather than spend time on a simplified "run by line" feature. There really wouldn't be any reason to use the latter if we have a full DAP/proper debugger experience implemented, right?

[rchiodo]

@davidanthoff

I think my preference would be to try to get the full debugger working for Julia inside Jupyter notebooks, rather than spend time on a simplified "run by line" feature. There really wouldn't be any reason to use the latter if we have a full DAP/proper debugger experience implemented, right?

Run by line was created first because of the feedback we got from data scientists. They didn't want a debugger, didn't see the reason for one and were not interested in using it. They just did printf debugging, why would they need something else? Run by line was our kind of debugger lite scenario so that these users would feel more comfortable with it. Right now we have about 40,000 people using it per month. I'm sure some of them would prefer full debugging, but we think run by line removes the worries about having to 'learn a debugger'.

[davidanthoff]

Maybe the question of UI and underlying implementation is distinct? I could imagine something where the button "run line by line" just starts the debugger? And maybe there is some UI for "next line" that is really hooked to "run next statement"? It just feels that one should easily be able to implement a line-by-line UI feature if there is an existing DAP implementation for a language, without adding another API/execution model or something like that.

Also, just out of curiosity: this feature really would have to be "run statement by statement", right? Is there some API to detect the extend of the next statement? At least in Julia executing line-by-line doesn't make that much sense because there are many multi-line statements that would just be syntax errors if one tried to execute them line-by-line. We do have the feature to detect the next syntactically complete statement in our language server, but it is a custom protocol extension and we would need some more API to surface that to somewhere else, right?

[DonJayamanne]

feature really would have to be "run statement by statement", right?

Yes that's right. That's what it is today.

is a custom protocol extension and we would need some more API to

No it isn't. Is built on top of the existing dap.

feels that one should easily be able to implement a line-by-line UI feature if there is an existing DAP implementation for a language, without adding another API/execution model or something like that.

Yes technically, that's what how we implemented this for python in our webview based notebook.

[rchiodo]

Run by line currently does a step into on every 'F10' by the user. It then checks the frame when the stop event comes in to see where the step into landed. If in the same cell, it stops the UI. If not in the same cell, it executes a step out command. This is repeated for every step by the user until it lands outside of the current cell (or meaning the frame that called the cell to begin with)

[davidanthoff]

@DonJayamanne Ah, cool! So essentially from our end, we "just" need to get the DAP story going with the notebooks :) I like that, because we of course want to do that in any case.

@rchiodo Ah, so it actually will descend into a nested function calls step-by-step as long as those functions are all defined in one cell? What about things like higher order functions? Say you have this Julia code:

function foo(i)
  return i*2
end

x = map(foo, sourceArray)

If you run a step-into command on the line with map, you'll stop inside the map implementation in Julia base, and I think by the logic you describe it would then just do a step-out right away and never break inside foo, right? Whereas if someone called foo directly from the global scope, in the logic you describe it would break in foo, right? Similar, if a user were to call any intermediate function that was defined in a different cell that would then call foo, it would not stop there, right? Maybe another way to implement this feature would be to set breakpoints under the hood on every line in the cell? That way one would get a really consistent experience where "run line-by-line" in a cell really stops on every line, no matter what.

I think I'm still wondering, though, whether all of this just primarily introduces more choices, and more special cases, and whether one couldn't just make the normal debug experience "non scary"? But I think I should probably just try out the existing line-by-line to get a sense how that works :)

[rchiodo]

For your map case (at least with the debugger for python code), it won't stop in map cause that's considered 'system' code. There's a 'just my code' feature of the DAP that prevents stepping through stuff that isn't your own code.

But yes it doesn't work for cells that jump through intermediate functions that you defined that are not in the same cell. There's a lot of cases where Run by line doesn't really 'work'. We argued about a lot of them :) The behavior we have right now is where it ended up.

There's also talk of run by line auto running to where the cursor is. Cause that's where the user is currently investigating. Why make them step all the way through the code above the cursor.

[rchiodo]

@davidanthoff is Julia thinking about supporting debugging in JupyterLab? We thought if kernels supported debugging there, they'd want to use the same code in VS code to light up debugging. (meaning the kernel would expose the DAP instead of an extension)