Provide public API to attach debugger in excepthook and see unhandled exception

[Timmmm]

Hi,

I'm trying to set up debugpy to listen() and wait_for_client() when an exception is thrown. Unfortunately I have to do it like this because I'm looking for one exception from a janky hodgepodge of embedded Python processes and Makefiles.

With the following code:

import debugpy

print("Waiting for debugpy on port 5678.")
debugpy.listen(5678)
debugpy.wait_for_client()

raise Exception("test")

If I run python3 main.py and then connect to it, it catches the exception (great):

But I can't figure out how to wait for the client on demand when an exception is thrown. Here is my attempt:

import sys
from types import TracebackType
from typing import Any, Callable, Type

def make_debugpy_excepthook() -> Callable[[Type[BaseException], BaseException, TracebackType], Any]:
    """
    Return an excepthook that will initialise debugpy and then call through to
    its exception handler.
    """
    import debugpy

    original_debugpy_excepthook = sys.excepthook

    def debugpy_excepthook(
        type_: Type[BaseException],
        value: BaseException,
        traceback: TracebackType,
    ) -> Any:
        """
        Callback called when an exception is hit and debugpy is enabled.
        """
        if not debugpy.is_client_connected():
            print("Exception thrown. Waiting for debugpy on port 5678.")
            debugpy.listen(5678)
            debugpy.wait_for_client()

        original_debugpy_excepthook(type_, value, traceback)

    return debugpy_excepthook

sys.excepthook = make_debugpy_excepthook()

raise Exception("test")

Unfortunately if I run it it does wait for the debugger, but then it just prints the traceback in the console. I can't figure out how debugpy actually catches exceptions. Does it even use excepthook? Any help appreciated!

[fabioz]

In the debugger, the exception is tracked by using the tracing hook (i.e.: sys.settrace), by doing some analysis to determine if at a given frame an exception would be caught/uncaught, so, when it gets to the excepthook, it's already too late for the debugger to actually handle it.

Although there's no public API for that, with some incantations you can make it show the exception through the excepthook though:

import sys
from types import TracebackType
from typing import Any, Callable, Type

def make_debugpy_excepthook() -> Callable[[Type[BaseException], BaseException, TracebackType], Any]:
    """
    Return an excepthook that will initialise debugpy and then call through to
    its exception handler.
    """
    import debugpy

    original_debugpy_excepthook = sys.excepthook

    def debugpy_excepthook(
        type_: Type[BaseException],
        value: BaseException,
        traceback: TracebackType,
    ) -> Any:
        """
        Callback called when an exception is hit and debugpy is enabled.
        """
        if not debugpy.is_client_connected():
            print("Exception thrown. Waiting for debugpy on port 5678.")
            debugpy.listen(5678)
            debugpy.wait_for_client()

        import pydevd
        import threading

        py_db = pydevd.get_global_debugger()
        thread = threading.current_thread()
        additional_info = py_db.set_additional_thread_info(thread)
        additional_info.is_tracing += 1
        try:
            arg = (type_, value, traceback)
            py_db.stop_on_unhandled_exception(py_db, thread, additional_info, arg)
        finally:
            additional_info.is_tracing -= 1
        original_debugpy_excepthook(type_, value, traceback)

    return debugpy_excepthook

sys.excepthook = make_debugpy_excepthook()

raise Exception("test")

[Timmmm]

This worked perfectly, thank you so much!!

[wookayin]

@fabioz Thanks so much for sharing the snippet. I extracted the core lines as a helper function for my use cases (I don't use excepthook but am trying to write a pytest plugin as in #1103) -- Entering a "post-mortem" debugging mode works perfectly for me too.

def maybe_debugpy_postmortem(excinfo):
    """Make the debugpy debugger enter and stop at a raised exception.

    excinfo: A (type(e), e, e.__traceback__) tuple. See sys.exc_info()
    """
    import threading
    try:
        import pydevd
    except ImportError:
        return  # debugpy or pydevd not available, do nothing

    py_db = pydevd.get_global_debugger()
    if py_db is None:
        return  # Do nothing if not running with a DAP debugger
    thread = threading.current_thread()
    additional_info = py_db.set_additional_thread_info(thread)
    additional_info.is_tracing += 1
    try:
        py_db.stop_on_unhandled_exception(py_db, thread, additional_info, excinfo)
    finally:
        additional_info.is_tracing -= 1

UPDATE: Followed a suggestion in #1103 (comment). We can make the code fail-safe as well by raising an error when pydevd or debugpy is not available, depending on use cases, but since this is a proof-of-concept implementation exception is catched.

[int19h]

I think #1201 should cover the original scenario.

[brechtm]

This was working great for me before, but recently I'm running into a problem. After attaching VSCode to the process and opening the debugging console, I get the following message:

pydevd: Something is wrong. Tracker being added twice to the same thread id.

At this point, it's not possible to inspect variables in the debugger.

[Andrej730]

Very useful snippet for postmortem debugging from fabioz.

Though would be great if it's possible to include it somehow in api to make it available as part of the package, as it is generally useful.

[brechtm]

I was able to fix the issue I was running into before. I also added some bells and whistles. Since it may be useful to other people as well, I'm sharing my code in this gist: Python post-mortem debugging with Visual Studio Code.

[Andrej730]

Snippet from @fabioz worked really well for me, but it seems to break in Python 3.12.
Haven't investigated in details yet - any ideas for a workaround?

uv venv --python 3.12
.venv\Scripts\activate
uv pip install debugpy
python test.py
# 2.04s - Error getting exception type.
# Traceback (most recent call last):
#   File ".venv\Lib\site-packages\debugpy\_vendored\pydevd\_pydevd_bundle\pydevd_comm.py", line 1550, in build_exception_info_response
#     stype = frames_list.exc_type.__qualname__
#             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
# AttributeError: 'NoneType' object has no attribute '__qualname__'

# test.py
def test():
    for i in range(5):
        print("Hello, world!", i)
        if i == 3:
            raise ValueError("Test exception")

def setup_post_portem():
    import sys
    from types import TracebackType
    from typing import Any, Callable, Type

    def make_debugpy_excepthook() -> Callable[[Type[BaseException], BaseException, TracebackType], Any]:
        """
        Return an excepthook that will initialise debugpy and then call through to
        its exception handler.
        """
        import debugpy

        original_debugpy_excepthook = sys.excepthook

        def debugpy_excepthook(
            type_: Type[BaseException],
            value: BaseException,
            traceback: TracebackType,
        ) -> Any:
            """
            Callback called when an exception is hit and debugpy is enabled.
            """
            if not debugpy.is_client_connected():
                print("Exception thrown. Waiting for debugpy on port 5678.")
                debugpy.listen(5678)
                debugpy.wait_for_client()

            import pydevd
            import threading

            py_db = pydevd.get_global_debugger()
            thread = threading.current_thread()
            additional_info = py_db.set_additional_thread_info(thread)
            additional_info.is_tracing += 1
            try:
                arg = (type_, value, traceback)
                py_db.stop_on_unhandled_exception(py_db, thread, additional_info, arg)
            finally:
                additional_info.is_tracing -= 1
            original_debugpy_excepthook(type_, value, traceback)

        return debugpy_excepthook

    sys.excepthook = make_debugpy_excepthook()

if __name__ == "__main__":
    setup_post_portem()
    test()