Standard tqdm displays on multiple lines when jupyter_server_nbmodel is installed

[eleonorecharles]

Description

When using the standard tqdm progress bar (not the notebook/IPyWidget version) in a Jupyter notebook with jupyter_server_nbmodel installed, the progress bar appears on multiple lines instead of properly overwriting a single line. This creates a cluttered output with each progress update appearing on a new line, making it difficult to read and significantly increasing cell output size.

Reproduce

1. Install the required packages:

pip install jupyter_server_nbmodel[lab] tqdm

2. Start JupyterLab and create a new notebook
3. Execute the following code in a cell:

from tqdm import tqdm  # Standard tqdm, not tqdm.notebook
import time

for i in tqdm(range(100)):
    time.sleep(0.1)  # Simulate long-running process

4. Observe that instead of updating a single progress bar line, each update creates a new line, resulting in multiple lines of output

Expected behavior

The standard tqdm progress bar should display on a single line with updates replacing the previous state (using carriage returns), even when jupyter_server_nbmodel is installed.

[cben]

Can reproduce without external packages.

print(11111, end='\r')
time.sleep(1)
print(2, end=' ')
time.sleep(1)
print(3, end='')
print(4)

• Normal jupyter (uvx --python 3.12 --with "jupyterlab==4.4.0" jupyter server):
  Initially prints 11111, then becomes 2 111 and finally:

  2 341
  

• server-side execution (uvx --python 3.12 --with "jupyterlab==4.4.0,jupyter_server_nbmodel[lab]==0.1.1a2" jupyter server):
  After every sleep(1), starts a new line, resulting in:

  11111
  2 
  34
  

  Without sleep() it results in same 2 341, so the logic for appending to same line, and even interpreting \r exists!

  And small sleeps like 0.1sec sometimes break onto new line, sometimes continue on same line...
  A loop with small sleeps suggests every ~0.2sec get batched into a single line?

[cben]

Giving the server --debug logs the server-side flow — zmq messages and YStore updates + sync to client.

If it's written fast enough, I see one stream message and one output line; with large sleep, 3 stream messages and 3 lines, and even with random behavior with 0.1, it seems the number of stream messages is exactly same as number of resulting lines!

Here is log for execution that resulted in:

2 111
34

with 2 stream messages:

[D 2025-05-25 13:28:26.484 ServerApp.jupyter_server_nb_model] Processing execution request c92e8fe8-c9f2-49ef-980b-e9da0fcace14 for kernel fe6b4c28-a3dc-47d3-bd28-44bfc5ffe373…
[D 2025-05-25 13:28:26.485 ServerApp.jupyter_server_nb_model] c92e8fe8-c9f2-49ef-980b-e9da0fcace14 print(11111, end='\r')
    time.sleep(0.1)
    print(2, end=' ')
    time.sleep(0.1)
    print(3, end='')
    print(4) {'cell_id': '3f8b8d39-0b1c-40c3-acfe-5f480b912b03', 'document_id': 'json:notebook:33bf308e-07e7-4490-bb10-c3b4497842ad', 'record_timing': False}
[D 2025-05-25 13:28:26.485 ServerApp] Sending Y update to client with endpoint: json:notebook:33bf308e-07e7-4490-bb10-c3b4497842ad
[D 2025-05-25 13:28:26.485 ServerApp] Writing Y update to YStore
[D 2025-05-25 13:28:26.488 ServerApp] activity on fe6b4c28-a3dc-47d3-bd28-44bfc5ffe373: status (busy)
[D 2025-05-25 13:28:26.488 ServerApp] activity on fe6b4c28-a3dc-47d3-bd28-44bfc5ffe373: execute_input
[D 2025-05-25 13:28:26.500 ServerApp] Accepting token-authenticated request from 127.0.0.1
[D 2025-05-25 13:28:26.511 ServerApp] 200 GET /api/contents?content=1&hash=0&1748168906495 (4253f280c9044213aca2b1573b9e27bd@127.0.0.1) 11.27ms
[D 2025-05-25 13:28:26.603 ServerApp] Accepting token-authenticated request from 127.0.0.1
[D 2025-05-25 13:28:26.604 Extension] 202 GET /api/kernels/fe6b4c28-a3dc-47d3-bd28-44bfc5ffe373/requests/c92e8fe8-c9f2-49ef-980b-e9da0fcace14?1748168906599 (4253f280c9044213aca2b1573b9e27bd@127.0.0.1) 0.92ms
[D 2025-05-25 13:28:26.689 ServerApp] activity on fe6b4c28-a3dc-47d3-bd28-44bfc5ffe373: stream
[D 2025-05-25 13:28:26.690 ServerApp] Sending Y update to client with endpoint: json:notebook:33bf308e-07e7-4490-bb10-c3b4497842ad
[D 2025-05-25 13:28:26.690 ServerApp] Writing Y update to YStore
[D 2025-05-25 13:28:26.690 ServerApp] activity on fe6b4c28-a3dc-47d3-bd28-44bfc5ffe373: stream
[D 2025-05-25 13:28:26.691 ServerApp] Sending Y update to client with endpoint: json:notebook:33bf308e-07e7-4490-bb10-c3b4497842ad
[D 2025-05-25 13:28:26.691 ServerApp] Writing Y update to YStore
[D 2025-05-25 13:28:26.693 ServerApp] activity on fe6b4c28-a3dc-47d3-bd28-44bfc5ffe373: status (idle)
[D 2025-05-25 13:28:26.694 ServerApp.jupyter_server_nb_model] Execution request c92e8fe8-c9f2-49ef-980b-e9da0fcace14 processed for kernel fe6b4c28-a3dc-47d3-bd28-44bfc5ffe373.
[D 2025-05-25 13:28:26.695 ServerApp] Sending Y update to client with endpoint: json:notebook:33bf308e-07e7-4490-bb10-c3b4497842ad
[D 2025-05-25 13:28:26.695 ServerApp] Writing Y update to YStore

Which suggests the buffering of sufficiently-close writes (with correct \r handling) happens in ipykernel,
whereas the (incorrect) splitting of writes to separate lines happens here in jupyter-server-nbmodel.
(And the client JS is innocent, it just shows what the server put in YStore. I see same stream==lines behavior if I close browser tab during a print & sleep loop.)

---

EDIT: with regular jupyter, same equality holds, which makes sense if buffering happens in ipykernel.

for i in range(200):
    print(i, end=' ')
    import time; time.sleep(0.02)

Again, I see only 20 stream messages, not 200, and I don't see every number appearing, they appear in batches of ~10. Only there it's client JS interpretting stream messages, and it correctly appends to single long line.

=> Buffering is irrelevant, use sleep(1) for repro, bug is in server handling of stream messages not ending in \n.

[cben]

jupyter-server-nbmodel/jupyter_server_nbmodel/actions.py

Lines 107 to 109 in 8bf649c

	# FIXME Logic is quite complex at https://github.com/jupyterlab/jupyterlab/blob/7ae2d436fc410b0cff51042a3350ba71f54f4445/packages/outputarea/src/model.ts#L518
	if text.endswith((os.linesep, "\n")):
	text = text[:-1]

😁 which links to JS logic fixBackspace() and fixCarriageReturn().

---

Is there similar handling in ipykernel which may be easier to steal, cause in Python?

• Here is flush_interval = 0.2. And yes you can see/change sys.stdout.flush_interval at runtime.
• But I don't see \b, \r processing 😕 Buffering is a (per-thread? dict of) StringIO, which does no such processing.

In fact, it's easy to call into ipykernel's code to see how it builds a message:

import sys
print('foo\r', end='')
print('bar', end='')
list(sys.stdout._flush_buffers())

[({'session': '0be01a5e-5e01-4475-9d28-846b7dbe3091',
   'version': '5.2',
   'msg_id': '204d883c-51c9-4539-a1c8-df2f4ab29e1d',
   'msg_type': 'execute_request',
   'subshell_id': None,
   'username': '',
   'date': datetime.datetime(2025, 5, 25, 11, 14, 49, 72000, tzinfo=tzutc())},
  'foo\rbar')]

i.e. buffering/flushing just concatenates them over some time, and carriage return is sent over zmq as-is!?
But then the correct handling of single stream message with e.g. 11111\r2 must happen in jupyter-server-nbmodel? But where? 👀

[cben]

• With server-side execution (adding more logging...):

  When it fits into 1 message, I see zmq message has text=['11111\r2 34'], and resulting Y cell_outputs=[{"text":["11111\r2 34"],"output_type":"stream","name":"stdout"}]. And indeed the saved .ipynb contains cell outputs.text=["11111\r2 34"].
  So it looks like 2 341 only thanks to frontend JS processing \r in the model during rendering

  And with 3 messages, the resulting Y cell_outputs=["11111\r", "2 ", "34"]. And indeed the saved .ipynb contains cell outputs.text=["11111\r", "2 ", "34"].

• With regular jupyterlab:

  When it fits into 1 message, the saved .ipynb cell outputs.text=["2 341\n"] ✨.
  And when it takes 3 messages, the saved .ipynb has same outputs.text=["2 341\n"] 🧹✨!
  So when frontend JS drives execution, it processes \r in stream messages, updating the model each time (with sufficient sleep() I can Ctrl+S the notebook and check representation at each stage)

=> Simply appending without growing cell_outputs could "work", but is not the right solution.

It would lead to notebooks keeping full history of progress bar, growing and maybe slowing down...
And I'm not sure frontends are supposed to process \r during render, it might break in future or jpterm?
The FIXME comment is correct, the right (and compatible to client-side execution) solution is duplicate the JS \r & \b logic in server-side execution logic, updating the Y model in-place.

[aftoul]

I've seen this issue when I used this module on a remote machine. As @cben pointed out, the issue is from the simplistic text handling in _output_hook. I suggest adding some text preprocessing that handles the carriage return to simulate the behavior of a terminal emulator. I wrote a simple implementation for this:

def handle_carriage_return(s: str) -> str:
    """Handle text the same way that a terminal emulator would display it

    Args:
        s: The message
    """
    lines = s.split('\n')
    processed_lines = []

    for line in lines:
        result = []
        i = 0
        while i < len(line):
            if line[i] == '\r':
                # Move cursor to start of the line
                # Reset the result buffer and prepare to overwrite
                i += 1
                overwrite_chars = []
                while i < len(line) and line[i] != '\r':
                    overwrite_chars.append(line[i])
                    i += 1
                for j, c in enumerate(overwrite_chars):
                    if j < len(result):
                        result[j] = c
                    else:
                        result.append(c)
            else:
                result.append(line[i])
                i += 1
        processed_lines.append(''.join(result))

    return '\n'.join(processed_lines)


def handle_backspace(s: str) -> str:
    """Simulate backspaces in the text

    Args:
        s: The message
    """
    new_str = []
    for c in s:
        if c == '\b':
            if len(s) > 0 and new_str[-1] != '\n':
                new_str.pop()
        else:
            new_str.append(c)
    return ''.join(new_str)


def _output_hook(outputs: list[NotebookNode], ycell: y.Map | None, msg: dict) -> None:
    """Callback on execution request when an output is emitted.

    Args:
        outputs: A list of previously emitted outputs
        ycell: The cell being executed
        msg: The output message
    """
    msg_type = msg["header"]["msg_type"]
    if msg_type in ("display_data", "stream", "execute_result", "error"):
        # FIXME support for version
        output = nbformat.v4.output_from_msg(msg)
        outputs.append(output)

        if ycell is not None:
            cell_outputs = ycell["outputs"]
            if msg_type == "stream":
                with cell_outputs.doc.transaction():
                    text = handle_backspace(output["text"])

                    if text.endswith((os.linesep, "\n")):
                        text = text[:-1]

                    if (not cell_outputs) or (cell_outputs[-1]["name"] != output["name"]):
                        output["text"] = [handle_carriage_return(text)]
                        cell_outputs.append(output)
                    else:
                        last_output = cell_outputs[-1]
                        new_text = ""
                        old_text = last_output['text'][-1] if len(last_output['text']) > 0 else ''
                        if '\r' in text or '\r' in old_text:
                            if text[-1] == '\r':
                                text = text[:-1]
                                add = '\r'
                            else:
                                add = ''
                            new_text = handle_carriage_return(old_text + text) + add
                            last_output["text"][-1] = new_text
                        else:
                            last_output["text"].append(text)
                        cell_outputs[-1] = last_output
            else:
                with cell_outputs.doc.transaction():
                    cell_outputs.append(output)

    elif msg_type == "clear_output":
        # FIXME msg.content.wait - if true should clear at the next message
        outputs.clear()

        if ycell is not None:
            del ycell["outputs"][:]

    elif msg_type == "update_display_data":
        # FIXME
        ...

---

The implementation above checks if there was a carriage return in relation to the previous output lines, and adds the correct outputs to the cells. This version works with TQDM and displays the output correctly. I need someone to validate the code sample and I can do a pull request quickly.

[krassowski]

The reference implementation is:

https://github.com/jupyterlab/jupyterlab/blob/9241bcb7a2283e1749ac6bc8934e4eef670ef603/packages/outputarea/src/model.ts#L533-L643

one thing I see at a first glance is that it also handles \b.

[krassowski]

If you would like to open a PR I am happy to test and review!