Skip to content

Commit f8015f0

Browse files
committed
fix tqdm in notebook for multiprocessing
1 parent f62f124 commit f8015f0

File tree

1 file changed

+4
-0
lines changed

1 file changed

+4
-0
lines changed

src/nlp/arrow_dataset.py

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1165,6 +1165,10 @@ def _map_single(
11651165
# Reduce logging to keep things readable in multiprocessing with tqdm
11661166
if rank is not None and get_verbosity() < WARNING:
11671167
set_verbosity_warning()
1168+
# Print at least one thing to fix tqdm in notebooks in multiprocessing
1169+
# see https://github.com/tqdm/tqdm/issues/485#issuecomment-473338308
1170+
if rank is not None and "notebook" in tqdm.__name__:
1171+
print(" ", end="", flush=True)
11681172

11691173
if function is None:
11701174
function = lambda x: x # noqa: E731

0 commit comments

Comments
 (0)