Skip to content

Commit 9b91433

Browse files
committed
Added description for StreamHandlerWithBuffer
1 parent cbaa773 commit 9b91433

File tree

1 file changed

+134
-1
lines changed

1 file changed

+134
-1
lines changed

README.md

Lines changed: 134 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ Even multiprocessing logging becomes a breeze.
2121
* [Temporary disable logging](#8-temporary-disable-logging)
2222
* [Limit traceback line length](#9-limit-traceback-lines-length)
2323
* [Analyze `raise` exception line](#10-analyze-raise-exception-line)
24+
* [StreamHandler with buffer](#11-streamhandler-with-buffer)
2425

2526
* [Sample config](#sample-config)
2627
* [YAML format](#1-yaml-format)
@@ -611,6 +612,111 @@ RuntimeError: Too much laughing with a=haha and b=hihi
611612
setup_logging(analyze_raise_statement=True)
612613

613614

615+
### 11. StreamHandler with buffer:
616+
This handler is mainly to solve the problem of outputting a tremendous amount of logs to GUI applications in real-time.
617+
618+
GUI applications use threading to display content while listening for user input (button click, key pressing, mouse scroll).
619+
But since python actually running only one thread at a time due to the GIL (global interpreter lock),
620+
processing to display a tremendous amount of logs to the GUI widget will lock a thread for quite a long time.
621+
During this time, no user input will be handled and the app seems unresponsive.
622+
623+
The answer to this problem is to cache the log and output them at once after some interval or number of cached lines reached a threshold.
624+
This significantly reduces the overhead on the widget side and makes the app responsive.
625+
The solution is implemented in the new `StreamHandlerWithBuffer` which is inside `logger_tt.handlers`.
626+
There are 2 steps to use this handler.
627+
628+
* config the handler in the `log_config` file.
629+
630+
```yaml
631+
handlers:
632+
buffer_stream_handler:
633+
class: logger_tt.handlers.StreamHandlerWithBuffer,
634+
level: DEBUG,
635+
formatter: brief,
636+
stream: ext://sys.stdout,
637+
buffer_time: 0.5,
638+
buffer_lines: 0,
639+
debug: False
640+
641+
root:
642+
level: DEBUG
643+
handlers: [console, error_file_handler, buffer_stream_handler]
644+
```
645+
646+
Its parameters look exactly like the `console` handler except the last 3 ones.
647+
* `buffer_time`: the cache time interval in **seconds** before it flush the log out
648+
* `buffer_line`: the **number of line** to cache before it flush the log out
649+
* `debug`: log the time that it flush the log out or not<br>
650+
651+
For `buffer_line`, to avoid the last lines of log not printed out as the number of line is below threshold,
652+
you should set `buffer_time` to a certain number too.
653+
654+
Then, you also need to add this handler to the root logger's handlers.
655+
656+
657+
* replace the `stream`:
658+
you need to replace the `stdout` with your GUI widget stream.
659+
You cannot do this in the config file since your widget is only appear during runtime.
660+
You have to do this in the code.
661+
662+
```python
663+
config = setup_logging(config_path='log_config.json')
664+
config.replace_handler_stream(index=HANDLER_INDEX, stream=WIDGET_STREAM)
665+
```
666+
Setup your logging as usual with your config file,
667+
then call `replace_handler_stream` to replace the with your widget's stream.
668+
* `index`: the index of the handler, in the root logger' `handlers` list, that you want to replace the stream
669+
* `stream`: the stream that you want to put into this handler.
670+
671+
672+
Below is the example with `PySimpleGUI v4.57.0` with the above config.
673+
```python
674+
import time
675+
from threading import Thread
676+
677+
import PySimpleGUI as sg
678+
from logger_tt import setup_logging, logger
679+
680+
stop_flag = False
681+
682+
# GUI config
683+
sg.theme('DarkAmber')
684+
layout = [
685+
[[sg.Button("Show Log", key="-show_log-"),
686+
sg.Button("stop", key="-stop-", button_color='red')],
687+
sg.Multiline(size=(80, 30), font='Courier 10', key='log', autoscroll=True)]
688+
]
689+
window = sg.Window('Logging tool', layout, finalize=True)
690+
691+
# logging config
692+
config = setup_logging(config_path='log_config.json')
693+
config.replace_handler_stream(index=2, stream=window["log"])
694+
695+
696+
def my_func():
697+
"""long running task in the background"""
698+
while not stop_flag:
699+
for i in range(1000):
700+
logger.warning(f" {i} Function is empty")
701+
time.sleep(1)
702+
703+
704+
while True:
705+
event, values = window.read()
706+
if event == sg.WINDOW_CLOSED:
707+
break
708+
if event == "-show_log-":
709+
logger.info(__name__)
710+
Thread(target=my_func).start()
711+
if event == "-stop-":
712+
stop_flag = True
713+
logger.info(f"stop button pressed")
714+
715+
window.close()
716+
```
717+
You could set `index=0` to use the normal `StreamHandler` and
718+
see the difference while clicking the `stop` button for yourself.
719+
614720

615721
# Sample config:
616722
Below are default config files that used by `logger-tt`. You can copy and modify them as needed.
@@ -647,6 +753,15 @@ handlers:
647753
backupCount: 15
648754
encoding: utf8
649755
when: midnight
756+
757+
buffer_stream_handler:
758+
class: logger_tt.handlers.StreamHandlerWithBuffer,
759+
level: DEBUG,
760+
formatter: brief,
761+
stream: ext://sys.stdout,
762+
buffer_time: 0.5,
763+
buffer_lines: 0,
764+
debug: False
650765
651766
loggers:
652767
urllib3:
@@ -710,7 +825,17 @@ logger_tt:
710825
"backupCount": 15,
711826
"encoding": "utf8",
712827
"when": "midnight"
713-
}
828+
},
829+
830+
"buffer_stream_handler": {
831+
"class": "logger_tt.handlers.StreamHandlerWithBuffer",
832+
"level": "INFO",
833+
"formatter": "simple",
834+
"stream": "ext://sys.stdout",
835+
"buffer_time": 0.5,
836+
"buffer_lines": 0,
837+
"debug": false
838+
}
714839
},
715840
716841
"loggers": {
@@ -747,6 +872,14 @@ logger_tt:
747872
```
748873

749874
# Changelog
875+
## 1.7.0
876+
* Fixed:
877+
* multiprocessing: log file rollover fails as child process keep opening the file.
878+
* multiprocessing: if the log path is set by variable with time,
879+
child process creates a new redundant log path.
880+
* New functionality: Added `StreamHandlerWithBuffer`.
881+
GUI app could use this handler to keep the app responsive while having a tremendous log output.
882+
750883
## 1.6.1
751884
* Added `limit_line_length` parameter: log only maximum `n` characters for each traceback line.
752885
This prevents dumping the whole huge content of the variable into the log. `n=1000` by default.

0 commit comments

Comments
 (0)