@@ -21,6 +21,7 @@ Even multiprocessing logging becomes a breeze.
2121 * [ Temporary disable logging] ( #8-temporary-disable-logging )
2222 * [ Limit traceback line length] ( #9-limit-traceback-lines-length )
2323 * [ Analyze ` raise ` exception line] ( #10-analyze-raise-exception-line )
24+ * [ StreamHandler with buffer] ( #11-streamhandler-with-buffer )
2425
2526* [ Sample config] ( #sample-config )
2627 * [ YAML format] ( #1-yaml-format )
@@ -611,6 +612,111 @@ RuntimeError: Too much laughing with a=haha and b=hihi
611612 setup_logging(analyze_raise_statement=True)
612613
613614
615+ ### 11. StreamHandler with buffer:
616+ This handler is mainly to solve the problem of outputting a tremendous amount of logs to GUI applications in real-time.
617+
618+ GUI applications use threading to display content while listening for user input (button click, key pressing, mouse scroll).
619+ But since python actually running only one thread at a time due to the GIL (global interpreter lock),
620+ processing to display a tremendous amount of logs to the GUI widget will lock a thread for quite a long time.
621+ During this time, no user input will be handled and the app seems unresponsive.
622+
623+ The answer to this problem is to cache the log and output them at once after some interval or number of cached lines reached a threshold.
624+ This significantly reduces the overhead on the widget side and makes the app responsive.
625+ The solution is implemented in the new ` StreamHandlerWithBuffer ` which is inside ` logger_tt.handlers ` .
626+ There are 2 steps to use this handler.
627+
628+ * config the handler in the ` log_config ` file.
629+
630+ ``` yaml
631+ handlers :
632+ buffer_stream_handler :
633+ class : logger_tt.handlers.StreamHandlerWithBuffer,
634+ level : DEBUG,
635+ formatter : brief,
636+ stream : ext://sys.stdout,
637+ buffer_time : 0.5,
638+ buffer_lines : 0,
639+ debug : False
640+
641+ root :
642+ level : DEBUG
643+ handlers : [console, error_file_handler, buffer_stream_handler]
644+ ` ` `
645+
646+ Its parameters look exactly like the ` console` handler except the last 3 ones.
647+ * `buffer_time`: the cache time interval in **seconds** before it flush the log out
648+ * `buffer_line`: the **number of line** to cache before it flush the log out
649+ * `debug`: log the time that it flush the log out or not<br>
650+
651+ For `buffer_line`, to avoid the last lines of log not printed out as the number of line is below threshold,
652+ you should set `buffer_time` to a certain number too.
653+
654+ Then, you also need to add this handler to the root logger's handlers.
655+
656+
657+ * replace the `stream`:
658+ you need to replace the `stdout` with your GUI widget stream.
659+ You cannot do this in the config file since your widget is only appear during runtime.
660+ You have to do this in the code.
661+
662+ ` ` ` python
663+ config = setup_logging(config_path='log_config.json')
664+ config.replace_handler_stream(index=HANDLER_INDEX, stream=WIDGET_STREAM)
665+ ` ` `
666+ Setup your logging as usual with your config file,
667+ then call `replace_handler_stream` to replace the with your widget's stream.
668+ * `index`: the index of the handler, in the root logger' `handlers` list, that you want to replace the stream
669+ * `stream`: the stream that you want to put into this handler.
670+
671+
672+ Below is the example with `PySimpleGUI v4.57.0` with the above config.
673+ ` ` ` python
674+ import time
675+ from threading import Thread
676+
677+ import PySimpleGUI as sg
678+ from logger_tt import setup_logging, logger
679+
680+ stop_flag = False
681+
682+ # GUI config
683+ sg.theme('DarkAmber')
684+ layout = [
685+ [[sg.Button("Show Log", key="-show_log-"),
686+ sg.Button("stop", key="-stop-", button_color='red')],
687+ sg.Multiline(size=(80, 30), font='Courier 10', key='log', autoscroll=True)]
688+ ]
689+ window = sg.Window('Logging tool', layout, finalize=True)
690+
691+ # logging config
692+ config = setup_logging(config_path='log_config.json')
693+ config.replace_handler_stream(index=2, stream=window["log"])
694+
695+
696+ def my_func():
697+ """long running task in the background"""
698+ while not stop_flag:
699+ for i in range(1000):
700+ logger.warning(f" {i} Function is empty")
701+ time.sleep(1)
702+
703+
704+ while True:
705+ event, values = window.read()
706+ if event == sg.WINDOW_CLOSED:
707+ break
708+ if event == "-show_log-":
709+ logger.info(__name__)
710+ Thread(target=my_func).start()
711+ if event == "-stop-":
712+ stop_flag = True
713+ logger.info(f"stop button pressed")
714+
715+ window.close()
716+ ` ` `
717+ You could set `index=0` to use the normal `StreamHandler` and
718+ see the difference while clicking the `stop` button for yourself.
719+
614720
615721# Sample config:
616722Below are default config files that used by `logger-tt`. You can copy and modify them as needed.
@@ -647,6 +753,15 @@ handlers:
647753 backupCount: 15
648754 encoding: utf8
649755 when: midnight
756+
757+ buffer_stream_handler:
758+ class: logger_tt.handlers.StreamHandlerWithBuffer,
759+ level: DEBUG,
760+ formatter: brief,
761+ stream: ext://sys.stdout,
762+ buffer_time: 0.5,
763+ buffer_lines: 0,
764+ debug: False
650765
651766loggers:
652767 urllib3:
@@ -710,7 +825,17 @@ logger_tt:
710825 "backupCount": 15,
711826 "encoding": "utf8",
712827 "when": "midnight"
713- }
828+ },
829+
830+ "buffer_stream_handler": {
831+ "class": "logger_tt.handlers.StreamHandlerWithBuffer",
832+ "level": "INFO",
833+ "formatter": "simple",
834+ "stream": "ext://sys.stdout",
835+ "buffer_time": 0.5,
836+ "buffer_lines": 0,
837+ "debug": false
838+ }
714839 },
715840
716841 "loggers": {
@@ -747,6 +872,14 @@ logger_tt:
747872` ` `
748873
749874# Changelog
875+ # # 1.7.0
876+ * Fixed:
877+ * multiprocessing: log file rollover fails as child process keep opening the file.
878+ * multiprocessing: if the log path is set by variable with time,
879+ child process creates a new redundant log path.
880+ * New functionality: Added `StreamHandlerWithBuffer`.
881+ GUI app could use this handler to keep the app responsive while having a tremendous log output.
882+
750883# # 1.6.1
751884* Added `limit_line_length` parameter: log only maximum `n` characters for each traceback line.
752885 This prevents dumping the whole huge content of the variable into the log. `n=1000` by default.
0 commit comments