model_worker raises error

[wangxiyuan]

Case: Deploy fastchat with fastchat.serve.model_worker

Problem: Not sure what input raised the error, but the log shows below, it'll cause an error that the Semaphore lock will not released. When this kind of error raised 5 times, the log will show that Semaphore(value=0, locked=True), then all job will be blocked.

2023-09-07 13:42:01 | INFO | stdout | INFO:     10.244.0.1:3993 - "POST /worker_generate_stream HTTP/1.1" 200 OK
2023-09-07 13:42:01 | ERROR | stderr | ERROR:    Exception in ASGI application
2023-09-07 13:42:01 | ERROR | stderr | Traceback (most recent call last):
2023-09-07 13:42:01 | ERROR | stderr |   File "/usr/local/miniconda3/envs/torch_npu/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi
2023-09-07 13:42:01 | ERROR | stderr |     result = await app(  # type: ignore[func-returns-value]
2023-09-07 13:42:01 | ERROR | stderr |   File "/usr/local/miniconda3/envs/torch_npu/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
2023-09-07 13:42:01 | ERROR | stderr |     return await self.app(scope, receive, send)
2023-09-07 13:42:01 | ERROR | stderr |   File "/usr/local/miniconda3/envs/torch_npu/lib/python3.8/site-packages/fastapi/applications.py", line 292, in __call__
2023-09-07 13:42:01 | ERROR | stderr |     await super().__call__(scope, receive, send)
2023-09-07 13:42:01 | ERROR | stderr |   File "/usr/local/miniconda3/envs/torch_npu/lib/python3.8/site-packages/starlette/applications.py", line 122, in __call__
2023-09-07 13:42:01 | ERROR | stderr |     await self.middleware_stack(scope, receive, send)
2023-09-07 13:42:01 | ERROR | stderr |   File "/usr/local/miniconda3/envs/torch_npu/lib/python3.8/site-packages/starlette/middleware/errors.py", line 184, in __call__
2023-09-07 13:42:01 | ERROR | stderr |     raise exc
2023-09-07 13:42:01 | ERROR | stderr |   File "/usr/local/miniconda3/envs/torch_npu/lib/python3.8/site-packages/starlette/middleware/errors.py", line 162, in __call__
2023-09-07 13:42:01 | ERROR | stderr |     await self.app(scope, receive, _send)
2023-09-07 13:42:01 | ERROR | stderr |   File "/usr/local/miniconda3/envs/torch_npu/lib/python3.8/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
2023-09-07 13:42:01 | ERROR | stderr |     raise exc
2023-09-07 13:42:01 | ERROR | stderr |   File "/usr/local/miniconda3/envs/torch_npu/lib/python3.8/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
2023-09-07 13:42:01 | ERROR | stderr |     await self.app(scope, receive, sender)
2023-09-07 13:42:01 | ERROR | stderr |   File "/usr/local/miniconda3/envs/torch_npu/lib/python3.8/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
2023-09-07 13:42:01 | ERROR | stderr |     raise e
2023-09-07 13:42:01 | ERROR | stderr |   File "/usr/local/miniconda3/envs/torch_npu/lib/python3.8/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
2023-09-07 13:42:01 | ERROR | stderr |     await self.app(scope, receive, send)
2023-09-07 13:42:01 | ERROR | stderr |   File "/usr/local/miniconda3/envs/torch_npu/lib/python3.8/site-packages/starlette/routing.py", line 718, in __call__
2023-09-07 13:42:01 | ERROR | stderr |     await route.handle(scope, receive, send)
2023-09-07 13:42:01 | ERROR | stderr |   File "/usr/local/miniconda3/envs/torch_npu/lib/python3.8/site-packages/starlette/routing.py", line 276, in handle
2023-09-07 13:42:01 | ERROR | stderr |     await self.app(scope, receive, send)
2023-09-07 13:42:01 | ERROR | stderr |   File "/usr/local/miniconda3/envs/torch_npu/lib/python3.8/site-packages/starlette/routing.py", line 69, in app
2023-09-07 13:42:01 | ERROR | stderr |     await response(scope, receive, send)
2023-09-07 13:42:01 | ERROR | stderr |   File "/usr/local/miniconda3/envs/torch_npu/lib/python3.8/site-packages/starlette/responses.py", line 277, in __call__
2023-09-07 13:42:01 | ERROR | stderr |     await wrap(partial(self.listen_for_disconnect, receive))
2023-09-07 13:42:01 | ERROR | stderr |   File "/usr/local/miniconda3/envs/torch_npu/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 597, in __aexit__
2023-09-07 13:42:01 | ERROR | stderr |     raise exceptions[0]
2023-09-07 13:42:01 | ERROR | stderr |   File "/usr/local/miniconda3/envs/torch_npu/lib/python3.8/site-packages/starlette/responses.py", line 273, in wrap
2023-09-07 13:42:01 | ERROR | stderr |     await func()
2023-09-07 13:42:01 | ERROR | stderr |   File "/usr/local/miniconda3/envs/torch_npu/lib/python3.8/site-packages/starlette/responses.py", line 262, in stream_response
2023-09-07 13:42:01 | ERROR | stderr |     async for chunk in self.body_iterator:
2023-09-07 13:42:01 | ERROR | stderr |   File "/usr/local/miniconda3/envs/torch_npu/lib/python3.8/site-packages/starlette/concurrency.py", line 63, in iterate_in_threadpool
2023-09-07 13:42:01 | ERROR | stderr |     yield await anyio.to_thread.run_sync(_next, iterator)
2023-09-07 13:42:01 | ERROR | stderr |   File "/usr/local/miniconda3/envs/torch_npu/lib/python3.8/site-packages/anyio/to_thread.py", line 33, in run_sync
2023-09-07 13:42:01 | ERROR | stderr |     return await get_asynclib().run_sync_in_worker_thread(
2023-09-07 13:42:01 | ERROR | stderr |   File "/usr/local/miniconda3/envs/torch_npu/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
2023-09-07 13:42:01 | ERROR | stderr |     return await future
2023-09-07 13:42:01 | ERROR | stderr |   File "/usr/local/miniconda3/envs/torch_npu/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 807, in run
2023-09-07 13:42:01 | ERROR | stderr |     result = context.run(func, *args)
2023-09-07 13:42:01 | ERROR | stderr |   File "/usr/local/miniconda3/envs/torch_npu/lib/python3.8/site-packages/starlette/concurrency.py", line 53, in _next
2023-09-07 13:42:01 | ERROR | stderr |     return next(iterator)
2023-09-07 13:42:01 | ERROR | stderr |   File "/opt/FastChat/fastchat/serve/model_worker.py", line 240, in generate_stream_gate
2023-09-07 13:42:01 | ERROR | stderr |     for output in self.generate_stream_func(
2023-09-07 13:42:01 | ERROR | stderr |   File "/usr/local/miniconda3/envs/torch_npu/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 35, in generator_context
2023-09-07 13:42:01 | ERROR | stderr |     response = gen.send(None)
2023-09-07 13:42:01 | ERROR | stderr |   File "/opt/FastChat/fastchat/serve/inference.py", line 245, in generate_stream
2023-09-07 13:42:01 | ERROR | stderr |     if i == max_new_tokens - 1:
2023-09-07 13:42:01 | ERROR | stderr | UnboundLocalError: local variable 'i' referenced before assignment

[wangxiyuan]

This is what I changed to fix the error. in quick. It looks works well in my env.
wangxiyuan@7a9018f

[merrymercy]

@wangxiyuan Good catch. Could you send a pull request to fix this?

[wangxiyuan]

@merrymercy Hi, my code is a quick fix. Not sure if it's the best way since I didn't spend much time on analysis.
#2404

Feel free to reject if the fix is not right.