async method call the sync code, and using semaphore together

[colinguozizhong]

I saw this code in model_worker.py & multi_model_worker.py

@app.post("/worker_generate")
async def api_generate(request: Request):
    params = await request.json()
    await acquire_worker_semaphore()
    worker = worker_map[params["model"]]
    output = worker.generate_gate(params)
    release_worker_semaphore()
    return JSONResponse(output)

In this asynchronous method, call the synchronous code worker.generate_gate(params), it seems to cause service congestion. In this case, await acquire_worker_semaphore() seems to be invalid because there is always only one model running in the service?

is this right？

[merrymercy]

the semaphore is also used for other endpoint such as worker_generate_stream

[lvxuan263]

@merrymercy worker.generate_gate(params) is runnning in main thread. The program is not blocked by semaphore. StreamResponse will start new subthread, so /worker_generate_stream doesn't blocked.

example:
first request: POST /worker_generate , Try to resp for as long as possible
second request: POST /worker_get_status, When the first request is not completed, the second request will be blocked.

solution:
I don't know much about python. Is it elegant in this way? Place generate_gate in non main thread

output = await asyncio.to_thread(worker.generate_gate, params)

thanks.