In model_worker /worker_generate API can blocked all request

[lvxuan263]

When Request /worker_generate API and semaphore value bigger than 0, The program is blocked by worker.generate_gate(params) and not await acquire_worker_semaphore(). because worker.generate_gate(params) is runnning in main thread. other request will wait.

/worker_generate_stream doesn't blocked, maybe StreamResponse will start new subthread.

example:
first request: POST /worker_generate , Try to resp for as long as possible
second request: POST /worker_get_status, When the first request is not completed, the second request will be blocked.

solution:
I don't know much about python. Is it elegant in this way? Place generate_gate in non main thread

output = await asyncio.to_thread(worker.generate_gate, params)

thanks.

relate issue: #2285

[merrymercy]

Do you have a script to reproduce this bug?

I can run two requests concurrently without any problems.

python3 -m fastchat.serve.test_message --model-name vicuna-7b-v1.5 --max-new 1024
python3 -m fastchat.serve.test_message --model-name vicuna-7b-v1.5 --max-new 1024

[lvxuan263]

The /worker_generate is NOT ok.
The /worker_generate_stream is fine.

Reproduce:

1. loop.sh

#!/bin/bash
for i in $(seq 1 100)
do
    echo -n "`date`: "
    curl -XPOST http://localhost:21001/worker_get_status 
    echo ""
    sleep 0.5
done

localhost:21001 is model_worker_address. /worker_get_status endpoint are usually not blocked. this script will repeat request this endpoint and echo the response body.

2. fastchat/serve/test_message.py #commit:5573aae4
   change the line 49 code , the endpoint from /worker_generate_stream to /worker_generate

# line 48 - 53
response = requests.post(
    worker_addr + "/worker_generate",
    headers=headers,
    json=gen_params,
    stream=True,
)

then run the scripts in 2 terminal.

# terminal 1 
./loop.sh

# terminal 2
python3 -m fastchat.serve.test_message --model-name vicuna-7b-v1.5 --max-new-tokens 1024

Expect:
Terminal 1 continuously outputs worker status every 0.5 seconds.
Terminal 2 waiting llm generate all text, then display resp text.

Actual:
when terminal2 starting request, terminal1 will be blocked. waiting terminal2 get the resp, the terminal1 will be continue.

[merrymercy]

Aha, I see. It is definitely a problem. I think your solution looks good. Could you submit a PR?

[lvxuan263]

No problem. I'd be happy to contribute.