Mac running error (--device mps running error; --device cpu can run)

[Phil-U-U]

Running python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --device mps --load-8bit led to TypeError: forward() got an unexpected keyword argument 'padding_mask'.

[Phil-U-U]

The error comes from the following code (Line 116) in this file /fastchat/model/monkey_patch_non_inplace.py where
def replace_llama_attn_with_non_inplace_operations():
"""Avoid bugs in mps backend by not using in-place operations."""
transformers.models.llama.modeling_llama.LlamaAttention.forward = forward

the customized forward method does not have the input argument 'padding_mask' as the LlamaAttention.forward do.

Adding this line "padding_mask: Optional[torch.LongTensor] = None, " as the last input argument for the customized forward function solves the issue.

[Phil-U-U]

@merrymercy Can I make a pull request to fix the bug?

[merrymercy]

Yes. Please submit a pull request

[Phil-U-U]

@merrymercy Cool. PR submitted. Thanks for the reply 🤝

[Phil-U-U]

@merrymercy Thanks for the pushing up to the main branch. I'm gonna close this issue. Cheers!