Skip to content

questions in vicuna_bench with reference_answer cannot run #2508

@toslunar

Description

@toslunar

I'd like to use vicuna_bench. After python gen_api_answer.py --bench-name vicuna_bench --model gpt-3.5-turbo succeeds, python gen_judgment.py --bench-name vicuna_bench --model-list gpt-3.5-turbo failed with

88%|█████████████████████████████████████████████████████████████████████████████████████████▎            | 70/80 [17:27<02:29, 14.97s/it]
Traceback (most recent call last):
  File ".../FastChat/fastchat/llm_judge/gen_judgment.py", line 309, in <module>
    play_a_match_func(match, output_file=output_file)
  File ".../FastChat/fastchat/llm_judge/common.py", line 203, in play_a_match_single
    score, user_prompt, judgment = run_judge_single(
  File ".../FastChat/fastchat/llm_judge/common.py", line 141, in run_judge_single
    kwargs["ref_answer_2"] = ref_answer["choices"][0]["turns"][1]
IndexError: list index out of range

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions