Skip to content

Config

Jump to bottom
luics edited this page May 22, 2025 · 5 revisions

Config

  • apps/eval/src/config.json5, this file must be created locally

    {
  "models": ["claude-3-5-sonnet-20241022", "openai/gpt-4o"],
  "agentMode": "local"
}

  • apps/eval/.env, this file must be created locally

    ANTHROPIC_API_KEY=xxx
OPENROUTER_API_KEY=xxx
# Comment

# OPENAI_API_KEY=xxx

    XXX_API_KEY is used in apps/eval/src/model.json

    {
    "models": [
      {
        "model": "claude-3-5-sonnet-20241022",
        "apiKey": "{{ANTHROPIC_API_KEY}}"
      },
      {
        "model": "openai/gpt-4o",
        "apiKey": "{{OPENROUTER_API_KEY}}"
      }
    ]
  }

Parameters

parameter type default description
projects string[] [] If empty, all projects will pass. Value is like "@web-bench/calculator".
agentMode "local" | "http" "local"
agentEndPoint string "" When agentMode is set to "http", set http API for network requests.
models string[] [] 'models' field in apps/eval/src/model.json
maxdop number 30 max degree of parallelism
logLevel "info" | "warn" | "debug" | "error" "info"
httpLimit number 10 When agentMode is set to "http", maximum concurrent requests

Evaluation | arXiv Paper | Leaderboard

Clone this wiki locally