Skip to content

Conversation

key4ng
Copy link
Collaborator

@key4ng key4ng commented Aug 20, 2025

Motivation

Enable sgl-router to use OpenAI-compatible APIs as backend services, allowing seamless integration with OpenAI, OCI GenAI, or any OpenAI-compatible endpoints. This provides flexibility to route requests to external AI services beyond traditional SGLang workers.

Modifications

This initial PR support the chat completion.

  • New OpenAI Router (src/routers/openai_router.rs): Routes chat/completion requests to OpenAI API
  • OpenAI Worker Type (src/core/worker.rs): Added WorkerType::OpenAI with dedicated health checks and circuit breaker
  • Router Factory (src/routers/factory.rs): Extended to create OpenAI routers based on configuration
  • Configuration (src/config/types.rs): Added RoutingMode::OpenAI with API key, model, and base URL settings

Tests

Command used:

cargo run -- --openai-backend --model openai.gpt-4o-mini --base-url https://inference.generativeai.us-chicago-1.oci.oraclecloud.com/20231130/actions --api-key sk-***TmR --port 30000

Test ran:

curl -X POST http://localhost:30000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "openai.gpt-4o-mini",
      "messages": [
        {
          "role": "user",
          "content": "Hello! Can you tell me a short joke?"
        }
      ],
      "max_tokens": 100,
      "temperature": 0.7
    }'
{"id":"chatcmpl-C77Fhc7lkKJcbjxQxmCnZpARv0Gco","choices":[{"index":0,"message":{"content":"Sure! Why did the scarecrow win an award? \n\nBecause he was outstanding in his field!","refusal":null,"tool_calls":null,"role":"assistant","function_call":null,"audio":null},"finish_reason":"stop","logprobs":null}],"created":1755812209,"model":"openai.gpt-4o-mini","service_tier":"default","system_fingerprint":"fp_51db84afab","object":"chat.completion","usage":{"prompt_tokens":17,"completion_tokens":20,"total_tokens":37,"prompt_tokens_details":{"audio_tokens":0,"cached_tokens":0},"completion_tokens_details":{"accepted_prediction_tokens":0,"audio_tokens":0,"reasoning_tokens":0,"rejected_prediction_tokens":0}}}

Benchmarking and Profiling

Checklist

@key4ng key4ng changed the title [wip] Add OpenAI backend [router] Add OpenAI backend support - core function Aug 21, 2025
@key4ng key4ng marked this pull request as ready for review August 21, 2025 22:39
@key4ng key4ng marked this pull request as draft August 22, 2025 00:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants