Skip to content

Conversation

narengogi
Copy link
Collaborator

@narengogi narengogi commented Jul 17, 2024

Title:

  • Gemini on Vertex AI throws an exception, when there is an message part with empty string
  • Also updated tool_config keys to snake_case (it works with camelCase also, but official documentation is in snake_case)
{
    "error": {
        "message": "vertex-ai error: Unable to submit request because it has an empty text parameter. Add a value to the parameter and try again. Learn more: https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/gemini",
        "type": "INVALID_ARGUMENT",
        "param": null,
        "code": "400"
    },
    "provider": "vertex-ai"
}
  • Gateway appends a message with empty string part in between two messages of the same role and it's failing thus
  • Handling this by merging the messages with the same role, as gemini supports multiple parts inside the same message
{
  "model": "gemini-1.5-flash-001",
  "contents": [
    {
      "role": "user",
      "parts": [
        {
          "text": "<|context_start|>\n(Context omitted for historical message)\n<|context_end|>\n\nhello what are the discounts?"
        },
        {
          "text": "I am hungry"
        }
      ]
    },
    {
      "role": "model",
      "parts": [
        {
          "text": "That's okay, all of us get hungry sometimes?"
        }
      ]
    }
  ],
  "systemInstruction": {
    "parts": [
      {
        "text": "You're Sidekick"
      }
    ],
    "role": "system"
  },
  "generationConfig": {
    "temperature": 0.1
  }
}

Related Issues: (optional)

Testing Done:

  • Tested with user and model messages in succession
  • Tested with assistant messages with tool responses and tool messages, verified correct exception is thrown

@narengogi narengogi marked this pull request as ready for review July 17, 2024 07:50
…elCase. It works either ways, but changed because official documentation is in snake_case
@narengogi narengogi force-pushed the fix/vertex-ai-gemini-role-alternation-fix branch from b6d222a to ae11fb9 Compare July 19, 2024 07:35
@VisargD VisargD merged commit d68542c into Portkey-AI:main Jul 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants