[New Model]: Multimodal Embedding Model GME.

[Adenialzz]

The model to consider.

Multimodal Embedding Model GME.
https://huggingface.co/collections/Alibaba-NLP/gme-models-67667e092da3491f630964d6

The closest model vllm already supports.

Qwen2 VL、VLM2Vec

What's your difficulty of supporting the model you want?

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[xsank]

In fact, the Alibaba gme model have the same arch as qwen2-vl, which means the vllm already support the model you mentioned, just have a try.

[Adenialzz]

Thanks for your suggestion! Yes. Same arch with qwen2 vl. But how can I get embeddings like gme_inference.py?

[DarkLight1337]

The model seems to apply last pooling and normalization. You can simply set the task to "embed" which should use those settings automatically: https://docs.vllm.ai/en/latest/models/pooling_models.html

[Adenialzz]

Thanks! I'll try this.

[Adenialzz]

INFO: 10.203.37.211:0 - "POST /v1/embeddings HTTP/1.1" 500 Internal Server Error

ERROR: Exception in ASGI application

Traceback (most recent call last):

File "/opt/conda/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi

result = await app( # type: ignore[func-returns-value]

File "/opt/conda/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__

return await self.app(scope, receive, send)

File "/opt/conda/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__

await super().__call__(scope, receive, send)

File "/opt/conda/lib/python3.10/site-packages/starlette/applications.py", line 112, in __call__

await self.middleware_stack(scope, receive, send)

File "/opt/conda/lib/python3.10/site-packages/starlette/middleware/errors.py", line 187, in __call__

raise exc

File "/opt/conda/lib/python3.10/site-packages/starlette/middleware/errors.py", line 165, in __call__

await self.app(scope, receive, _send)

File "/opt/conda/lib/python3.10/site-packages/starlette/middleware/cors.py", line 85, in __call__

await self.app(scope, receive, send)

File "/opt/conda/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 62, in __call__

await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)

File "/opt/conda/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app

raise exc

File "/opt/conda/lib/python3.10/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app

await app(scope, receive, sender)

File "/opt/conda/lib/python3.10/site-packages/starlette/routing.py", line 714, in __call__

await self.middleware_stack(scope, receive, send)

File "/opt/conda/lib/python3.10/site-packages/starlette/routing.py", line 734, in app

await route.handle(scope, receive, send)

File "/opt/conda/lib/python3.10/site-packages/starlette/routing.py", line 288, in handle

await self.app(scope, receive, send)

File "/opt/conda/lib/python3.10/site-packages/starlette/routing.py", line 76, in app

await wrap_app_handling_exceptions(app, request)(scope, receive, send)

File "/opt/conda/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app

raise exc

File "/opt/conda/lib/python3.10/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app

await app(scope, receive, sender)

File "/opt/conda/lib/python3.10/site-packages/starlette/routing.py", line 73, in app

response = await f(request)

File "/opt/conda/lib/python3.10/site-packages/fastapi/routing.py", line 301, in app

raw_response = await run_endpoint_function(

File "/opt/conda/lib/python3.10/site-packages/fastapi/routing.py", line 212, in run_endpoint_function

return await dependant.call(**values)

File "/opt/conda/lib/python3.10/site-packages/vllm/entrypoints/utils.py", line 56, in wrapper

return handler_task.result()

File "/opt/conda/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 475, in create_embedding

generator = await handler.create_embedding(request, raw_request)

File "/opt/conda/lib/python3.10/site-packages/vllm/entrypoints/openai/serving_embedding.py", line 118, in create_embedding

) = await self._preprocess_chat(

File "/opt/conda/lib/python3.10/site-packages/vllm/entrypoints/openai/serving_engine.py", line 454, in _preprocess_chat

if request.mm_processor_kwargs is not None:

File "/opt/conda/lib/python3.10/site-packages/pydantic/main.py", line 891, in __getattr__

raise AttributeError(f'{type(self).__name__!r} object has no attribute {item!r}')

AttributeError: 'EmbeddingChatRequest' object has no attribute 'mm_processor_kwargs'

Hi, I got this error when request to gme embed server following here. How can I solve it?

[DarkLight1337]

You need to update the version of vLLM. This issue was fixed in a recent version (forgot which one)

[Adenialzz]

Thanks, I updated vLLM and it works. It seems that openai api only support for text embedding, so we have to use raw http request for multimodal embedding? How can I pass mm_processor_kwargs with raw requests?

[DarkLight1337]

See https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html#multi-modal-inputs

[wujiekd]

See https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html#multi-modal-inputs

I noticed that the embedding model does not support direct input of Image read by PIL.IMAGE.

@Adenialzz Have you tried any relevant methods and solved this problem?

[DarkLight1337]

You can look at these examples to convert PIL image into base64 URL: https://docs.vllm.ai/en/latest/serving/multimodal_inputs.html

[wujiekd]

You can look at these examples to convert PIL image into base64 URL: https://docs.vllm.ai/en/latest/serving/multimodal_inputs.html

I tried to support it without any problem, but I tested that the embeddings with and without image output are the same, indicating that the model did not process images. Here is my code, can you help me take a look?

`def encode_image_base64(
image: Image.Image,
image_mode: str = "RGB",
format: str = "JPEG",) :

image_format=format

with BytesIO() as buffer:
    image = image.convert(image_mode)
    image.save(buffer, image_format)
    data = buffer.getvalue()

return base64.b64encode(data).decode('utf-8')

base64_image = encode_image_base64(Image.open(images[0]))
outputs = llm.embed(
[
{
"prompt": "What is this?",
"image_embeds": f"data:image/jpeg;base64,{base64_image}",
},
{
"prompt": "What is this?",
}
]
)

for o in outputs:
generated_embedding = o.outputs.embedding
print(sum(generated_embedding))`

[DarkLight1337]

You should be passing "image" key instead of "image_embeds" key. The "image_embeds" key is for inputting embeddings, not outputting them.

[wujiekd]

You should be passing "image" key instead of "image_embeds" key. The "image_embeds" key is for inputting embeddings, not outputting them.

Changing to "image" key also produces the same result.

[DarkLight1337]

Also, the root key for passing multi-modal data should be "multi_modal_data". So it should be

outputs = llm.embed(
    [
        {
            "prompt": "What is this?",
            "multi_modal_data": {"image": f"data:image/jpeg;base64,{base64_image}"},
        },
        {
            "prompt": "What is this?",
        }
    ]
)

[wujiekd]

Also, the root key for passing multi-modal data should be "multi_modal_data". So it should be

outputs = llm.embed(
    [
        {
            "prompt": "What is this?",
            "multi_modal_data": {"image": f"data:image/jpeg;base64,{base64_image}"},
        },
        {
            "prompt": "What is this?",
        }
    ]
)

I tried and encountered the following error:
ValueError: You set or defaulted to image=1 in --limit-mm-per-prompt`, but passed 247943 image items in the same prompt.