By combining the strengths of traditional text-based search algorithms with the visual recognition capabilities of deep learning models, hybrid vector search allows users to search for products using a combination of text and images. This can be especially useful for product searches, where customers may not know the exact name or details of the item they are looking for.

This project is inspired from here and creates a small web app using Flask to showcase a hybrid e-commerce search engine.

Before you jump in to use this app, start by reading what Pinecone is and what are Vector Databases.

This project uses open source models from Huggingface. CLIP for creating the embeddings for the images and A multilingual clip model for the text queries.

We will also use the fashion dataset for this app.

pip install -r requirements.txt

If you have not already setup Pinecone, use this Quickstart guide to get your free API key. The api key need to be set in the python environment as "PINECONE_API_KEY"

Next, run the setup file while which will download the dataset from HuggingFace and create the vector embeddings:

python setup.py

This could take a while to run

After the setup is complete, run the flask app

flask --app app.py --debug run 

and open http://127.0.0.1:5000/ after the app starts