Use the HuggingFace model intfloat/e5-mistral-7b-instruct for sentence embedding?

[grossjohann]

I have modified serving.properties for the HuggingFace model intfloat/e5-mistral-7b-instruct and I'm not clear if this made any sense at all.

I ran djl-convert -m intfloat/e5-mistral-7b-instruct -o model/e5-mistral-7b-instruct and then use the following to point to the resulting model:

HfZooProvider zooProvider = new HfZooProvider();
Criteria<String, float[]> criteria = Criteria.builder()
		.optModelPath(Path.of(".../model/e5-mistral-7b-instruct"))
		.optApplication(Application.NLP.TEXT_EMBEDDING)
		.setTypes(String.class, float[].class)
		.optTranslatorFactory(new DeferredTranslatorFactory())
		.build();

Then I got this error message:

Exception in thread "main" java.lang.IllegalArgumentException: Invalid pooling model, must be one of [mean, max, cls, mean_sqrt_len, weightedmean].

I see that djl-convert put pooling=lasttoken in serving.properties.

After some searching around, other djl-convert produced models that did work for text embedding didn't have the pooling line in serving.properties and it seems mean is the default value. So I hand-edited the file to say pooling=mean and now the Java code loads the model correctly.

But is this going to produce any embeddings that make sense?

Where can I start reading to understand what this line is doing and whether it might make sense?

The HuggingFace page for this model has some (Python) code using SentenceTransformers and some more code using AutoTokenizer and AutoModel but I'm not able to map that to DJL.

[frankfliu]

intfloat/e5-mistral-7b-instruct uses last_tokens_pool, somehow it's missed in TextEmbeddingTranslator, will take a look add the missing implementation