🪙 feat: Add check_embedding_ctx_length Flag

[Fjf]

When using an OpenAI compatible endpoint with e.g., vLLM, the referenced models may not be one of the default openai models. The langchain OpenAIEmbeddings will then pre-tokenize the input using a local tiktoken tokenizer, before sending this to the endpoint as tokenized input.
This of course will not work if the tokenizer of the vLLM model is different from the openai tokenizer.

Disabling this check will directly send input to the given endpoint without checking the tokenized input first.
For vLLM I have checked, and giving input larger than the context length will not cause any errors, but it will simply be truncated and will return an embedding as expected.

[danny-avila]

@Fjf can you also edit the README to include this new variable?

[Fjf]

@danny-avila updated