Enhancement: Upload Documents as input Context vs RAG Workflow

[achhabra2]

What features would you like to see added?

It would be great if we could upload PDFs or text documents and have it processed as input context versus the current workflow which uses the RAG API instead. Some models now have 200K - 1M context window and we could utilize that outside of pasting large blocks of text into the chat window.

More details

We may need to have a secondary upload button implemented or something that signifies which type of workflow you are using. If you use chatGPT or Gemini web interface, etc. those documents get processed as context.

Below added from #3791

---

In the Claude UI, if you paste a large piece of text, it automatically gets attached and treated like a document. This is very easy to use as I can just paste large pieces of text from different sources, they get treated as separate documents, and then chat with them.

But LibreChat is more like ChatGPT, where any amount of pasted text gets added in the text box like a normal message. So having the above behavior I think is beneficial in many ways, at least as a toggle switch in the settings.

Sorry if this is duplicated; I couldn't find anything like this in the Issues. Loving LibreChat so far; really great alternative to paying for ChatGPT, Claude, and Gemini separately. Thanks!

Paste a large amount of text (threshold could be customizable maybe) and it will get uploaded as a TXT file instead of appearing in the chatbox.

Second, when clicking on such a file, a UI popup opens up where we can check the file.

Which components are impacted by your request?

No response

Pictures

This is what I'm referring to.

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

This request already exists, but your specifications are more clear so I will close the other in favor of yours. Thanks for the write up!

Closing #2335

[raphaelgurtner]

+1, this would be a major improvement, especially for use with gemini 1.5 models with their large context sizes. If it helps: supported mime types for each model can be found here https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/send-multimodal-prompts#media_requirements

[danny-avila]

Google will get a lot of love soon due to their improved dev tools. May have something to do with @logankilpatrick joining the team 😊

[wesselhuising]

Hi, any update on this? I would like to send PDFs to the Gemini 1.5 pro model instead using the RAG API. Cheers.

[marcelamsler]

This would improve the value of LibreChat massively and we are looking forward to this. Is there any timeline you could share?

[schnaker85]

upvote +1 🚀

[amir-ghasemi]

FYI everyone, Librechat already supports this as it comes with a rag_use_full_context flag which puts the entire document into context. One just needs to control this via the .env or add a setting in the UI.

[schnaker85]

FYI everyone, Librechat already supports this as it comes with a rag_use_full_context flag which puts the entire document into context. One just needs to control this via the .env or add a setting in the UI.

https://github.com/danny-avila/LibreChat/blob/main/api/app/clients/prompts/createContextHandlers.js#L25
I think it still requires to use the RAG_API_URL to be setup.

[amir-ghasemi]

FYI everyone, Librechat already supports this as it comes with a rag_use_full_context flag which puts the entire document into context. One just needs to control this via the .env or add a setting in the UI.

https://github.com/danny-avila/LibreChat/blob/main/api/app/clients/prompts/createContextHandlers.js#L25 I think it still requires to use the RAG_API_URL to be setup.

Yes, indeed. We still need the RAG for question answering against knowledgebases consisting of 1000s of documents so that feature should not go away. This flag and the existing endpoint in rag api allows for including the full document in context with minimal changes.

[raphaelgurtner]

FYI everyone, Librechat already supports this as it comes with a rag_use_full_context flag which puts the entire document into context. One just needs to control this via the .env or add a setting in the UI.

A document included in the context in this way is however still subject to any preprocessing/text-extraction on LibreChat's part right? The idea of this feature request would be to circumvent that (if the user so desires) and have the model (endpoint) deal with the document as-is. This would allow many different use-cases / document types, not only pdf but even sound/video/csv etc. - basically whatever is supported by the model-endpoints.

Is that feature considered out of scope (since it's not in the roadmap currently at all) or just low priority? If it's just low priority, would help implementing it still be appreciated?

[danny-avila]

A document included in the context in this way is however still subject to any preprocessing/text-extraction on LibreChat's part right?

No I think it would be nice to have a simple "use full text" option while uploading. If it's text-based, the browser can handle it and the server never interacts with the file, other than adding it to the AI request, it would just get appended to the user message.

[marcelamsler]

Is there an update here? We would love to use Librechat to compare Contracts in PDF Format with Gemini 1.5.