RAG with Azure AI Search: Session Resources

[multispark]

This is the resources and discussion thread for this live stream: https://aka.ms/raghack/aisearch
It will be updated with links after the session.

Recording: https://www.youtube.com/watch?v=ys9CYtqP2Ow
Slides: View PPT in browser

Please ask any follow-up questions here, and the speakers will see them.

[namtran6701]

Can we extract page numbers as an index for each chunk when using the Text Split skill? Also, is this possible with the doc intelligence web API skill?

[pamelafox]

Are you referring to the built-in skills for integrated vectorization? As far as I understand, none of the built-in skills include page number extraction. There's a thread about it here, from devs trying to use integrated vectorization with our RAG sample:
Azure-Samples/azure-search-openai-demo#1380
I'll ask the AI Search team if there are other options.

[namtran6701]

Thank you so much Pamela!

[pamelafox]

I got a further response from the AI Search team:

The few suggested ways today to get the pages is:

• Through AI Document intelligence, using a custom skill, they would just need to modify this to not only use this vs. splitting the docs with split skill, but they can get the page number as required: azure-search-vector-samples/demo-python/code/indexers/document-intelligence-custom-skill/document-intelligence-custom-skill.ipynb at main · Azure/azure-search-vector-samples (github.com)
• Through their own custom skill, using the library/method of their preference to extract the data, they could use GPT-4o as data extraction method with a custom skill and change the prompt to retrieve the page too. They can use this sample code for the extraction and adapt the outputs for a custom skill: liamca/GPT4oContentExtraction: Using Azure OpenAI GPT 4o to extract information such as text, tables and charts from Documents to Markdown (github.com)

[namtran6701]

Very helpful. Thank you so much, Pamela! I will try to integrate it to my own workflow

[bhanukacodes]

Are there any techniques that can be used to dynamically improve the accuracy of RAG outputs based on feedback (e.g., ratings, reactions) from the end-user?

[pamelafox]

It's definitely a good practice to get feedback from users about what responses they consider to be low quality, so that you can incorporate those QA pairs into evaluation pipelines when considering changes. I've also seen several platforms for A/B experimentation, like Langfuse, which you could use for dynamically ramping up to whichever experiment was doing better.
I'm not sure about actual dynamic responses though, as you'd need to know what parameters to change to improve the output. I'll ask around to see what other folks have seen.

[bhanukacodes]

Thank you very much for the response @pamelafox. Could you please explain a bit more about what you meant by incorporating those QA pairs into evaluation pipelines?

[pamelafox]

Here's a blog post where I discuss an overall offline evaluation approach:
https://blog.pamelafox.org/2024/01/evaluating-rag-chat-app-approach-sdks.html

We first generate ground truth data and then evaluate the live RAG app against those ground truth pairs. However, the best ground truth data reflects what users are asking, so ideally you update the ground truth QA pairs with examples plucked from users, along with the correct answers for them. That ensures your evaluated QA pair set doesn't "drift" away from actual user input.

[bhanukacodes]

Thank you very much for the explanation @pamelafox.

[zzulueta]

Just a comment. I created an index using Azure AI Studio and the field containing the embedding is not embedding but contentVector

So I modified the code appropriately:
r = search_client.search(
search_text=None,
top=5,
vector_queries=[VectorizedQuery(
vector=search_vector,
k_nearest_neighbors=5,
fields="contentVector")])

[pamelafox]

Yes, the field name is up to the developer. We use "embedding" in azure-search-openai-demo, but application developers may pick whatever name they prefer. You may also add multiple embedding fields, like titleEmbedding and descriptionEmbedding, for example. Lots of flexibility!

[zzulueta]

In the presentation, it was mentioned that
"The Azure AI Search team even did their own research to determine the optimal input size to send to LLMs. Here, you can see that they found the best recall with an input size of 512 tokens. They also researched the best strategy for splitting the chunks, and found the best recall when they preserved sentence boundaries and overlapped each chunk by 25%."

Is this the current setting for the Import and Vectorize built in feature in Azure AI Search now?

[pamelafox]

I believe they have configurable settings for the chunking, as described in this article:
https://learn.microsoft.com/en-us/azure/search/vector-search-how-to-chunk-documents

They have options for the size of the chunks and the page overlap percentage. Unfortunately, that chunking skill is currently character-based, not token-based, so you will need to make a good decision based on the language of your documents and the standard token/word ratio. (More on that in the internationalization session this week!)
I have put in a feature request to the AI Search team to implement token-based chunking for that skill.