chunking_eval_experiment

[mtawbiaws]

Identifying the best documents chunking strategy is key Building GenAI solution powered By RAG

Finding the right combination of the Chunking Strategy, Chunks size and Chunks overlap is a balance between responses accuracy, latency and cost
The goal of the notebook is to define a tool / framework allowing to test quickly multiple combinations of the three parameters above

This notebook demonstrates how to evaluate different chunking strategies, chunk sizing and overlap using RAGA framework for evaluation, Claude 3 Sonnet on Bedrock for LLM, FAISS for Vector Store and RAGA for Evaluation

tools and libraries used

1. LLM : anthropic.claude-3-sonnet on Bedrock
2. Embbeding: amazon.titan-embed-text-v1 on Bedrock
3. Vector Store: FAISS
4. RAG Retriever results evaluation :RAGA  (context_recall,context_precision, answer_relevancy)
5. Evaluated Chunking strategy
   1. NLTKTextSplitter
   2. LatexTextSplitter
   3. MarkdownTextSplitter
   4. RecursiveCharacterTextSplitter
   5. SpacyTextSplitter
6. Dataset: AWS 2019-2022 Shareholder-Letter

Workflow

1. Install required libraries
2. Create a bedrock Client
3. Load documents example in a local folder
4. Use a custom function "textsplitterStrategy" with 3 params : Strategy , chunk size and overlap
5. Run the function for 5 chunking strategies and load the documents in FAISS (5 stores)
6. Use Claude3 Sonnet and RAGA to evaluate context_recall,context_precision, answer_relevancy
7. plot the evaluation results