🔥 How to Learn RAG in 2024: Go from Beginner to Expert (Step by Step) 🚀

#beginners #programming #python #ai

Everyone seems to be worried about how AI can take away our jobs.

But it is surprising how very few people have actually gotten into even the fundamental facets of working with AI models in a real practical setting.

By now, most technical people have heard of RAG - Retrieval Augmented Generation. In simple terms, RAG is just a way to link documents or some knowledge source to AI models.

Sounds easy enough if you're thinking of it with let's say 5 documents and ChatGPT. However, if you think about how anyone, or a company, would need to do this with thousands, tens of thousands, or millions of files, it is a different problem.

This is an issue that almost all companies have. That is why I am a huge advocate for everyone having at least a foundational understanding of what RAG is, because it is one of the fundamental pieces of knowledge you need to work with AI models.

Upskill Your AI Knowledge in 2024

This is what inspired us at LLMWare to create free step-by-step videos in YouTube that teach you the foundational elements of RAG, so that in 7 short videos (ranging from 8 to 15 min each), you can learn RAG. Note: Basic Python is a prerequisite.

As with anything in life, if you take it seriously, this will be the launching point of becoming an AI expert. Even if you are not interested in becoming an AI expert, knowing how all the pieces of RAG works will definitely serve you well as many companies will be incorporating these workflows.

Ready to get started? Introduction

This Introduction to RAG video walks you through the basic components of RAG so you can start your AI journey with LLMWare.

1. Parsing, Text Chunking, Indexing

Create your first library and get started with the basic steps. The documents in your library need to be parsed into a uniform format, and separated into smaller texts (chunking) then indexed with all the metadata. This video will walk you through this step.

2. Build Embeddings

What are embeddings, embedding models, vectors and vector databases? In this tutorial, learn the fundamental concepts behind embeddings, embedding models, vectors and vector databases.

You will build your first embeddings with models from Hugging Face to store to a database and use these embeddings to run your queries.

3. Prompt Models

Learn how to prompt models by loading and prompting models from Hugging Face and OpenAI.

Start inferencing models using this example and see how you can check if the model is providing the right answer using the context that was passed. Learn how to capture model's usage data such as token consumption, the output, and the total processing time.

4. RAG with Text Query

With this video, you will start to search. We will be taking some form of knowledge in a library with embeddings and bring the pieces together with a model.

Learn how to put together the right RAG strategy with a thoughtful retrieval and querying strategy combined with the right model to do the job.

5. RAG with Semantic Query

By now, you're really making progress. You are now ready to start semantic searching. Also known as natural language querying, this is where we reap the benefits of embeddings and vector databases.

You will be able to query your knowledge base using natural language to ask questions to derive answers from even the most complex legal documents.

6. RAG with Multi-Step, Hybrid Query

It's Graduation Day! (If you have been following along this far). We will quickly recap through all that you have learned in the previous videos and learn how to use a quantized DRAGON-YI-6b-GGUF model from Hugging Face on a laptop.

Perform multi-step hybrid queries to get the responses you need.

Also learn how to perform evidence verification (guard against model hallucinations) and how to save all the output as JSON or CSV files for future datasets or audits.

Once you are all done with these videos, you are ready to dig in and play with all the really cool aspects of Generative AI, including building complex Agent workflows.

Please check out our Github and leave a star! https://github.com/llmware-ai/llmware