Jumping on the LLM bandwagon with a RAGbot to answer questions on my lecture notes

Unless you have just emerged from a cave you will have heard of ChatGPT, Google Gemini, Claude, etc – collectively LLMs (Large Language Models, also called Generative AI models). Wikipedia calls Google Gemini a generative artificial intelligence chatbot. They are fancy and powerful, and touching many parts of the world including education. My employer (as I am sure many universities are) is trying to keep up.

One way to do this is via what are called RAGbots. Somewhat to my surprise this term seems to be so new that there is, at the time of writing, no Wikipedia page defining RAGbot*. But Microsoft Copilot goes for:

A RAGbot is a chatbot or AI assistant built using Retrieval-Augmented Generation (RAG), which combines a large language model (LLM) with a knowledge retrieval system to deliver accurate, context-aware answers.

• Retrieval: The bot searches through a knowledge base (documents, databases, websites, or user-provided files) to find relevant information.
• Augmented Generation: The retrieved information is then fed into a generative AI model, which crafts a natural, conversational response.
• Result: Instead of relying only on what the model was trained on, the bot can use up-to-date, domain-specific, or private data to answer questions more reliably

In education, the idea would be to make a RAGbot by feeding the learning materials of a module into a RAGbot and then the student can ask questions of the RAGbot. The student can then learn in an interactive way by basically chatting to the RAGbot. This would be more engaging than just reading notes and hopefully if the student has a question about something in the course materials they could just ask the RAGbot.

It is surprisingly easy to make a RAGbot yourself, eg see Thomas Janssen’s Build a RAG in 10 minutes! YouTube video. Unless you are fast coder and are just focused in getting something up and running as fast as possible, then it does take longer than 10 minutes but it is easy to do in an evening. A very simple one as a Jupyter notebook is here.

The basic workflow of what a RAGbot (that uses text from a pdf) does is:

1. Read in the pdf that’s the source of the text you want to input into the RAGbot, and extract the text from the pdf
2. Split text string from the pdf into overlapping chunks. Each chunk is a string of the text of length a few hundred characters long (and with some overlap between successive text chunks)
3. Use the embed function of an LLM (I use a local LLM with ollama) to associate with each text chunk a vector (of length 100+ numbers) that encodes the semantic ‘meaning’ of the chunk in the sense that LLMs encode meaning or content
4. The chunks and associated vectors are then stored in a database (chromadb)
5. The user then asks a question, which itself a string. The ‘meaning’ vector for this string is generated. Then the database is searched for the 2 or 3 chunks of text whose ‘meaning’ vectors are closest to the question text
6. An LLM chatbot (I use a local LLM with ollama) is then fed: “Based on the following text [retrieved text chunks] Answer the query “. This answer uses the LLM algorithm and so its training data, as well as the (hopefully) most relevant text from the document, to produce the answer.

And so you basically just need to convert the above steps into Python, installing ollama (and using it to pull an LLM) and the required Python libraries as you go. A Python Jupyter notebook that does all this is here.

In my hands it seems to work. I fed it some basic notes from a biological physics course, that cover some of basic molecules like DNA and proteins that we are made of, and it could answer simple questions on it.

For example, I asked the RAGbot: “Is DNA a polymer?” The answer is yes and hopefully my notes are clear on this point. The RAGbot answered:

Yes, DNA is a polymer.

The text describes DNA as a polymer made up of a chain of monomers called nucleotides.

which is correct and helpful.

So RAGbots are fun things and I assume their use will rapidly increase inside education and out. RAGbots may also be a way of understanding a bit about how LLMs work, for instance in LLM-language the meaning of the question “Is DNA a polymer?” is the vector of length 384 (using model: all-miniLM):

[-0.04467454, -0.0072343615, -0.02843518, -0.0075524542, -0.050055068, ….]

I don’t know what semantic meaning you assign to asking if the molecule that carries our genes is a polymer, but I have seen what the all-miniLM model thinks the question means and it is a whole lot of numbers.

* I only learnt what a RAGbot is from brother in law, who can write much fancier ones than I can.