AI helper for teachers.

An AI project where students can then ask the teacher a question on Discord, and the AI will respond with a video of the teacher answering the question. The answers are based on recordings what the teacher said in class.

• 👂 Whisper from OpanAI to transcribe the audio
• 🦙 [LlamaIndex] (https://www.llamaindex.ai/) to look up relevant transcript snippets
• 🧠 GPT-4 to write the answers
• 👄 ElevenLabs to generate the audio
• 🧑‍🏫 D-ID to generate the video
• 💬 Discord for the students to interact with the service
• 🐍 Coded in Python

It can be easily adapted to other teachers, or really any other character. Here is a demo video of how it looks like.

Over the last few months, I recorded audio files from lessons in Cynthia’s and Judith’s classes, transcribed them, divided the transcripts into chunks, and stored them in a database (Steps A-G in the diagram below).

When a user asks a relevant question on Discord (starting with “Hi Cynthia” or “Hi Judith”) the code searches the database of embeddings to find the chunks with the highest similarity (Steps 1-3). It then provides Chat GPT with these chunks to write the answer and uses ElevenLabs with the right teacher’s voice model (Steps 4-6). Then the audio is combined with a photo of the teacher and lip movements are added. This makes it look as if the teacher is answering the question (Steps 6-7). Then the video is sent to the user who can watch the AI teacher answer the questions on Discord. (Step 8)

• Whisper Whisper is a powerful model developed by OpenAI that transcribes spoken language into text. It is the first step in processing the input by converting spoken questions into a written format that other models can understand.

• lLama Index and Open Ai's embedding model Llama Index leverages OpenAI's advanced embedding model to provide context-rich interpretations of the transcribed text. This method allows for a deeper understanding of the questions posed, enhancing the ability to generate relevant and accurate responses based on the curriculum or educational goals.

• GPT4 Once the context is established via Llama Index and OpenAI's embeddings, GPT (Generative Pre-trained Transformer) is used to generate the final answers. GPT models are adept at producing coherent and contextually appropriate text, making them ideal for crafting educational responses that are both informative and engaging.

• Eleven Labs Eleven Labs technology is used to convert the text responses generated by GPT into high-quality, lifelike audio. This makes the responses more engaging and easier for students to understand, mirroring a natural teacher-student interaction.

• D-ID D-ID is utilized to animate the responses, providing a visual representation of the teacher's avatar speaking. It helps in creating a more personalized and engaging learning experience by animating the voice responses into synchronized lip movements on a digital avatar, enhancing the interactivity of the tool.

• Discord Discord acts as the user interface where students can submit their questions and receive answers. It is crucial for interaction management, providing a familiar and accessible platform for users to engage with AI Teacher Helper.

  

If I ask the question ‘Hi Cynthia, Tell me about Alexander the Great’ it will first say one of 3 acknowledgment phrases like ‘Stop rushing me, your teacher and I are trying to find the answer to your question.’ then it will prepare the file and say ‘Here's your file!’ and send you the file!

The file will be a video of Cynthia saying “Alexander the Great was a historical figure who traveled along the Silk Road and was responsible for opening trade routes. He is known for his conquests and for spreading Greek culture throughout the regions he conquered.”

• Marco Mascorro: He was my mentor for this project and helped me with the code and helped me with the ideas around this project, plus helped me make answers for questions people might have when I present it.
• Both my teachers gave me content for the audio files and let me make them the guinea pigs for this.