FalkorDB’s Post

https://lnkd.in/eQH-TdP6 I found this explanation of Vector Databases very comprehensive. It reminds the importance of vector embeddings in RAG built on top of LLMs to reduce their hallucinations.

https://lnkd.in/eCWzYEkB Another view why we need rag and vector db in gen-ai :)

https://lnkd.in/g7sKiSbk Nathan Smith explains how to enhance #semanticsearch in #RetrievalAugmentedGeneration (#RAG) applications by using #Neo4j with #GraphDataScience to extract and cluster topics from documents, enabling more relevant document retrieval through #vectorsimilarity. #graphdatabases #llms #largelanguagemodels #knowledgegraphs #kgs

🔎 𝐇𝐲𝐛𝐫𝐢𝐝 𝐑𝐞𝐭𝐫𝐢𝐞𝐯𝐚𝐥! With the rise of Retrieval Augmented Generation (RAG), people are looking for practical ways to improve retrieval (finding relevant documents for a query). Hybrid Retrieval is a technique that tries to combine the strengths of 🔑 keyword-based approaches (BM25) ⬈ vector/semantic retrieval. Today we have a new #haystack tutorial covering this topic, crafted just for you by our 💙 contributor Nicola Procopio! 🖥️ Check it out: https://lnkd.in/dfe5c49E For a deeper understanding of Hybrid Retrieval, don't miss the comprehensive blog post by Isabelle Nguyen: 📚 https://lnkd.in/dW2ecrvR #retrieval #rag #vectorsearch #semanticsearch #neuralsearchpills

𝐇𝐚𝐧𝐝𝐬-𝐨𝐧 𝐓𝐞𝐱𝐭 𝐄𝐦𝐛𝐞𝐝𝐝𝐢𝐧𝐠 𝐀𝐩𝐥𝐥𝐢𝐜𝐚𝐭𝐢𝐨𝐧s 𝐸𝑚𝑏𝑒𝑑𝑑𝑖𝑛𝑔𝑠 𝑎𝑟𝑒 𝑛𝑢𝑚𝑒𝑟𝑖𝑐𝑎𝑙 𝑟𝑒𝑝𝑟𝑒𝑠𝑒𝑛𝑡𝑎𝑡𝑖𝑜𝑛𝑠 𝑜𝑓 𝑡𝑒𝑥𝑡 𝑖𝑛 𝑡ℎ𝑒 𝑓𝑜𝑟𝑚 𝑜𝑓 𝑣𝑒𝑐𝑡𝑜𝑟𝑠 𝑡ℎ𝑎𝑡 𝑐𝑎𝑝𝑡𝑢𝑟𝑒 𝑡ℎ𝑒 𝑠𝑒𝑚𝑎𝑛𝑡𝑖𝑐 𝑚𝑒𝑎𝑛𝑖𝑛𝑔 𝑜𝑓 𝑝𝑎𝑟𝑎𝑔𝑟𝑎𝑝ℎ𝑠 𝑡ℎ𝑟𝑜𝑢𝑔ℎ 𝑡ℎ𝑒𝑖𝑟 𝑝𝑜𝑠𝑖𝑡𝑖𝑜𝑛 𝑖𝑛 𝑎 ℎ𝑖𝑔ℎ-𝑑𝑖𝑚𝑒𝑛𝑠𝑖𝑜𝑛𝑎𝑙 𝑣𝑒𝑐𝑡𝑜𝑟 𝑠𝑝𝑎𝑐𝑒. It is a method of extracting features text or documents so that we can input those features into a machine-learning model to work with text data. They try to preserve syntactical and semantic information. 𝐌𝐞𝐭𝐡𝐨𝐝𝐬 𝐨𝐟 𝐓𝐞𝐱𝐭 𝐄𝐦𝐛𝐞𝐝𝐝𝐢𝐧𝐠 - Bag Of Words(Bow) - Term frequency-inverse document frequency (TF-IDF) - Word2Vec - Continuous Bag of Words(CBOW) - Skip-Gram - GloVe - Fasttext - BERT (Bidirectional Encoder Representations from Transformers) 𝐒𝐨𝐦𝐞 𝐀𝐩𝐩𝐥𝐢𝐜𝐚𝐭𝐢𝐨𝐧𝐬 𝐨𝐟 𝐓𝐞𝐱𝐭 𝐄𝐦𝐛𝐞𝐝𝐝𝐢𝐧𝐠 - Semantic Search - Clustering - Anamoly / Outlier detection - Classification - Retrieval 𝐼𝑛 𝑡ℎ𝑖𝑠 𝑛𝑜𝑡𝑒𝑏𝑜𝑜𝑘, 𝐼 𝑢𝑠𝑒𝑑 𝑆𝑒𝑛𝑡𝑒𝑛𝑐𝑒𝑇𝑟𝑎𝑛𝑠𝑓𝑜𝑟𝑚𝑒𝑟 𝑓𝑜𝑟 𝑒𝑚𝑏𝑒𝑑𝑑𝑖𝑛𝑔 𝑎𝑛𝑑 𝑝𝑒𝑟𝑓𝑜𝑟𝑚𝑖𝑛𝑔 ℎ𝑎𝑛𝑑𝑠-𝑜𝑛 𝑡𝑒𝑥𝑡 𝑒𝑚𝑏𝑒𝑑𝑑𝑖𝑛𝑔 𝑎𝑝𝑝𝑙𝑖𝑐𝑎𝑡𝑖𝑜𝑛𝑠. GitHub Link --------- https://lnkd.in/gQ8gyUgj

Cool article from amazing engineers Rithvik Panchapakesan and Danmei Xu about search, embeddings and quantization! These are the folks powering Snowflake's semantic search! #snowflake #vectorsearch #ai

Vector databases have been gaining a lot of fame recently, with many companies raising hundreds of millions of dollars to build them. They are being called a new type of database for the AI era, but for some projects, they might be an overkill solution. Using a traditional database or even just a numpy ND array might work just fine. However, there's no denying that vector databases are extremely fascinating, especially when you want to give large language models like gpt4 long-term memory. Firstly, over 80 percent of the data out there is unstructured, such as social media posts, images, videos, or audio data. You cannot easily fit them into a relational database. But if you want to put an image into a relational database to search for similar images, you often manually assign keywords or tags to it. We can find a different representation to store the data, which brings us to vector embeddings and vector databases. A vector database indexes and stores vector embeddings for fast retrieval and similarity search. It uses clever algorithms to calculate vector embeddings, which is done by machine learning models. A vector embedding is a list of numbers that represent the data in different ways. For example, you can calculate an embedding for a single word, a whole sentence, or an image. Performing a query across thousands of vectors based on its distance metric would be extremely slow, and this is why those vectors also need to be indexed. The indexing process maps the vectors to a new data structure that enables faster searching. We can use vector databases to equip large language models with long-term memory, semantic search when we are searching based on the meaning or context of our question, similarity search for images, audio or video data, and as a ranking and recommendation engine for online retailers. There are several vector databases available, such as Pinecone, vv8, Chroma, Redis, Cool trans, Milvus, and Vespa AI. They each have different features and characteristics, so it's worth exploring them before choosing the right one for your project. https://lnkd.in/dHgU6eSK

Optimise RAG retrieval with this technique 👇 🧱 Chunking is one of the ways you can improve retrieval by optimising how you split out bodies of text. We have reviewed different chunking methodologies for you & determined the best ones Check out the research from Kristóf Horváth & Mór Kapronczay 👉 Review of different chunking methodologies 👉 Performance on different datasets 👉 The impact of reranking on performance  👉 Best performing reranking models Now on VectorHub 🥳 https://lnkd.in/ee-H58kV #rag #retrieval #vectorsearch