🚀 Jeremy ARANCIO

700-Slides Slide Deck: Why LLM AI Agents Can't KNOW, While Human Agents Can..: SSRN Download PDF https://lnkd.in/e9-2fPdF YouTube Video: https://lnkd.in/eVZmjeHk : "You can know the #name of a bird in all the languages of the world, but when you're finished, you'll #know absolutely #nothing whatever about the bird... So let's look at the bird and see what it's #doing -- that's what #counts." -- "What is #Science?" The #Physics Teacher, Richard P. Feynman So, 'seeing' a trillion 'pictures of a cat' doesn't make [human or AI agent] any more knowledgeable about a cat... try pulling the tail of a cat [better, try "catching tiger by its tail"] and you'll KNOW! Nor does showing a trillion 'pictures of a cat' [to human or AI agent] make one really KNOW "what a cat is" or make one an expert in "herding cats" as many "cat herders" know... Now having read what you just read, unless you know about the '#interpretive #flexibility' of the #text-based #MEANINGS of the quoted TEXT "catching tiger by its tail", "herding cats" and "cat herders", you [as well as AI-agent] will not be even aware of how this phrase of text-based-words applies to #diverse #contexts that have NOTHING TO DO WITH 'cat #pictures' AND 'cats' [or 'tigers']. So that is the DIFFERENCE between #KNOWING #text-based #WORDS vs their REAL #MEANING in #diverse #contexts! See, e.g.: The Joint Staff The #JointStaff: Existential Survival: Far From Madding Crowd Echo Chambers: https://lnkd.in/g_rPnJpg #AWS #Partner #Interview: "Walking the Walk" vs. "Talking the Talk": https://lnkd.in/exMTBcyP "In the eyes of the wise, #Knowledge & #Action are one!" Above observation from a Systems Theorist (inspiration for my PhD R&D on #AI-#ML, #IT-#KM, #RiskManagement, #Controls, #StructuralEquation #Models) sparked interesting dialog at late-1990s #OrganizationScience Winter Conference in Keystone, CO, where I was a final year PhD fellow with top senior #scholars, #practitioners and #policymakers well known in the worlds of higher #academia, #policy, & #practice of the discipline just born then known as #KnowledgeManagement (KM). A key question posed by a most cited Management scholar at that time was: 'Even though I am publishing studies of practitioners, do I have real Knowledge...' Coming from global #IT-#Management practices as the youngest top-ranked KM scholar-practitioner and a proponent of #STRONG #EMPIRICISM, I think of #KNOWLEDGE as ALWAYS rooted in #ACTIONS-and-#BEHAVIORS. It is ONLY in #DOING and #LIVING, that we #KNOW! 1993-Current: R&D: #Meaning-Aware #Human-Centered AI inspired by #AI-#GA pioneer John Holland https://lnkd.in/gcYqAWqm. AWS-Quantum Valley Building the Future of AI-Quantum Networks: Global Risk Management Network LLC-NY Silicon Valley's Next Big Thing™: CEO-CTO-CFO Know-Build-Monetize™ Networks: Join The CxO Metaverse™ C4I-Cyber Quantum Valley-SiliconValley Digital Pioneer USAF-USSF Ventures Engineering Sustainability

9

Reasoning models are really interesting: https://lnkd.in/dEbydJHZ Whom it's for: "...and by developers in all fields to build and execute multi-step workflows" Learning to reason with LLMs: https://lnkd.in/dgnBiXfU

1

To quote Yann LeCun , "Fine-tuned open source models outperform generalist proprietary models on any specific task." This is no different when it comes to using LLM's as evaluators. Excited to share my recent article on fine-tuning models for LLM based evaluation. As part of my work with GroundedAI, I am training and open sourcing smaller, more efficient models that are competing with, and even surpassing, larger generalized black box models. Check it out! #ai #openai #llms #microsoftresearch #datascience #machinelearning #llama3 #whatinspiresme #startups #finetuning

5

𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 𝗠𝗲𝘁𝗿𝗶𝗰𝘀 𝗳𝗼𝗿 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝘃𝗲 𝗠𝗼𝗱𝗲𝗹𝘀 𝟭. 𝗜𝗻𝗰𝗲𝗽𝘁𝗶𝗼𝗻 𝗦𝗰𝗼𝗿𝗲 (𝗜𝗦) Use case: Image generation tasks, particularly when working with labeled datasets. Imagine you're developing a GAN to generate images of different dog breeds. The Inception Score would help you assess: How realistic the generated dog images look and whether your model is producing a diverse range of breeds, rather than just a few. 𝟮. 𝗙𝗿é𝗰𝗵𝗲𝘁 𝗜𝗻𝗰𝗲𝗽𝘁𝗶𝗼𝗻 𝗗𝗶𝘀𝘁𝗮𝗻𝗰𝗲 (𝗙𝗜𝗗) Use case: Photorealistic image generation Let's say you're creating a model to generate human faces for a video game. FID would help you evaluate: How closely the generated faces resemble real human faces. Whether the distribution of features (like eye color, hair style) matches that of real faces. 𝟯. 𝗟𝗼𝗴-𝗟𝗶𝗸𝗲𝗹𝗶𝗵𝗼𝗼𝗱 Use case: Language models and text generation Suppose you're building a chatbot that generates responses to customer inquiries. Log-likelihood would help you assess: How probable (how natural-sounding) the generated responses are and whether the model can generate coherent and contextually appropriate text. 𝟰. 𝗥𝗲𝗰𝗼𝗻𝘀𝘁𝗿𝘂𝗰𝘁𝗶𝗼𝗻 𝗘𝗿𝗿𝗼𝗿 Use case: Autoencoders for image denoising or compression Imagine you're developing an autoencoder to remove noise from medical images. Reconstruction error would help you evaluate: How accurately your model can reconstruct the original, clean image from a noisy input and the level of detail preserved in the reconstruction process. 𝟱. 𝗞𝗲𝗿𝗻𝗲𝗹 𝗗𝗲𝗻𝘀𝗶𝘁𝘆 𝗘𝘀𝘁𝗶𝗺𝗮𝘁𝗶𝗼𝗻 (𝗞𝗗𝗘) Use case: Anomaly detection in sensor data Let's say you're building a model to detect anomalies in temperature readings from industrial equipment. KDE would help you: Estimate the probability distribution of normal temperature readings and identify readings that fall outside this distribution, potentially indicating equipment malfunction. 𝟲. 𝗛𝘂𝗺𝗮𝗻 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 Use case: Creative content generation, such as music or art Suppose you're developing an AI that generates original melodies. Human evaluation would be crucial to assess: The musical quality and appeal of the generated melodies and how well the AI captures specific styles or emotions in the music. You can always use multiple metrics to get a comprehensive understanding of your model's performance. The choice of metrics should align with your specific goals and the nature of your generative task. #GenerativeAI #EvaluatingGenerativeModels #AIEvaluationMetrics #MachineLearning #DeepLearning #ArtificialIntelligence #GANs #VAEs #NeuralNetworks #AIResearch

41

4 komentáře

Debugging "AI" models is not always trivial. If you have worked with torch, you have likely met an error like this. I spent some time figuring out what was wrong and could not find anything as everything passed checks as required. I decided to go for a run and took a second look at the preprocessing steps of the data. It turned out to be a simple case of calling a global variable which corrupted the data. Fixing this felt great and reminded me of my earlier days building open source projects. This is why I love building products, it expands my thought process and teaches me something new. What is a bug you solved that felt great? #ai #aicommunity #huggingface #datascience

1