Arize AI

Tune at 9am PST today as Tuana Çelik and John Gilhuly cover one of the trickiest areas of evaluating agents: identifying when they’ve entered into excessive loops. 😵💫 👉 Break them free!! They'll cover... - Techniques for identifying loop patterns - Diagnostic approaches for understanding loop causes - Strategies for optimizing agents and breaking problematic loops 🔗 Register now for this session and the entire series: https://lnkd.in/gPA2-RZp

Warning: Real humans talking in this podcast, not NotebookLM (yet)! 🙅♂️ 🤖 📒 https://lnkd.in/gwYAbVDa John Gilhuly and I broke down the OpenAI o1 (preview/mini) and some of the learnings from the blog post and benchmarks released. Here are some human generated (but NotebookLM helped!) notes from the podcast: 🔵 OpenAI's new o1 model excels at reasoning, logic, coding, and math problems, surpassing GPT-4 in these areas. 🔵 o1 uses a "chain-of-thought" reasoning process to break down problems into smaller steps, analyze them, and reflect on previous steps to self-correct. We'd love to learn more here, but details are scarce. 🔵 Better at math, but not better at writing: while o1 demonstrates superior performance in logical tasks, it may not be the best choice for creative writing or text generation, where GPT-4 still holds an edge. 🔵 Not great for customer facing products requiring real time interaction (yet): One of the drawbacks of o1 is its slow inference time, making it better for offline tasks that do not require instant responses. 🔵 This is just the beginning: The full release of o1 is yet to come, but the preview version shows promising improvements in safety, potentially reducing the risk of jailbreaks. Follow Deep Papers wherever you get your podcasts for more technical takes on AI research and products. I wonder what we'll cover next? 📒 🤔 ---------- Substack for more AI x Product: https://lnkd.in/dWjxwZp6

Wondering which model to use for Evaluation? 🤔 📊 Samantha White shows how you can take a data-driven approach to selecting and testing eval models using Arize and Phoenix. In <5 minutes, she covers: - How an LLM can be used to evaluate the performance of your application - Key factors to consider when choosing your LLM judge - Quick tips for implementing this approach

A huge thank you to everyone who joined us and Pinecone for some (spicy!) discussions on LLM safety last week in NYC. 🔥 We dove deep into the challenges of real-world AI deployment and strategized on building LLM solutions that are both powerful and responsible. Huge thanks to Bear Douglas and Safeer Mohiuddin for joining Jason Lopatecki for a great fireside chat.

Here's what I learned from building the same AI Agent in 3 different frameworks. Detailed write up: https://lnkd.in/g4fDskmD Full code: https://lnkd.in/gsF85iR5

Thanks to the deepset team for a great webinar! Check it out if you're building Agents with Haystack 🤖 🚜

Tomorrow we're taking a closer look at OpenAI's latest crop of o1 models. We'll also highlight some research our team did to see how o1 stacks up against Sonnet 3.5. 👀 If you’re curious about how these new models perform in practical scenarios and want to talk about what this means for the future of AI development, join John Gilhuly and Aman Khan tomorrow as they dive in. Sign up: https://lnkd.in/dmEY6C8F

Kudos to our partners at Airbyte for the 1.0 release! We're excited to see how this will drive innovation in data and AI. Visit www.airbyte.com/v1 to catch up on all the updates.

Don't skip class tomorrow 👉 Dat Daryl Ngo, Vibhu Sapra, and John Gilhuly are covering (notoriously difficult) agent evaluation--it will be fun tho. Explore metrics, benchmarks, and tools that can help you assess performance and point to problem areas in your application Sign up here: https://lnkd.in/gPA2-RZp

Introducing AI agent search by Arize, another way to debug AI with AI. LLM apps create millions of data points, making it time consuming to debug. You can identify patterns and problems in your data by just asking for what you need. You can search for: 1. frustrated queries 2. hallucinated responses 3. where you made users cry And more! I go through a quick overview of what this looks like below: https://lnkd.in/eZGVyVT8