Charles Chevalier

❓ So, how “intelligent” is AI really (Part 2: GPT-4o vs. Claude 3.5 Sonnet)❓ A week ago, I tested GPT-4o's "intelligence" using Bongard Problems. To recap... 📑 From Wikipedia: “The idea of a Bongard problem is to present two sets of relatively simple diagrams, say A and B. All the diagrams from set A have a common factor or attribute, which is lacking in all the diagrams of set B. The problem is to find, or to formulate, convincingly, the common factor.” Out of the 4 Bongard Problems presented to GPT-4o, it was only able to correctly solve one...not great! Today, Anthropic release Claude 3.5 Sonnet which scores highly on a selection of LLM benchmark problem sets (interestingly, the two benchmarks it does not get top scores on belong to GPT-4o). Other announced features of the model include: ✅ Improved response speed (2x) over Claude 3 Opus ✅ Improved vision model (we'll see about that!) ✅ "Artifacts" for "build[ing] upon Claude's creations in real-time" (I'm not sure how this is different than existing conversational AI, but I also haven't played with it) Anyway...how did Claude 3.5 Sonnet do compared to GPT-4o in the Bongard Problem test? Check out the results below! #ai #artificialintelligence #claude3 #gpt #llm

2

listen hear speak opensource AI: kyutai:

9

💥 New day, new model 💥 We ran our performance benchmarks on Anthropic Claude 3.5 Sonnet v/s OpenAI's GPT-4o & the results weren't what you'd expect ⏬ Anthropic says Claude 3.5 Sonnet outperforms all competitor models on key evaluations, at twice the speed of Claude 3 Opus and one-fifth the cost. In our benchmarks 3.5 Sonnet & 4o came out neck & neck. Ultimately, you need to run evaluations on your task at hand and measure the quality yourself. 🔍 Data Extraction: Both models identified 60-80% of data correctly, but neither excelled in this task. 📊 Classification: Claude 3.5 Sonnet (72%) outperformed GPT-4o (65%) in mean accuracy. However, GPT-4o led in precision (86.21%), crucial for accurately classifying customer tickets. 🧠 Verbal Reasoning: GPT-4o outperformed Claude 3.5 Sonnet with 69% accuracy, versus 44%. Both models did well on analogy and relationship questions. 📜 Context window: Claude 3.5 Sonnet boasts a larger context window (200K) compared to GPT-4o's 128K 🌪 Latency: Claude 3.5 Sonnet is 2x faster than Claude 3 Opus, but it’s still lags behind GPT-4o when it comes to latency (50% faster Time to First Token) Price and throughput for these models was very similar Want to learn more about how these models stack up? Check out our full analysis on the Vellum blog: https://lnkd.in/dwEcQJe4 Oh and did I say, Claude 3.5 Sonnet is already in Vellum? Sign up for a demo to run your evals & start building!

68

1 commentaire