AI Specialist and Distinguished Engineer (NLP & Search). Inventor of weightwatcher.ai . TEDx Speaker. Need help with AI ? #talkToChuck
From the X-verse. many at open source LLms on the leaderboards appear to be overfit to the leaderboard metrics Here’s an example. The data preparation for wizard-coder uses the human eval pass@1 scores to decide to evolve the dataset further or not Optimizing solely for the test that defeats the point having a test set. https://lnkd.in/gaqyr9Us This is consistent with results from weightwatcher as well when applied to the LoRA updates directly
Pierre-Carl Langlais
11mo
Although many closed LLMs are doing the same but we won’t even know it (so at least kudos to the Wizard team to openly state their own bias)
Igor Carron
11mo
Interesting!
AI Specialist and Distinguished Engineer (NLP & Search). Inventor of weightwatcher.ai . TEDx Speaker. Need help with AI ? #talkToChuck
11moHere's an example from weightwatcher