There’s a reason why the open llm leaderboard was changed a while ago.
Basically, scores didn’t improve much anymore and many tests were contained in the training data.See this blogpost for more info.
“close to meaningless” sums up my expert opinion on the whole current AI hype machine sales pitch.
Highly tuned models for incredibly specific, not-dangerous use cases is the next pragmatic step. There’s a lot to excited about, in that very narrow band.
Anyone selling more than that is part of a con, or in very rare cases, doing genuine “fuck off and ask me again in a decade” kinds of research.
Goodhart’s law:
When a measure becomes a target, it ceases to be a good measure.
The Turing Test (as some people believe it to be): if you can have a conversation with a computer and not tell if it’s a computer, then it must be intelligent.
AI companies: writes ML model that is specifically designed to convincingly play one side of a conversation, even though it has no ability to understand the things it talks about.
It’s worth emphasizing that the “Turing Test” is not a good test since it’s not at all scientific.
It’s just another thought experiment that grifters have taken to the bank.
Also as Turing proposed it it’s meant to be infinitely repeatable. The test isn’t supposed to just be if a machine can convince one person with one conversation. That would be trivial. The real Turing test is the converse, it says that there should be no conversation one could have with the machine where it wouldn’t convince you it’s a human.
Much like IQ tests for humans are flawed too. Figuring out series of numbers or relations in a graphic representation, only tells how good you are at these specific tasks, and doesn’t provide a reliable picture of “general” intelligence.
