Screenshot of this question was making the rounds last week. But this article covers testing against all the well-known models out there.
Also includes outtakes on the ‘reasoning’ models.
Screenshot of this question was making the rounds last week. But this article covers testing against all the well-known models out there.
Also includes outtakes on the ‘reasoning’ models.
I tried this with a local model on my phone (qwen 2.5 was the only thing that would run, and it gave me this confusing output (not really a definite answer…):
it just flip flopped a lot.
E: also, looking at the response now, the numbers for the car part doesn’t make any sense
200 m huh.
I like that it’s twice as far to drive for some reason. Maybe it’s getting added to the distance you already walked?
Honestly that’s a lot more coherent than what I would expect from an LLM running on phone hardware.
I notice that the “internal thinking” of Opus 4.6 is doing more flip-flopping than earlier modelss like Sonnet 4.5, and it’s coming out with correct answers in the end more often.