because there seems to be a diminishing return in training/inference power to usefulness

Be careful not to be caught up in the application of Goodhart’s Law going on in the field right now.

There’s plenty of things GPT-4 trounces everything else on, they just tend to be things outside the now standardized body of tests, which suggests the tests have become the target and are no longer effective measurements.

This is perhaps most apparent in things like Orca, where we directly use the tests as the target, have GPT-4 generate synthetic data that improves Llama performance on the target, and then see large gains in smaller models on the tests.

But those new models don’t necessarily have the same capabilities on more abstract capabilities, such as the recent approach of using analogy to solve problems.

We are arguably becoming too myopic in how we are measuring the success of new models.