Apropos of this comment from Simon Willison:

I’ve been mostly unconvinced by the ongoing discourse around LLMs hitting a plateau. The areas I’m personally most excited about are multi-modality (images, audio and video as input) and model efficiency. Both of those have had enormous leaps forward in the past year.

I don’t particularly care about “AGI”. I want models that can do useful things that I tell them to, quickly and inexpensively — and that’s exactly what I’ve been getting more of over the past twelve months.

Even if progress on these tools entirely stopped right now, the amount I could get done with just the models I’ve downloaded and stashed on a USB drive would keep me busy and productive for years.

Things which can be true about generative AI/LLMs at the same time:

• Scaling may be “done”, in the sense that the past few years’ rapid increase in capabilities by creating larger models with more data may no longer hold.

• There is still significant untapped potential for improving the “scaling” of these models in the sense of being able to get a given level of performance with significantly less “compute” — running models that are comparable in performance to last year’s frontier models on a beefy laptop is now possible.

• Even if the models do not improve via scaling in the first sense at all, the implications of what already exist will take years to work out in detail, and there is — for all the many problems with generative AI, nonetheless some real potential there.

• Whether that potential is worth the costs is a separate but equally important question: the current training regime is quite possibly illegal and in my view certainly unethical. That does not mean that holds for the technology.

• Many of the ways it is currently deployed are also unethical.