We're very clearly seeing exponential progress - even above trend, on METR, whose slope keeps getting revised to a higher and higher estimate each time. Explain your perspective on the objective evidence against exponential progress?
Writing the code itself was never the main bottleneck. Designing the bigger solution, figuring out tradeoffs, taking to affected teams, etc. takes as much time as it used to. But still, there's definitely a significant improvement in code production part in many areas.
Because that requires adoption. Devs on hackernews are already the most up to date folks in the industry and even here adoption of LLMs is incredibly slow. And a lot of the adoption that does happen is still with older tech like ChatGPT or Cursor.
I think this is an open question still and very interesting. Ilya discussed this on the Dwarkesh podcast. But the capabilities of LLMs is clearly exponential and perhaps super exponential. We went from something that could string together incoherent text in 2022 to general models helping people like Terrance Tao and Scott Aaronson write new research papers. LLMs also beat IMO and the ICPC. We have entered the John Henry era for intellectual tasks...
Very spurious claims, given that there was no effort made to check whether the IMO or ICPC problems were in the training set or not, or to quantify how far problems in the training set were from the contest problems. IMO problems are supposed to be unique, but since it's not at the frontier of math research, there is no guarantee that the same problem, or something very similar, was not solved in some obscure manual.
Sir, we're in a modern economy, we don't ever ever look at productivity graphs (this is not to disparage LLMs, just a comment on productivity in general)
LLMs from late 2024 were nearly worthless as coding agents, so given they have quadrupled in capability since then (exponential growth, btw), it's not surprising to see a modestly positive impact on SWE work.
Also, I'm noticing you're not explaining yourself :)
I think this is happening by raising the floor for job roles which are largely boilerplate work. If you are on the more skilled side or work in more original/ niche areas, AI doesn't really help too much. I've only been able to use AI effectively for scaling refactors, not really much in feature development. It often just slows me down when I try to use it. I don't see this changing any time soon.
Hey, I'm not the OG commentator, why do I have to explain myself! :)
When Fernando Alonso (best rookie btw) goes from 0-60 in 2.4 seconds in his Aston Martin, is it reasonable to assume he will near the speed of light in 20 seconds?
> Hey, I'm not the OG commentator, why do I have to explain myself! :)
The issue is that you're not acknowledging or replying to people's explanations for _why_ they see this as exponential growth. It's almost as if you skimmed through the meat of the comment and then just re-phrased your original idea.
> When Fernando Alonso (best rookie btw) goes from 0-60 in 2.4 seconds in his Aston Martin, is it reasonable to assume he will near the speed of light in 20 seconds?
This comparison doesn't make sense because we know the limits of cars but we don't yet know the limits of LLMs. It's an open question. Whether or not an F1 engine can make it the speed of light in 20 seconds is not an open question.
It's not in me to somehow disprove claims of exponential growth when there isn't even evidence provided of it.
My point with the F1 comparison is to say that a short period of rapid improvement doesn't imply exponential growth and it's about as weird to expect that as it is for an f1 car to reach the speed of light. It's possible you know, the regulations are changing for next season - if Leclerc sets a new lap record in Australia by .1 ms we can just assume exponential improvements and surely Ferrari will be lapping the rest of the field by the summer right?
How long before introduction of computers lead to increases in average productivity? How long for the internet? Business is just slow to figure out how to use anything for its benefit, but it eventually gets there.
The best example is that even ATM machines didn't reduce bank teller jobs.
Why? Because even the bank teller is doing more than taking and depositing money.
IMO there is an ontological bias that pervades our modern society that confuses the map for the territory and has a highly distorted view of human existence through the lens of engineering.
We don't see anything in this time series, because this time series itself is meaningless nonsense that reflects exactly this special kind of ontological stupidity:
As if the sum of human interaction in an economy is some kind of machine that we just need to engineer better parts for and then sum the outputs.
Any non-careerist, thinking person that studies economics would conclude we don't and will probably not have the tools to properly study this subject in our lifetimes. The high dimensional interaction of biology, entropy and time. We have nothing. The career economist is essentially forced to sing for their supper in a type of time series theater.
Then there is the method acting of pretending to be surprised when some meaningless reductionist aspect of human interaction isn't reflected in the fake time series.
This guy is one of the top names in AI. This is pure propaganda written to instill "fear of missing out" and encouraging people to buy into his platform, lest they become "obsolete."
It’s a little shocking to me that this sentiment hasn’t floated higher in the discussion. Regardless of how he feels, this is the way he wants you to feel.
Big picture it’s about emotional intelligence and if you are losing your shit you’re going to flail around. I think you should pick up some near-frontier tools and use them to improve your usual process, always keeping your feet on the ground. “Vibe coding” was always about getting you and keeping you over your head. Resist it!
given that the 3 hares seem to currently lack a signification, I'd be up for squatting? Or would Paul prefer 3 fennecs? Should anyone wish to oppose us, as Bigwig said: "silflay hraka, u embleer rah"
a slightly more pragmatic story for shunya as better mousetrap: just as we now routinely have our calculations done for us in binary, but record results in decimal (in PDF invoices, say), ancient romans (among other cultures) would have someone do their calculations on a counting https://en.wikipedia.org/wiki/Counting_board board, but recorded (only the non-zero) results in roman numerals.
(these days we can spot the algebraists via a sibboleth: they start their papers and books with section/chapter 0)
> « Les hommes sont comme les chiffres : ils n'acquièrent de valeur que par leur position. » —NB
To be fair I did have just a touch of thought disorder which led me to write "vive" instead of "vibe" and I did correct it when it was pointed out without explaining it which made that comment seem even weirder than it originally was.
I actually read their comment as "vibe vibe live" which combined with the unknown terms in the next line (a reference to Dune combined with something else, I guess?) made GGP's question fit quite well.
on the other hand, it does currently feel like when angular and react were starting to come out, and there was a billion different javascript libraries to learn with a new one coming out every couple weeks, and you arent quite sure what you should spend your time on and how much, vs now where you just learn react, and maybe extend to next.js
LLM forward development has a lot of things going on, and it really isn't clear yet what is going be the common standard in a few years time in terms of dev ux, async tools, ci/cd tools, in production and offline workflows, etc.
its an easy time to hop down a wrong path picking subpar tools or not experimenting further, but if you just wait, the people who try the right tools are going to be way ahead on making products for their customers.
Uncharitable take.
His last public stance on this a few months ago when he released nanochat was that he didn’t use coding LLM for it, even though he tried, because they were not good enough and he was just losing time, so coded everything manually. Andrej is already set for life, and has moved into education where most of what he does is released for free.
No it’s absolutely not. But I thought it’d be fun to offer Adams’ brilliant hyperbole for an affectionate ribbing of Karpathy. Both of them are great communicators of ideas.
If I were Anthropic I would have some kind of TOS restriction saying that you can't use their trademark to represent what you use their API to enable. It's just inappropriate. Even if you are a full anti-AI activist, it seems clear that the blame for specific things 'Claude' does in response to a deliberate prompt should fall on the person(s) operating it, and as such they shouldn't be allowed to make it appear that this is what Anthropic designed Claude to do.
I think it's good, they're raising the size (and price) of flash a bit and trying to position Flash as an actually useful coding / reasoning model. There's always lite for people who want dirt cheap prices and don't care about quality at all.
reply