Labor market impacts of AI: A new measure and early evidence
anthropic.com267 points by jjwiseman 17 hours ago
267 points by jjwiseman 17 hours ago
I was at a big tech for last 10 years, quit my job last month - I feel 50x more productive outside than inside.
Here is my take on AI's impact on productivity:
First let's review what are LLMs objectively good at: 1. Writing boiler plate code 2. Translating between two different coding languages (migration) 3. Learning new things: Summarizing knowledge, explaining concepts 4. Documentation, menial tasks
At a big tech product company #1 #2 #3 are not as frequent as one would think - most of the time is spent in meetings and meetings about meetings. Things move slowly - it's designed to be like that. Majority devs are working on integrating systems - whatever their manager sold to their manager and so on. The only time AI really helped me at my job was when I did a one-week hackathon. Outside of that, integrations of AI felt like more work rather than less - without much productivity boost.
Outside, it has proven to be a real productivity boost for me. It checks all the four boxes. Plus, I don't have to worry about legal, integrations, production bugs (eventually those will come).
So, depends who you are asking -- it is a huge game changer (or not).
having been an early employee and founder of a few startups and then working at a few larger companies, most people who only ever worked at FAANG have no idea how much more productive tiny teams with ownership are.
Been a startup founder - work at Meta currently.
AI is making everyone faster that I’ve seen. I’d say 30% of the tickets I’ve seen in the last month have been solved by just clicking the delegate to AI button
How did you decide to work at Meta?
I'll be honest, just the idea of working there makes me feel like vomiting. For me, they are bizarrely evil. They're not evil like, "we're going to destroy our competition through anti competitive practices," (which they do), but "let's destroy a whole generation of minds."
And now with the glasses. I mean, jeeze. Can there be a stronger signal of not caring for others?
It's as if Meta sees people as cattle. Though I think a lot of techies see humans as cattle, truthfully.
What was your rationale?
I guess this question is out-of-the-blue, and I don't mean for you to justify your existence, but I've never understood why people choose to work for Meta.
I have a theory that when you have 2 developers working in synergy, you're at something like 1.8x what 1 person can do. As you add more people you approach 2x until some point after which you start to decrease productive. I don't think that point is far beyond 5.
I love tiny teams. I hate big corp.
Big corporations are full with people who love to entertain 20+ people in video calls. 1-2 people speak, the other nod their heads while browsing Amazon.
I wouldn’t be sad if those jobs vanished.
Well, you should be terrified of those jobs vanishing I think.
All of these people will consequently be on the job market competing for your opportunities.
Yes you may feel superior to their capabilities - and may even be justified in your opinion (I know nothing about you beyond this comment)... But it'll still significantly impact your professional future if this actually happens. It would massively impact wages at the very least
Your viewpoint is incredibly short-sighted and not actually realizing the broad effect on the industry as a whole such a change would bring.
This. Hell even a company that is 100 people or more. Ive seen companies just grind to snails pace around 80-90 people and then still scale to 400-500 and then it's impossible to really do anything meaningful. I have tried to test for this in interviews over the years but ultimately I just end up disappointed. At this point I don't even look, just work in small independently organized groups or coops.
Yes, throughput is determined by the bottleneck and above a certain organization size, the bottleneck often is coordination costs.
2. Translating between two different coding languages (migration)
I have a game written in XNA
100% of the code is there, including all the physics that I hand-wrote.
All the assets are there.
I tried to get Gemini and Claude to do it numerous times, always with utter failure of epic proportions with anything that's actually detailed. 1 - my transition from the lobby screen into gameplay? 0% replicated on all attempts 2 - the actual physics in gameplay? 0% replicated none of it works 3 - the lobby screen itself? non-functional
Okay so what did it even do? Well it put together sort of a boilerplate main menu and barebones options with weird looking text that isn't what I provided (given that I provided a font file), a lobby that I had to manually adjust numerous times before it could get into gameplay, and then nonfunctional gameplay that only handles directional movement and nothing else with sort of half-working fish traveling behavior.
I've tried this a dozen times since 2023 with AI and as late as late last year.
ALL of the source code is there every single thing that could be translated to be a functional game in another language is there. It NEVER once works or even comes remotely close.
The entire codebase is about 20,000 lines, with maybe 3,000 of it being really important stuff.
So yeah I don't really think AI is "really good" at anything complex. I haven't really been proven wrong in my 4 years of using it now.
I crave to see people saying "Here's the repo btw: ..." and others trying out porting it over, just so we see all of the ways how AI fails (and how each model does) and maybe in the middle there a few ways to improve its odds. Until it eventually gets included in training data, a bit like how LLMs are oddly good at making SVGs of pelicans on bicycles nowadays.
And then, maybe someone slightly crazy comes along and tries seeing how much they can do with regular codegen approaches, without any LLMs in the mix, but also not manual porting.
Agreed -- coding agents / LLMs are definitely imperfect, but it's always hard to contextualize "it failed at X" without knowing exactly what X was (or how the agent was instructed to perform X)
I'm sure someone who regularly programs games in the destination language I want who also has worked with XNA in the past as a game developer could port it in a week or something yeah
- Split it in different modules / tasks
- Do not say: "just convert this"
- On critical sections you do a method-per-method-translation
- Dont forget: your 20.000 lines source at a whole will make any model to be distracted on longer tasks (and sessions, for sure)
- Do dedicated projects within Claude per each sub-module
This matches my experience. Unless it's been done to death online (crud etc) it falls on its face every time.
It's okay shills and those with stock in the AI copies brainwashed themselves inside out and will spam forever on this website that if you just introduce the right test conditions agents can do anything. Never mind engineering considerations if it passes the tests it's good homie! Just spent an extra few hundred or thousand a month on it! Especially on the company I have stock in! Give me money!
In those situations you basically need to guide llm to do it properly. It rarely one shots complex problems like this, especially in non web dev, but could make it faster than doing it manually.
Oh believe me I broke it down super finely, down to single files and even single functions in some places
It still is completely and utterly hopeless
I've done this multiple times in various codebases, both medium sized personal ones (approx 50k lines for one project, and a smaller 20k line one earlier) and am currently in the process of doing a similar migration at work (~1.4 million lines, but we didn't migrate the whole thing, more like 300k of it).
I found success with it pretty easily for those smaller projects. They were gamedev projects, and the process was basically to generate a source of truth AST and diff it vs a target language AST, and then do some more verifier steps of comparing log output, screenshot output, and getting it to write integration tests. I wrote up a bit of a blog on it. I'm not sure if this will be of any use to you, maybe your case is more difficult, but anyway here you go: https://sigsegv.land/blog/migrating-typescript-to-csharp-acc...
For me it worked great, and I would (and am) using a similar method for more projects.
People who are saying they're not seeing productivity boost, can you please share where is it failing?
Because, I am terrified by the output I am getting while working on huge legacy codebases, it works. I described one of my workflow changes here: https://news.ycombinator.com/item?id=47271168 but in general compared to old way of working I am saving half of the steps consistently, whether its researching the codebase, or integrating new things, or even making fixes. I have stopped writing code, occasionally I jump into the changes proposed by LLM and make manual edits if it is feasible, otherwise I revert changes and ask it to generate again but based on my learnings from the past rejected output
I am terrified about what's coming
I find LLMs are good at essentially boilerplate code. It's clear what to do and it needs to be typed in. Or areas where I really have no idea where to start, because I'm not familiar with the codebase.
I find anything else, I spend more time coaxing them into doing 85% of what I need that I'm better off doing it myself.
So they're not useless but there's only so many times in a week that I need a function to pretty-print a table in some fashion. And the code they write on anything more complex than a snippet is usually written poorly enough that it's a write-once-never-touch-again situation. If the code needs to be solid, maintainable, testable, correct (and these are kind of minimal requirements in my book) then LLMs make little impact on my productivity.
They're still an improvement on Google and Stack exchange, but again - only gets you so far.
YMMV
> I find anything else, I spend more time coaxing them into doing 85% of what I need that I'm better off doing it myself.
You must be working in a very niche field with very niche functionality if that's the case? I work at a company just outside of FAANG and I work in compliance. Not a terribly complex domain but very complicated scale and data integrity requirements.
I haven't written a single line of code manually in 2 weeks. Opus 4.6 just... works. Even if I don't give it all the context it just seems to figure things out. Occasionally it'll make an architectural error because it doesn't quite understand how the microservices interact. But these are non-trivial errors (i.e. humans could have made them as well) and when we identify such an error, we update the team-shared CLAUDE.md to make sure future agents don't repeat the error.
I often wonder what I am missing. Recently I wanted to wrap a low level vendor API with a callback API (make a request struct and request id, submit, provide a callback fn, which gets called with request IDs and messages received from vendor) to async Python (await make_request(...)). Kinda straightforward - lots of careful code of registering and unregistering callbacks, some careful thread synchronisation (callbacks get called in another thread), thinking about sane exception handling in async code. Fiddly but not rocket science.
What I got sort of works, as in tests pass - this with Opus 4.5. It is usable, though it doesn't exist cleanly on errors despite working to death with Claude about this. On exception it exits dirtily and crashes, which is good enough for now. I had some fancy ideas about logging messages from the vendor to be able to replay them, to be able to then reproduce errors. Opus made a real hash of it, lots of "fuck it comment out the assert so the test passes". This part is unusable and worse, pollutes the working part of the project. It made a valiant effort at mocking the vendor API for testing but really badly, instead of writing 30 lines of general code, it wrote 200 lines of inconsistent special cases that don't even work altogether. Asked to fix it it just shuffles around the special cases and gets stuck.
It's written messily enough that I wouldn't touch this even to remove the dead code paths. I could block a few days for it to fix but frankly in that time I can redo it all and better. So while it works I'm not gonna touch it.
I did everything LLM proponents say. I discussed requirements. Agent had access to the API docs and vendor samples. I said think hard many times. Based on this we wrote a detailed spec, then detailed inplementation plan. I hand checked a lot of the high level definitions. And yet here I am. By the time Opus went away and started coding, we had the user facing API hammered out, key implementation details (callback -> queue -> async task in source thread routing messages etc), constraints (clean handling of exceptions, threadsafe etc). Tests it has to write. Any minor detail we didn't discuss to death was coded up like a bored junior.
And this also wasn't my first attempt, this was attempt #3. First attempt was like, here's the docs and samples, make me a Python async API. That was a disaster. Second was more like, let's discuss, make a spec, then off you go. No good. Even just taking the last attempt time, I would have spent less time doing this by hand myself from scratch.
> On exception it exits dirtily and crashes, which is good enough for now
Silent failures and unexplained crashes are high on my list of things to avoid, but many teams just take them for granted in spite of the practical impact.
I think that a lot of orgs have a culture of "ship it and move on," accompanied by expectations like: QA will catch it, high turnover/lower-skill programmers commit stuff like this all the time anyway, or production code is expected to have some rough edges. I've been on teams like that, mostly in bigger orgs with high turnover and/or low engineering standards.
> I spend more time coaxing them into doing 85% of what I need that I'm better off doing it myself
What was the last thing you built in which you felt this was the case?
The companies laying off people have no vision. My company is a successful not for profit and we are hiring like crazy. It’s not a software company, but we have always effectively unlimited work. Why would anyone downsize because work is getting done faster? Just do more work, get more done, get better than the competition, get better at delivering your vision. We put profits back in the community and actually make life better for people. What a crazy fucking concept right?
I suspect it depends partly on how locked each individual is into a particular type of work, both skill-wise and temperamentally.
To give an example from a field where LLMs started causing employment worries earlier than software development: translation. Some translators made their living doing the equivalent of routine, repetitive coding tasks: translating patents, manuals, text strings for localized software, etc. Some of that work was already threatened by pre-LLM machine translation, despite its poor quality; context-aware LLMs have pretty much taken over the rest. Translators who were specialized in that type of work and too old or inflexible to move into other areas were hurt badly.
The potential demand for translation between languages has always been immense, and until the past few years only a tiny portion of that demand was being met. Now that translation is practically free, much more of that demand is being met, though not always well. Few people using an app or browser extension to translate between languages have much sense of what makes a good translation or of how translation can go bad. Professional translators who are able to apply their higher-level knowledge and language skills to facilitate intercultural communication in various ways can still make good money. But it requires a mindset change that can be difficult.
I'm not in translation, but a number of close friends are in the industry. Two trends I've noticed in the industry, which I think we're seeing mirrored in tech:
1. No one cares about quality. Even in fields you'd expect to require the 'human touch' (e.g. novel translation), publishers are replacing translators with AI. It doesn't matter if you have higher-level knowledge or skills if the company gains more from cutting your contract than it loses in sales.
2. Translation jobs have been replaced with jobs proofreading machine translations, which pays peanuts (since AI is 'doing most of the work') but in fact takes almost as much effort as translating from scratch (since AI is often wrong in very subtle ways). The comparison to PR reviews makes itself.
It is not entirely true that no one cares about quality. I'd like to stay optimistic and believe that those who are demanding on the quality of their production will acquire sufficient market differentiation to prevail.
After all, this has been Apple strategy since the 80's, and, even though there were some up's and down's, overall it's a success.
> It is not entirely true that no one cares about quality. I'd like to stay optimistic and believe that those who are demanding on the quality of their production will acquire sufficient market differentiation to prevail.
Maybe, but it probably requires a very strong and opinionated leader to pull off. The conventional wisdom in American business leadership seems to be to pursue the lowest level of quality you can get away with, and focus on cutting costs. And you'll have to fight that every second.
I don't think that's true at the individual-contributor level (pursing quality is very motivating), but they people who move up are the ones who sound "smart" by aping conventional wisdom.
> After all, this has been Apple strategy since the 80's, and, even though there were some up's and down's, overall it's a success.
I might give you that "since the late 90s," but there have been significant periods where that wasn't true (e.g. the early mid-90s Mac OS was buggy and had poor foundations).
This is exactly right IMO. I have never worked for a company where the bottleneck was "we've run out of things to do". That said, plenty of companies run out of actual software engineering work when their product isn't competitive. But it usually isn't competitive because they haven't been able to move fast enough
I think it depends on:
A) how old the product is: Twitter during its first 5 years probaby had more work to do compared to Twitter after 15 years. I suspect that is why they were able to get rid of so many developers.
B) The industry: many b2c / ecommerce businesses are straightforward and don't have an endless need for new features. This is different than more deep tech companies
That was my insight also. As a manager, you already have the headcount approved, and your people just allegedly got some significant percentage more productive. The first thought shouldn't be, great let's cut costs, it should be great now we finally have the bandwidth to deliver faster.
On a macro level, if you were in a rising economic tide, you would still be hiring, and turning those productivity gains into more business.
I wonder what the parallels are to past automations. When part producing companies moved from manual mills to CNC mills, did they fire a bunch of people or did they make more parts?
CNC machines drove down operator wages. Its similar to the translator example where the machine code is written by someone else, but the person running the machine still needs to understand. Simple pushing the go button is dangerous, being able to adapt is critical.
Jobs where a machinist is in charge of large chunks of the process are rarer. Large shop will have one person setting up many machines to maximize throughput.
I'm an EM as well and I've been telling my teams for a while now that I think they really only need to start worrying once our backlog starts going down instead of up. Generally, I still agree with that (and your) sentiment when you look at the long term, but in the short term, I think all of the following arguments can be made in favor of layoffs:
- AI tools are expensive so until the increased productivity translates to increased revenue we need to make room in the budget
- We expect the bottlenecks in our org to move from writing code to something else (PM or design or something) so we're cutting SWEs in anticipation of needing to move that budget elsewhere.
- We anticipate the skillsets needed by developers in the AI world to be fundamentally different from what they are now that it's cheaper to just lay people off, run as lean as possible, and rehire people with the skills we want in a year or two than it is to try and retrain.
I don't necessarily agree with those arguments (especially the last one), but I think they're somewhat valid arguments
I see similar arguments and I don't agree as well, here is why:
> rehire people with the skills we want in a year or two than it is to try and retrain.
before that future comes your company might become obsolete already, because you have lost your market share to new entrants
> We expect the bottlenecks in our org to move from writing code to something else
I would love to tell them, hey lets leverage current momentum and build, when those times come, we offer existing people with accumulated knowledge to retrain to a new type of work, if they think they're not good fit, they can leave, if they're willing, give them a chance, invest in people, make them feel safe and earn trust and loyalty from them
> AI tools are expensive so until the increased productivity translates to increased revenue we need to make room in the budget
1. Its not that expensive: 150$/seat/month -> 5 lunches? or maybe squeeze it from Sales personnel traveling with Business class?
2. By the time increased productivity is realized by others, company who resisted could be so far behind, that they won't be able to afford hiring engineers with those skillsets, if they think 150$ is expensive now, I am sure they will say "What??? 350k$ for this engineer?, no way, I will instead hire contractors"
You need certain company culture, to be able to scale up, and to capture this value. Most companies can not just add new developers.
AI needs documentation, automation, integration tests... It works very well for remote first company, but not for in-face informal grinding approach.
Just year ago, client told me to delete integration tests, because "they ran too long"!
>Just year ago, client told me to delete integration tests, because "they ran too long"!
Why are you surprised customers don't like spending money on the items that don't add business value. Add to that QA, documentation, security audits, etc.
They want to ship stuff that brings in customers and revenue day one, everything else is a cost.
> integration tests, QA etc ... the items that don't add business value
They absolutely do add value / prevent loss, but you need some understanding in order to see that. Not seeing it is a marker of not understanding.
>They absolutely do add value
Not to the non-technical bean counters. When they allocate money they want to see you prove how that extra money translates to an immediate ROI, and it's difficult to prove that in an Excel sheet exactly what the ROI will be without making stuff up on vibes and feels.
Like at one German company i was at ~15 years ago, all the devs wanted a second 19" monitor on our workstations for increased productivity, and the bean counters wouldn't approve that because they wanted proof of how that expense across hundreds of people will increase our productivity and by how much %, to see if that would offset the cost.
This is how these people think. If you don't bring hard numbers on how much their "line will go up", they won't give you money.
I know this is difficult to understand from the PoV of SV Americans where gazillions of dollars just fall from the sky at their tech companies.
Does that extra work bring in more revenue? I think that’s the key question.
Companies that do not reduce their workforce might outcompete you.
It might not be about bringing more revenues but retaining market share.
If your barrier to being competitive is a slow, bureaucratic org, restructuring and laying off might actually help long term.
I’ve been screaming this too https://news.ycombinator.com/item?id=47212237
It’s refreshing to see the same sentiment from so many other people independently here.
Because hiring less while getting more done increases margins. Your company is not for profit so doesnt care about margins. Others do.
These are words without weights. At some point the put money into software option will max out. Perhaps what we should all be doing is hiring more lawyers, there's always more legal work to be done. When you don't have weights then you can reason like this.
I don’t know what kind of software your used to but software is pretty much universally dog shit these days. I could probably count on one hand the number of programs that I actually like using. There is an astronomical room for improvement. I don’t think we are hitting diminishing returns any time soon.
I talk about this at length in one of my previous posts here: https://news.ycombinator.com/item?id=39963058. I definitely share your opinion and I think this will be exacerbated by vibe coding and having LoC as the main KPI for engineering teams.
The problem becomes if you are a service like Youtube, where you already have capture almost the entire customer base.
Yes, it's the lump of labor fallacy.
Doesn't exclude the possibility of short term distribution, though.
> Just do more work, get more done
That's one of the reasons why I am terrified, because it can lead to burn out, and I personally don't like to babysit bunch of agents, because the output doesn't feel "mine", when its not "mine" I don't feel ownership.
And I am deliberately hitting the brake from time to time not to increase expectations, because I feel like driving someone else's car while not understanding fully how they tuned their car (even though I did those tunings by prompting)
I'm currently a product manager (was a software engineer and technical architect before), so i already lost the feeling of ownership of code. But just like when you're doing product management with a team of software engineers, testers, and UXers, with AI you can still feel ownership of the feature or capability you're shipping. So from my perspective, nothing changes regarding ownership.
> So from my perspective, nothing changes regarding ownership.
The engineer who worked with you took ownership of the code! Have you forgotten this?
No, that’s why I wrote “from my perspective”. I started long ago writing 6502 and 68000 assembly, later c and even later Java. Every step you lose ownership of the underlying layer. This is just another step. “But it’s non deterministic!”, yes so are developers. We need QA regardless who or what write the lines of code.
It feels very much like leading a team of junior engineers or even interns who are very fast but have no idea about why we're doing anything. You have to understand the problems you're trying to solve and describe the solutions in a way they can be implemented.
It's not going to be written exactly like you would do it, but that's ok - because you care about the results of the solution and not its precise implementation. At some point you have to make an engineering decision whether to write it yourself for critical bits or allow the agent/junior to get a good enough result.
You're reviewing the code and hand editing anyway, right? You understand the specs even if your agent/junior doesn't, so you can take credit even if you didn't physically write the code. It's the same thing.
> It feels very much like leading a team of junior engineers or even interns who are very fast but have no idea about why we're doing anything
Yes, yes!
And this is problem for me, because of the pace, my brain muscles are not developing enough compared to when I was doing those things myself.
before, I was changing my mind while implementing the code, because I see more things while typing, and digging deeper, but now, because juniors are doing things they don't offer me a refactoring or improvements while typing the code quickly, because they obey my command instead of having "aha" moment to suggest better ways
There’s some hope that the industry will realize that managing clueless LLMs at high pace isn’t sustainable and leads to worse results, and some middle ground has to be found. Or we will reach AGI, so AI won’t be clueless anymore and really take your engineering job.
I think a lot of companies have ineffective ways to measure productivity, poor management (e.g., people who were IC's then promoted to management but have no management training or experience), incentives aren't necessarily aligned between orgs and staff, so people end up with a perverse "more headcount" means I'm better than Sandy over there. Leadership and vision have been rare in my professional life (though the corporate-owned media celebrates mediocrity in leadership all the time with puff pieces).
Once you get to a certain size company, this means a lot of bloat. Heck, I've seen small(ish) companies that had as many managers and administrators as ICs.
But You're not wrong, I'm just pointing out how an org that has 4k people can lay off a few hundred with modest impact of the financials (though extensive impact on morale).
You would need to expand your capacity to find and define the work. I imagine that would be a major challenge.
I have an app which is fairly popular. This release cycle I used Claude Code and codex to implement all the changes / features. It definitely let me move much quicker than before.
However now that it's in the beta stage the amount of issues and bugs is insane. I reviewed a lot of the code that went in as well. I suspect the bug fixing stage is going to take longer than the initial implementation. There are so many issues and my mental model of the codebase has severely degraded.
It was an interesting experiment but I don't think I would do it again this way.
The last 10% takes up 90% of the time. Usually, the time is spent finding issues you didn't even know about. This was true before LLMs.
Rather than trying to fix the bugs yourself, have you tried asking Claude to fix them for you?
I have already been doing this. I could keep doing it but I'm not going to. I want to be able to understand my own code because that is what let's me make sound higher level decisions.
It's failing when there is no data in the training set, and there are no patterns to replicate in the existing code base.
I can give you many, many examples of where it failed for me:
1. Efficient implementation of Union-Find: complete garbage result 2. Spark pipelines: mostly garbage 3. Fuzzer for testing something: half success, non-replicateable ("creative") part was garbage. 4. Confidential Computing (niche): complete garbage if starting from scratch, good at extracting existing abstractions and replicating existing code.
Where it succeeds: 1. SQL queries 2. Following more precise descriptions of what to do 3. Replicating existing code patterns
The pattern is very clear. Novel things, things that require deeper domain knowledge, coming up with the to-be-replicated patterns themselves, problems with little data don't work. Everything else works.
I believe the reason why there is a big split in the reception is because senior engineers work on problems that don't have existing solutions - LLMs are terrible at those. What they are missing is that the software and the methodology must be modified in order to make the LLM work. There are methodical ways to do this, but this shift in the industry is still in baby shoes, and we don't yet have a shared understanding of what this methodology is.
Personally I have very strong opinions on how this should be done. But I'm urging everyone to start thinking about it, perhaps even going as far as quitting if this isn't something people can pursue at their current job. The carnage is coming:/
The last time I tried AI, I tested it with a stopwatch.
The group used feature flags...
if (a) {
// new code
} else {
// old code
}
void testOff() {
disableFlag(a);
// test it still works
}
void testOn() {
enableFlag(a);
// test it still works
}
I can do one flag in ~3minutes. Code edit, pr prepped and sent.
The AI can do one in 10mins, but I couldn't look away. It kept trying to use find/grep to search through a huge repo to find symbols (instead of the MCP service).
Then it ignored instructions and didn't clean up one or the other test, left unused fields or parameters and generally made a mess.
Finally, I needed to review and fix the results, taking another 3-5 minutes, with no guarantee that it compiled.
At that point, a task that takes me 3 minutes has taken me 15.
Sure, it made code changes, and felt "cool", but it cost the company 5x the cost of not using the AI (before considering the token cost).
Even worse, the CI/CD system couldn't keep up the my individual velocity of cleaning these up, using an automated tool? Yeah, not going to be pleasant.
However, I need to try again, everyone's saying there was a step change in December.
I did my own experiment with Claude Code vs Cursor tab completion. The task was to convert an Excel file to a structured format. Nothing fancy at all.
Claude Code took 4 hours, with multiple prompts. At the end, it started to break the previous fixes in favor of new features. The code was spaghetti. There was no way I could fix it myself or steer Claude Code into fixing it the right way. Either it was a dead-end or a dice roll with every prompt.
Then I implemented my own version with Cursor tab completion. It took the same amount of time, 4 hours. The code had a clear object-oriented architecture, with a structure for evolution. Adding a new feature didn't require any prompts at all.
As a result, Claude Code was worse in terms of productivity: the same amount of time, worse quality output, no possibility of (or at best very high cost of) code evolution.
Are you able to share your prompts to Claude Code? I assume not, they are probably not saved - but this genuinely surprised me, it seems like exactly the type of task an LLM would excel at (no pun intended!). What model were you using OOI?
> this genuinely surprised me
Me too. After listening to all the claims about Claude Code's productivity benefits, I was surprised to get the result I got.
I'm not able to share details of my work. I was using Claude Opus 4.5, if I recall correctly.
The exact same prompt ? Everything depends on the prompt and it’s different tools. These days the quality and what’s build around the prompt matters as much as the code. We can’t feed generic query.
Similar happened to me just now. Claude whatever-is-the-latest-and-greatest, in Claude Code. I also tried out Windsurf's Arena Mode, with the same failure. To intercept the inevitable "holding it wrong" comments, we have all the AGENTS.md and RULES.md files and all the other snake oil you're told to include in the project. It has full context of the code, and even the ticket. It has very clear instructions on what to do (the kind of instructions I would trust an unpaid intern with, yet alone a tool marketed as the next coming of Cyber Jesus that we're paying for), in a chat with minimal context used up already. I manually review every command it runs, because I don't trust it running shell scripts unsupervised.
I wanted it to finish up some tests that I had already prefilled, basically all the AI had to do was convert my comments into the final assertions. A few minutes later of looping, I see it finishes and all tests are green.
A third of the tests were still unfilled, I guess left as an exercise for the reader. Another third was modified beyond what I told it to do, including hardcoding some things which made the test quite literally useless and the last third was fine, but because of all the miscellaneous changes it made I had to double check those anyways. This is about the bare minimum where I would expect these things to do good work, a simple take comment -> spit out the `assert()` block.
I ended up wasting more time arguing with it than if I had just done the menial task of filling out the tests myself. It sure did generate a shit ton of code though, and ran in an impressive looking loop for 5-10 minutes! And sure, the majority of the test cases were either not implemented or hardcoded so that they wouldn't actually catch a breakage, but it was all green!!
That's ultimately where this hype is leading us. It's a genuinely useful tool in some circumstances, but we've collectively lost the plot because untold billions have poured into these systems and we now have clueless managers and executives seeing "tests green -> code good" and making decisions based on that.
What model, what harness and about how long was your prompt to fire off this piece of work? All three matters a lot, but importantly missing from your experience.
Because its failure rate is too high. Beyond boilerplate code and CRUD apps, if I let AI run freely on the projects I maintain, I spend more time fixing its changes than if I just did it myself. It hallucinates functionally, it designs itself into corners, it does not follow my instructions, it writes too much code for simple features.
It’s fine at replacing what stack overflow did nearly a decade ago, but that isn’t really an improvement from my baseline.
It’s not that it just makes mistakes but it also implements things in ways I don’t like or are not relevant to the business requirements or scope of the feature / project.
I end up replacing any saved time with QA and code review and I really don’t see how that’s going to change.
In my mind I see Claude as a better search engine that understands code well enough to find answers and gain understanding faster. That’s about it.
That's my experience too. It's okay at a few things that save me some typing, but it isn't really going to do the hard work for me. I also still need to spend significant amounts of time figuring out what it did wrong and correcting it. And that's frustrating. I don't make those mistakes, and I really dislike being led down bad paths. If "code smells" are bad, then "AI" is a rotting corpse.
Something I've been thinking about lately is if there is value in understanding the systems we produce and if we expected to?
If I can just vibe and shrug when someone asks why production is down globally then I'm sure the amount of features I can push out increases, but if I am still expected to understand and fix the systems I generate, I'm not convinced it's actually faster to vibe and then try to understand what's going on rather than thinking and writing.
In my experience the more I delegate to AI, the less I understand the results. The "slowness and thinking" might just be a feature not a bug, at times I feel that AI was simply the final straw that finally gave the nudge to lower standards.
>if I can just vibe and shrug when someone asks why production is down globally
You're pretty high up in the development, decision and value-addition chain, if YOU are the responsible go-to person for these questions. AI has no impact on your position.
Naa, I'm just a programmer. Experience may vary depending on company and country, for me this has been true from tiny startups to global corporations.
Tangential, I don't even know what "responsible" in the corporate world means anymore, it seems to me no one is really responsible for anything. But the one thing that's almost certain is that I will fix the damn thing if I made it go boom.
AI dramatically increases velocity. But is velocity productivity? Productivity relative to which scope: you, the team, the department, the company?
The question is really, velocity _of what_?
I got this from a HN comment. It really hit for me because the default mentality for engineers is to build. The more you build the better. That's not "wrong" but in a business setting it is very much necessary but not sufficient. And so whenever we think about productivity, impact, velocity, whatever measure of output, the real question is _of what_? More code? More product surface area? That was never really the problem. In fact it makes life worse majority of the time.
The real question is, is it increasing their velocity?
They've already admitted they just 'throw the code away and start again'.
I think we've got another victim of perceived productivity gains vs actual productivity drop.
People sitting around watching Claude churn out poor code at a slower rate than if they just wrote it themselves.
Don't get me wrong, great for getting you started or writing a little prototype.
But the code is bad, riddled with subtle bugs and if you're not rewriting it and shoving large amounts of AI code into your codebase, good luck in 6-12 months time.
I work as an ML engineer/researcher. When I implement a change in an experiment it usually takes at least an hour to get the results. I can use this time to implement a different experiment. Doesn't matter if I do it by hand or if I let an agent do it for me, I have enough time. Code isn't the bottleneck.
I also heard an opinion that since writing code is cheap, people implement things that have no economic value without really thinking it through.
+1 on the economic value line. Not everything needs to be about money but if you get paid to ship code it's about money. And now we have coworkers shipping insane amounts of "features" because it's all free to ship and being an engineer, it ends there.
Only it doesn't, there's product positioning, UX, information architecture, onboarding and training, support, QA, change management, analytics, reporting… sigh
> but if you get paid to ship code it's about money.
Tip to budding software engineers: try to not work in these sort of places, as they're about "looking busy" rather than engineering software, where the latter is where real long-lasting things are built, and the former is where startup founders spend most their money.
The last paragraph is where the tricky and valuable parts are, and also where AI isn't super helpful today, and where you as a human can actually help out a lot if you're just 10% better than the rest of the "engineers" who only want to ship as fast as possible.
> People who are saying they're not seeing productivity boost, can you please share where is it failing?
At review time.
There are simply too many software industries that can't delegate both authorship _and_ review to non-humans because the maintenance/use of such software, especially in libraries and backwards-compat-concerning environments, cannot justify an "ends justifies the means" approach (yet).
I don’t think the objections are not necessarily in terms of lack of productivity although my personal experience is not that of massive productivity increases. The fact that you are producing code much faster is likely just to push the bottleneck somewhere else. Software value cycles are long and complicated. What if you run into an issue in 5 years the LLM fails to diagnose or fix due to complex system interactions? How often would that happen? Would it be feasible to just generate the whole thing anew matching functionality precisely? Are you making the right architecture choices from the perspective of what the preferred modus operandi of an llm is in 5 years? We don’t know. The more experienced folks tend to be conservative as they have experienced how badly things can age. Maybe this time it’ll be different?
"it works" is a very low standard when it comes to software engineering. Why are we not holding AI generated code to the same standards as we hold our peers during code reviews?
I have never heard anyone say "it works" as a positive thing when reviewing code..
Yes, there is a productivity boost but you can't tell me there is no decrease in quality
Sometimes I’m scared.
Sometimes I realise that this particular task has been slower than if I’d done it myself when I take in to account full wall clock time.
I can’t tell what type of task is going to work ahead of time yet.
> I have stopped writing code, occasionally I jump into the changes proposed by LLM and make manual edits if it is feasible, otherwise I revert changes and ask it to generate again but based on my learnings from the past rejected output
Isn't it a very inefficient way to learn things? Like, normally, you would learn how things work and then write the code, refining your knowledge while you are writing. Now you don't learn anything in advance, and only do so reluctantly when things break? In the end there is a codebase that no one knows how it works.
> Isn't it a very inefficient way to learn things?
It is. But there are 2 things:
1. Do I want to learn that? (if I am coming back to this topic again in 5 months, knowledge accumulates, but there is a temptation to finish the thing quickly, because it is so boring to swim in huge legacy codebase)
2. How long it takes to grasp it and implement the solution? If I can complete it with AI in 2 days vs on my own in 2 weeks, I probably do not want to spend too much time on this thing
as I mentioned in other comments, this is exactly makes me worried about future of the work I will be doing, because there is no attachment to the product in my brain, no mental models being built, no muscles trained, it feels someone else's "work", because it explores the code, it writes the code. I just judge it when I get a task
I don't know where it goes, but it sounds pretty dumb for the companies involved too. Tech companies are in the business of nurturing teams knowledgeable in things so they can build something that gives them an advantage over competition. If there is no knowledge being built, there is no advantage and no tech business.
> Tech companies are in the business of nurturing teams knowledgeable in things
It pains the anti-capitalist fibers in my body to say this, but no they are not. At the maximum the value is in organizational knowledge and existing assets (= source code, documentation), so that people with the least knowledge possible can make changes. In software companies in general, technical excellence and knowledge is not strongly correlated with economic success as long as you clear a certain bar (that's not that high). In comparison, in hardware/engineering companies, that's a lot more correlated.
In the concrete example of a legacy codebase we have here, there is even less value in trying to build up knowledge in the company, as it has already been decided that the system is to be discarded anyways.
> you would learn how things work and then write the code
In a legacy codebase this may require learning a lot of things about how things work just to make small changes, which may be much less efficient.
I might still be naive about the industry, but if you don't know how the legacy codebase works, you might either delegate the change to someone else in the company who does, or, if there is no one left, use this opportunity to become the person who knows at least something about it.
I don't want to generalize from my specific situation too much, but I want to offer an anecdote from my neck of the woods. On my personal sub, I agree it is kinda crazy the kind of projects I can get into now with little to no prior knowledge.
On the other hand, our corporate AI is.. not great atm. It was briefly kinda decent and then suddenly it kinda degraded. Worst case is, no one is communicating with us so we don't know what was changed. It is possible companies are already trying to 'optimize'.
I know it is not exactly what you are asking. You are saying capability is there, but I am personally starting to see a crack in corporate willingness to spend.
I work in commercials.
We can now make 1$ million dollar commercials with 100,000$ or less. So a 90% reduction in costs - if we use AI.
The issue is they don’t look great. AI isn’t that great at some key details.
But the agencies are really trying to push for it.
They think this is the way back to the big flashy commercials of old. Budgets are lower than ever, and shrinking.
Big issue here is really the misunderstanding of cause - budgets are lower, because advertising has changed in general (TV is less and less important ) and a lot of studies showed that advertising is actually not all that effective.
So they are grabbing onto a lifeboat. But I’m worried there’s no land.
I’ve planned my exit.
Advertisement is not that effective in general or just for certain channels, i.e. TV?
Also what are you existing to?
So my understanding - from a friend at WPP who told me the same and from a freakonomics episode - is that advertising was wildly oversold before digital.
When the metrics arrived with digital, they saw that advertising, in some ways, was just not as effective as they’d hoped. In some ways the ROI wasn’t there. Seth Godin agrees. He says that advertising in the digital era could be as simple as just having a good product. I think this is Tesla’s position on it - make the best product and the internet takes care of it.
Legacy companies have kept large ad budgets but those are diminishing. From what I spoke with my friend at WPP, he said their data science team showed that outside of a new product or a product that is not recognised by consumers, the actual outcomes from ads are marginal or incremental. Thats what he told me. If your product is already known to consumers, the ROI is questionable.
Advertising's foremost job is to sell the premise of advertising to business management. Selling the business's product is always secondary to that.
Always felt suspicious to me that so much of company dynamics are basically about selling yourself to management...and there's one team in the company who's full-time job is selling? Wonder how that will turn out.
None of my coworkers could figure out why I was laid off, and were shocked because I was important to getting the work done, but management made it clear I hadn't been selling myself to management.
My exit is storytelling. I think that’s the only thing that will remain. I suspect humans will still want to hear stories about and from other humans.
There’s something about AIs that feels wrong for storytelling. I just don’t think people will want AIs to tell them stories. And if they do… Well, I believe in human storytelling.
Basically it tricks you into making the code less maintainable, so while it seems to boost productivity, it's just delaying huge failures.
Same. Whenever an article like this one pops up the comments seem to be filled with confirmation bias. People who don't see a productivity boost agree with the article.
I work at tech company just outside of big tech and I feel fairly confident that we won't have a need for the amount of developers we currently have within 3-4 years.
The bottleneck right now is reviewing and I think it's just a matter of time before our leadership removes the requirement for human code reviews (I am already seeing signs of this ("Maybe for code behind feature flags we don't need code reviews?").
Whenever there's an incident, there is a pagerduty trigger to an agent looking at the metrics, logs, software component graphs, and gives you an hypothesis on what the incident is due to. When I push a branch with test failures, I get one-click buttons in my PR to append commits fixing those tests failures (i.e. an agent analyses the code, the jira ticket, the tests, etc. and suggests a fix for the tests failing). We have a Slack agent we can ping in trivial feature requests (or bugs) in our support channels.
The agents are being integrated at every step. And it's not like the agents will stop improving. The difference between GPT3.5 and Opus 4.6 is so massive. So what will the models look like in 5 years from now?
We're cooked and the easiest way to tell someone works at a company who hasn't come very far in their AI journey is that they're not worried.
I asked opus 4.6 how to administer an A/B test when data is sparse. My options are to look at conversion rate, look at revenue per customer, or something else. I will get about 10-20k samples, less than that will add to cart, less than that will begin checkout, and even less than that will convert. Opus says I should look at revenue per customers. I don't know the right answer, but I know it is not to look at revenue per customers -- that will have high variance due to outlier customers who put in a large order. To be fair, I do use opus frequently, and it often gives good enough answers. But you do have to be suspicious of its responses for important decisions.
Edit: Ha, and the report claims it's relatively good at business and finance...
Edit 2: After discussion in this thread, I went back to opus and asked it to link to articles about how to handle non-normally distributed data, and it actually did link to some useful articles, and an online calculator that I believe works for my data. So I'll eat some humble pie and say my initial take was at least partially wrong here. At the same time, it was important to know the correct question to ask, and honestly if it wasn't for this thread I'm not sure I would have gotten there.
A/B tests are a statistical tool, and outliers will mess with any statistical measure. If your data is especially prone to that you should be using something that accounts for them, and your prompt to Opus should tell it to account for that.
A good way to use AI is to treat it like a brilliant junior. It knows a lot about how things work in general but very little about your specific domain. If your data has a particular shape (e.g lots of orders with a few large orders as outliers) you have to tell it that to improve the results you get back.
I did tell it that I expect to see something like a power-law distribution in order value, so I think I pretty much followed your instructions here. Btw, if you do know the right thing to do in my scenario, I'd love to figure it out. This is not my area of expertise, and just figuring it out through articles so far.
I recommend reading Wikipedia and talking to LLMs to get this one. Order values do follow power-law distributions (you're probably looking for an exponential or a Zipf distribution.) You want to ask how to perform a statistical test using these distributions. I'm a fan of Bayesian techniques here, but it's up to you if you want to use a frequentist approach. If you can follow some basic calculus you can follow the math for constructing these statistical tests, if not some searching will help you find the formulas you need.
Thanks for the suggestions! I didn't want to do the math myself, but I did take your suggestion and found some articles discussing ways to make it work even with a non-normal distribution:
- https://cxl.com/blog/outliers/
- https://www.blastx.com/insights/the-best-revenue-significanc...
- (online tool to calculate significance) https://www.blastx.com/rpv-calculator
I'm not checking their math, but the articles make sense to me, and I trust they did implement it correctly. In the end the LLM did get me to the correct answer by suggesting the articles, so I guess I should eat some humble pie and say it _did_ help me. At the same time, if I didn't have the intuition that using rpv as-is in a t-test would be noisy, and the suggestions from this comment thread, I think I could have gone down the wrong path. So I'm not sure what my conclusion is -- maybe something like LLMs are helpful once you ask the right question.
One heuristic I like to use when thinking about this question (and I honestly wish the answer space here were less emotionally charged, so we could all learn from each other) is that: LLMs need a human to understand the shape of the solution to check the LLM's work. In fields that I have confirmed expertise in, I can easily nudge and steer the LLM and only skim its output quickly to know if it's right or wrong. In fields I don't, I first ask the LLM for resources (papers, textbooks, articles, etc) and familiarize myself with some initial literature first. I then work with the LLMs slowly to make a solution. I've found that to work well so far.
(I also just love statistics and think it's some of the most applicable math to everyday life in everything from bus arrival times to road traffic to order values to financial markets.)
When it comes to novel work, LLMs become "fast typers" for me and little more. They accelerate testing phases but that's it. The bar for novelty isn't very high either - "make this specific system scale in a way that others won't" isn't a thing an LLM can ever do on its own, though it can be an aid.
LLMs also are quite bad for security. They can find simple bugs, but they don't find the really interesting ones that leverage "gap between mental model and implementation" or "combination of features and bugs" etc, which is where most of the interesting security work is imo.
What was your take on this?
https://aisle.com/blog/what-ai-security-research-looks-like-...
I think your analysis is a bit outdated these days or you may be holding it wrong.
I am doing novel work with codex but it does need some prompting ie. exploring possibilities from current codebase, adding papers to prompt etc.
For security, I think I generally start a new thread before committing to review from security pov.
You can do novel work with an LLM. You can. The LLM can't. It can be an aid - exploring papers, gathering information, helping to validate, etc. It can't do the actual novel part, fundamentally it is limited to what it is trained on.
If you are relying on the LLM and context, then unless your context is a secret your competitor is only ever one prompt behind you. If you're willing to pursue true novelty, you need a human and you can leap beyond your competition.
of course you need a human but do not need nearly as many humans as there are currently in the labor force
Maybe, but I'm not really convinced. LLMs make some aspects of the job faster, mainly I don't have to type anymore. But... that was always a relatively small portion of the job. Design, understanding constraints, maintaining and operating code, deciding what to do, what not to do, when to do it, gaining consensus across product, eng, support, and customers, etc. I do all of those things as an engineer. Coding faster is really awesome, it's so nice, and I can whip up POCs for the frontend etc now, and that's accelerating development... but that's it.
The reality is that a huge portion of my time is spent doing similar work and what LLMs largely do is pick up the smaller tasks or features that I may not have prioritized otherwise. Revolutionary in one sense, completely banal and a really minor part of my job in many others.
I think the core issue (evidenced by constant stream of debates on HN) is the everyone’s experience with LLMs is different. I think we can all agree that some experiences are like yours while there are others that are vastly different than yours. Sometimes I hear “you just don’t know how to use them etc…” as if there is some magic setup that makes them do shit but the reality is that our actual jobs are drastically different even though we all technically have same titles. I have been a contractor for a decade now and have been on projects that require real “engineers” doing real hardcore shit. I have also been on projects where tens of people are doing work I can train my 12-year old daughter to be proficient in a month. I would gauge that percentage of the former is much smaller than later
People who are saying they're not seeing productivity boost, can you please share where is it failing?
I'm going to guess that many people on HN are also on the "free ChatGPT isn't that good at programming" train.
Not everyone has the capability to rent out data center tier hardware to just do their job. These things require so much damn compute you need some serious heft to actually daisy chain enough stages either in parallel or deep to get enough tokens/sec for the experience to go ham. If you're making bags o' coke money, and deciding to fund Altman's, Zuckernut's or Amazon/Google's/Microsoft's datacenter build out, that's on you. Rest of us are just trying to get by on bits and bobs we've kept limping over the years. If opencode is anything to judge the vibecoded scene by, I'm fairly sure at some point the vibe crowd will learn the lesson of isolating the most expensive computation ever from the hot loop, then maybe find one day all they needed was maybe something to let the model build a context, and a text editor.
Til then wtf_are_these_abstractions.jpg
> They're still using free ChatGPT copy and paste
Probably that's the reason why some people are sure their job is still safe.
Nature of job is changing rapidly
I totally get tech CEOs who threaten to fire their devs who do not embrace AI tools.
I'm not a tech CEO but people who are anti-LLM for programming have no place on my team.
And you are paying for their tokens on top of their salary, right? Right?
You can do a lot with just $20 Codex CLI subscription. Tokens are cheap compared to the $20k we're paying for a dev each month.
Even the $200 claude max monthly subscription is peanuts compared to salary cost.
Tell that to the company that I was just at that cut Intelij licenses as cost cutting measures.
If they really want to cut cost, fire the worst dev on the team and use that money to give everyone a Codex subscription.