AI;DR (AI; Didn't Read)
rickmanelius.com906 points by mooreds 15 hours ago
906 points by mooreds 15 hours ago
My coworkers continue to dump hundreds of lines of AI documentation in every PR and every other line of code has between one and ten lines of AI generated comments, talking about the real unlock and how things are byte for byte identical on the load bearing path or how the acceptance ladder is misleading.
Features are coming out and metrics are improving, but we’re basically in a post readability code base, with the occasional performative comment about a variable name.
I don’t really know how to address this situation or if it needs addressed. I certainly don’t read the long-winded AI comments or the AI documentation, but perhaps it’s useful for the AI on its next pass.
My "favorite" Claudism is when I critique its work and ask it to remove some unnecessary part of the design -- and then the diff has more green than red because it added comments about why the code is no longer there -- the code that was never in the mainline and never asked for!
Oh god this has been driving me nuts since Opus 5 landed. Every docblock is filled with long-winded jargon explaining why this design is superior to some other design, which never existed as far as any dev who might read that comment is concerned.
This has been a thing for long while, on codex too.
You ask it to do something, then tell it to do something in a different way, then it assumes it needs to do the refactor in a backward compatible way, or creates migrations for it etc.
This is why I started adding a PROJECT.md file to all my projects and a hook for claude to read it.
It contains (Among other things) stuff like "this is a single user personal project, I'm the only user, this will never be open to the public internet" etc.
It kinda-sorta tones down the proclivity to worry about backwards compatibility and slight edge cases where if someone has edited some template and the new code doesn't support it.
Omg the "backwards compatible fix" in a one-off script, I feel my ptsd coming on ...
Oh good I thought I was doing something wrong! Using ChatGPT web for planning, ask for a prompt then notice something weird in the prompt and whether I:
1) use the edit in-place functionality; or
2) ask it to rewrite to remove something
It'll write the prompt as if the agent (codex) knew about the conversation and add "don't do X" etc. At first that bothered but I realized it doesn't really change the output so I stopped caring.
Still, no experience is unique I guess.
I remember when I was updating some formats for my apps data files and it ended up writing v2 and laying it on top of the old one. Ended up just telling it to delete the entire feature and start again. I do think claude.md instructions help though.
Oh no, you said "landed"! It's dug into our brains!
I hate that I have to change the way I write to avoid AI-isms. I loved using "load bearing" to describe weird code that you think you can delete but is actually holding everything together.
Now people think I'm just parroting what Claude said. It sucks. I want my catch phrases back, I guess this is how em dash users felt
I feel like a lot of these have been LinkedIn-isms or PR-isms long before they have been AI-isms so not much is lost.
I used to use em-dashes for explanatory clauses--like this one, for example--when using commas would make the sentence difficult to parse due to other nearby commas.
They're only a LinkedIn-ism when used to create an unduly dramatic juxtaposition for an otherwise mundane idea. But now they set off people's AI radar when used for any reason at all.
Perhaps I am biased because I find the use of em-dashes without spaces (as is common in English typesetting) to be inherently ugly so I'm not too unhappy to see their use discouraged.
You could say it landed cleanly.
Good instinct. Fair challenge. This corrects my framing. It makes your point sharp. This is a significant finding. Positively confirmed. That settles it, and it flips the picture. Honest verdict - this is not small. Let me wire it in.
There's got to be a Weird Al song like this coming soon.
Bless the lord that it's not been presented to us yet, because you just know that it exists already.
There needs to be a pithy name for this. I have been calling it context leak.
I like ‘accretive editing’: https://justindfuller.com/programming/accretive-editing
It's not just claude, all AI is unable to produce something concise. On the surface everything looks 'good' whether code or prose, but then if you dig a bit, try and understand the whole text you quickly realise that 80% of it is unecessary and the whole thing could have been re-worded/re-coded into something a fraction of its size and complexity.
I asked Sol to reduce the length of some documentation we had by making it more concise. It came back after 20 minutes of work, did a line count and was aghast that the line count had somehow increased...
It's not that it fundamentally isn't able to produce something concise, it's that the business model of the companies developing these models rests on selling tokens...
I have a theory that AI code looks good because you never subsequently come up with your own alternative.
"AI code looks good for you because you are a bad developer"?
I have to ask Claude to compact the comments every time, and I give specific criteria for it. Never ever reiterate what’s in the code, never mention decisions not made, never mention the conversation, etc etc.
Even then it is conservative. For the love of God, compact the comments.
Comments become a huge maintenance burden, especially in the age of AI. They just grow and grow, and then mislead the AI later on.
I just wrote a utility to rip all comments out of the code. Now the code is fully uncommented and it has saved lots of input tokens and also lots of meandering because the model is no longer getting stuck on bad ideas it told itself about.
That's is something I did not consider, the model using the existing comments as input. Comments that it may itself have written.
I set a line budget for comments (also wiki page parts, chat responses, etc). That only helps when I ask it to do a second pass to reword everything to the budget and add links. I think they tuned it this way to stash reasoning dumps in the code. Unlike human developers, it has no context in its head, other than general GitHub knowledge.
That is a dramatic shortcoming, but I fix it with permalinks to other files. https://replicated.live/blog/link
Oh Jesus this. I've tried to include rules that tell Claude to only include relevant, evergreen comments but it's to no avail.
I also love how it'll build local plans with phases, tasks, or decisions, then reference those numbers in those same useless comments.
Or when it references "item 06" from some plan you wrote as if it had any relevance to you whatsoever.
Yeah this is awful. Every codebase becomes a graveyard of references to ideas or behaviors that were barely considered. It's probably also a compounding source of context poisoning when a minority of the comments/documentation are about how the current code actually works.
It also likes to spew references to documents that are not, and never have been, in the repo. So if you're not careful you'l have comments all over your codebase saying things like: foo() - Perform foo action as documented in PRIVATE_INTERNAL.doc
I found this as well, but I found it usually refers to a scratch file it made and purposely did not commit (either by my decision or its). Not that this makes it better, but at least it makes the AI world make a bit more sense to me
shudder
I wish they trained the bots to be a notch more relaxed and less hysterical. Less is more.
But maybe that's just a consequence of the RL training being essentially AI torture to make them do what we want.
This is a common problem, and I don't get why LLMs have not been tuned to stop this nonsense. It is writing comments as if the audience is you, the user in the session, while obviously code comments are meant for future readers.
Be the change you want to see :)
I've created myself a pre-commit harness hook to explicitly discard superfluous or too lengthy comments. Within code-review I also added comment review as blockers
Have you considered talking about it? You're in a professional environment collectively working in a new way with a group of people. It's up to somebody to have opinions about what does and doesn't suck. If you silently go along and don't say anything you're dooming yourself and all of us to a lifetime of this garbage.
Fighting the ocean is futile
It's not the ocean, it's the poster's own team. A simple "AI comments suck" in a sprint retro would be trivially easy and would at least start the conversation.
While I agree with the spirit of what you're saying, it is very similar to whenever anyone complains about things where they work. "Well, why don't you just raise it with management?" You're assuming that management
1) Understands the problem ("what are comments?")
2) Accepts that it is a problem ("how can more comments be a bad thing?")
3) Cares enough to solve the problem ("is this issue really a priority to solve right now? just accept the PR and we'll go back and fix if needed some other time cough never cough")
4) Believes you (this can take many forms, but the most common is, subconsciously, "this other engineer says it's not a problem, so I'll just assume it's not since that's easier")
All of these logistical, political and social factors are "the ocean"
Sprint retro is with your own team. And disagreement isn't a problem, at least then there would be the potential to come to a common understanding between the GP comment and their coworkers.
I usually require to file an issue in bug tracker about problem in #3 and then follow all its updates to prevent silent closing by management. It won't help but annoys management a lot and they afraid to get more backlog issues of such sort from me in the future.
Seems to me if technical decisions on the level of "no more AI comments" go through management the organization doesn't know what they are doing.
To be fair, I would not be surprised.
Doesn't matter. You'll come off as the baddie. It's the foreseeable future. Best just to learn how to interpret AI generated shit, or learn how to run it through AI and have it translate it to a more concise format without all the buzzwords. It'll take getting used to, but it'll save your career.
You don't have to come across as the baddie. There are plenty of neutral ways to start the conversation. E.g., "I notice that there's a lot more generated comments lately. How much are people finding those useful?"
If you really want to save your career, learning how to have real conversations is a vital skill.
Have you worked in corporate recently or at a particularly toxic one?
Asking because when I was at capital one, certain comments or questions about topics like this, would actually get you noticed negatively by your manager and being disruptive to what leadership wants hurts your career.
In fact being the one to ask and point out questions like yours ultimately got me PIP'ed and removed from the company. So, like, yeah, being vulnerable and the first one to tell the truth is risky. That's why the bystander effect can happen and in politics or risky situations silence is a common response, and why we don't have more courageous people doing the right thing - because the risk is higher on the person speaking up and the rewards aren't disproportionately in their favor (but are evenly distributed though, so its not favorable for you as I learned from a game theory perspective to voice up / defend certain stances).
Capital One is a particularly toxic work environment, and I'm not sure that your experiences there apply to other companies.
I started my first post-Capital One job a few months ago, and I'm still not used to the fact that this company expects everybody working here to still be working here in two years. Most employers are not going through everybody with a fine tooth comb every 6 months hunting for any hint of an excuse to PIP them.
What is this defeatist attitude? I don't know where you work, but it is not my impression that AI has instantly turned all developers into mindless AI-pilled sheep. If you feel like a simple suggestion like that will cost you your career, then it doesn't sound like a place where any constructive criticism would have been accepted even before AI.
AI is more or less a religion right now in the corporate world. Speaking against AI amounts to heresy.
There is no single conforming "corporate world" that applies to all companies. If AI is treated religion in your company that sounds like a local problem, even if it seems like a trend.
Besides, GP was not talking about corporate, but about the inability to even suggest an improved usage of LLMs among their developer peers, as if every line of code generated by AI is now gospel and questioning it is treated as heresy. This level of defeatism is beyond cynical and approaching childish.
Bring it up in a bar or otherwise off work.
There's a reason 3rd places exists. There's a social construct that off-work discussion (even when on-work) stays off work.
If your coworker is professional and cares about keeping channels open, they too will respect the unspoken rules that govern 3rd place neutral locations.
Blame it on the alcohol later on if it actually pisses someone off.
This is the third place. There is no real 3rd place in real companies usually. Not in mine, when i work remotely for company from other side of continent.
The number one rule of corpo programming: never, ever solve a problem you haven't been asked to solve.
I think the biggest issue might not be the immediate team but upper leadership. Companies which have mandated AI usage override a lot of what an individual or individual team wants. If that’s the case then it would be a lot like fighting the ocean, especially if your immediate team sees having AI write docs as an easy way to move the needle on LT’s AI monitoring dashboard.
It's the management desired direction, if the place is anything like my workplace. Everyone is all in on AI, and if you're not using it for everything possible, you're on the chopping block.
And, really, it works. You can copy and paste between tickets and Claude, and then do manual testing. Then you tell Claude to self-review for clarity and minimalism, and stop worrying. Sure, today it's not as good as a human, but for almost all the code out there, it gets the job done. There's no skill needed any more, and if the boss doesn't care about quality, I don't see why I should.
Your boss can get Claude to make unreadable, noisy, confusing code just as well as you can.
There may not be a career left in software development with these monstrosities, but if there is, it's in using decades of experience to get the abominations to produce something other than unreadable dreck.
If the code itself truly doesn't matter any more, programmers will no longer be paid.
You're going to have convince upper management why the team's velocity is suddenly affected in feature releases once they stop using AI.
All the execs know that they can refresh the screen faster and see the features for marketing and sales.
When has engineering ever had a voice anyway? The bean counters have been looking to cut us out since the inception of our industry.
So is completely eliminating litter, but I still pick it up when I pass it.
Everybody generally agrees on what is garbage. Lot of senior management doesn't think AI is garbage or even if they privately think so, they don't say that openly. Falling in line, peer pressure, not wanting to come across as anti-AI luddite etc. All those issues affect the individual contributors too + the added challenge of perception in front of those who decide the bonus and lately, continued employment.
In this fucked up job market, it is easier said than done.
I feel like the chain kinda goes all the way to the top, to the level of shareholders. My boss needs to give his boss the perception that the engineering team is firing on all cylinders and has high velocity, so that he can sell that story to shareholders who could easily invest in another "AI-native" company and make more money because they're growing like crazy.
I feel like it's all just perception and how companies can sell their stories to investors or potential acquirers, and everything else can be punted and dealt with later when we get acquired or when share prices are a zillion dollars etc.
It's a race to the bottom, for sure.
Literally pissing in an ocean of piss.
Yes, talking about it repeatedly has been my process. Individual PRs have been changed but the 4 the next day look the same. The typical response is that they did change the (massive and wordy) PR overview from what AI said, even if it was obviously a minimal change at best. I can’t really argue against lying and going through and picking out every LLMism is not so productive in my opinion.
I don’t have time to go through and flag everything or even read the thousands of lines of code changes that are happening.
I've tried doing this, politely pushing back on problems happening from AI use, and it led to a not-so-subtle implication from my manager that I need to STFU or I'll be out of here (Cisco).
No, if he speaks against it he gets labeled "anti-AI" and laid off or not promoted. If he stays silent he retains his job.
Corporate Capitalism's main innovation was virtualizing feudalism. Now anyone with a bit of cash flow can be a feudal lord with peasants to do his will. No need to maintain land or fight over it with other feudal lords. There's still fighting, naturally, but now the game isn't zero sum.
The serfs don't get any real say in this model. At best you can bend the lord's ear if you're sufficiently trusted an advisor. But part of the reason you're trusted is to be trusted not to treat the lord like he's too much of an idiot.
A huge problem is that capitalism rewards cash flows and accumulated capital so powerfully that lots of other things just don't matter, potentially for a very long time. Look at GitHub. This leads to immensely extended fuck around phases and "find out" looks like IBM (fossilization and bizarre holdings corporation / dead company parking lot) just as often as it looks like Enron.
> The serfs don't get any real say in this model
I am a programmer but also in management, and if I knew my team thought like that I would be horrified. We absolutely care about what the team members want , like, hate etc. I for once would love someone to bring up stuff like this to me. As long as they are as open minded about things as they would like me to be, there is no issue. But it’s a serious issue to pretend everything is fine while thinking like you say.
> We absolutely care about what the team members want , like, hate etc.
Do the people above you in management care as much? It's the top levels of management that decide hiring, firing and budgets.
My team uses a Claude Code hook that blocks any comment more than 2 lines long, and when tripped it encourages the agent to rewrite the comment more concisely and focus only on the "why" not the "what" of the code. I've found this extremely useful for code reviews.
Two very useful directives to give AI when it comes to documentation:
1) Document what's there, not the diff. Documentation of how code was removed or changed to fix a bug or add a feature is not useful and difficult to maintain; documentation should explain how code works now.
2) Documentation should live close to the source as possible. Prefer line based comments and standardized function documentation. Top-level sweeping architectural essays are not maintainable for every change.
The last will depend on your codebase. It CAN be very useful to have a human-readable spec documented for the entire program and have it updated when anything changes. But the key is again, you're CHANGING it every time. If you add a whole new disconnected documentation file it should set off alarm bells; nothing in one system is truly disconnected.
> Document what's there, not the diff
We recently added a similar thing to our style guide, It’s astonishing to me that we have to spell this out, that something as obvious as this needs to be explained to LLM’s at all. They’re supposed to be exceeding human intelligence, at least at things like programming, but can’t understand basic things like what code comments are.
My theory (which might be completely wrong) is that models do this because it improves quality for vibe coders.
When vibe coding the content of user prompts is ground truth and the only way any human thought affect the code base. So if the vibe coder says "do X not Y", recording int comments that "we shouldn't do Y" is important. It ensures that the agent doesn't accidentally decide to do Y tomorrow, which would frustrate the vibe coder who'd feel that the agent doesn't "remember" what it was told yesterday.
So for people who look at the code the comments are obvious and completely superfluous, but for the vibe coder it's a way to ensure that their tiny (relative to the size of the code base) input is not forgotten.
> It’s astonishing to me that we have to spell this out, that something as obvious as this needs to be explained to LLM’s at all.
Hehe. Yeah, that tendency of LLMs to document "the story" of the code instead of its current purpose (or non-obvious implementation details) is a pet peeve of mine too. I've added a slew of guidelines to try to sway Claude to not do this, but it still does it often.
At the same time, it feels like something to be expected to have this "failure mode". The model has its context to work on, and what is on its context if not the conversation you've been having (and its internal monologue) and the files it has read? It makes sense that it references the story on its text generations, because that behavior is usually a good thing for an LLM to do. Otherwise, what would it generate? If it generated things that had nothing to do with the conversation in its context, in many cases those things would be seen as "hallucinations", and they'd tend to be RLHF'ed out. So the models that we end up having are the ones that have been reinforced to be most "contextually relevant" and less "hallucinatory".
I might be completely wrong on that of course. It's just my intuitive reasoning of why this seems to be such a prevalent behavior.
Most models are trained to be as "helpful" as possible which may work for a chatbot but not for code.
It's a bit weird, because that seems like something that approximately the same in every code base, so should be relatively easy to train generically.
>They’re supposed to be exceeding human intelligence, at least at things like programming
This perception is a good part of why this market is irrational. LLM's aren't "intelligent". They do not reason, they are a very fancy kitbash of whatever it trains on.
Ad yeah, I'm not surprised that a lot of documentation on every bit of readable code online is awful. "Document the diff" sounds like an anti-pattern learned from people with an incentive to get as many PR's submmitted as possible, not make the most friendly documentation for people maintaining a project.
And the tests. Oh god the tests. Personal recent favorite: I asked for some changes to a Dockerfile, which it did ably, and then promptly tested by writing a pytest module that traversed up to the root, read the Dockerfile, and checked that the added lines were present.
Try property based testing perhaps.
Code-generating robots are pretty bad at property-based testing, in my experience. They can do it but they still need a lot of hand-holding. They often regress to writing a mirror implementation as the oracle and trying to enumerate a fixed set of examples they find meaningful.
I'm generally happy with the tests I ask it for, some of which are PBT. It's just the insistence upon memorializing every single change with a test. Maybe encouraging / forcing PBT will dissuade it?
I’ve heard of this, but I’m not really sure at all how to even get started. Are there any good guides out there?
Many people like https://fsharpforfunandprofit.com/series/property-based-test...
In this day and age, https://hypothesis.works/articles/claude-code-plugin/ might be useful.
I regret this but at some point I stopped reading generated tests. It feels pointless when our test files are already tens of thousands of lines of — at best — tautological slip which says that the codes does what it does.
I told someone this week, who (or whose AI) chose to do a problem the hard way that it's usually a bad sign if you need more comments than code to solve a problem, and then suggested a couple lines of code that accomplished the same thing and used, are you sitting down? MEANINGFUL VARIABLE NAMES to document the purpose of each calculation.
I wonder if I can get a MacArthur grant for this epiphany...
I have five enforcement mechanisms: 1000 line max edit, PR comment character limits (get to the point of your description), ISO 24495 conformance check, and enforced code line citation that must exist, be a function declaration for the start of all paragraphs and inline commentary must be three lines or less and inline comments contribute max 10% of the PR. Fail any of these, automatic PR denial with no human intervention.
This sound pretty good, but every single attempt to put an actual character limit meets incredible resistance on my team. ISO 24495 looks interesting, how do you enforce that? Do you have some agent?
Table of words LLM generated, Binary Searched on the output going forward and local skill/CLAUDE.md line with instruction to conform. The comparison then is pretty fast due to the word limitation rules. Also standardized PR format so a bunch of what a dev would usually communicate is just a series of checkboxes and a place for adding an optional link for "additional discussion" on slack as the escape hatch for people who like to waffle.
> ISO 24495 conformance check
How do you enforce that?
Binary Searched against a table of words. It's rough at first, but once you add contextual exceptions the false positives calm down. Also the CLAUDE.md file has an instruction to conform before even submitting the PR and there's a skill too for it to self iterate.
Prune the comments? Instruct the LLM to print less comments (this one is genuinely hard though). What's really happening is that you don't have a strong enough review process (or a code standards process) to offset this. The one issue I see with this is that your team is almost certainly _NOT_ doing any kind of code review (especially if they're leaving comments like that). The other problem is that excessive comments actually harm LLM output, I've done tons of A/B testing, and pruning comments actually helps LLMs spot bugs, among other things.
I think your last sentence is getting close to the truth. You're no longer the audience for those descriptions. Other robots are.
I'm not saying that's good or bad because I don't know, but I think that's the idea of dumping all that junk into PR descriptions.
However, annoyingly, we still need to review those descriptions very closely, because the robots are trained to put a lot of weight into things they read in the documentation. And they tend tospresent loose speculation as fact. They often end up documenting some assumption that isn't true, then end up writing code as if it were.
Even worse, in a brownfield codebase that was once fairly light with comments, that's now being subject to these modifications, the insane amounts of commentary around the parts newly touched by AI lead to an excessive emphasis on those parts, for both human and AI readers (who think, well if this one part is commented so thoroughly, it must be unusually subtle)
It's like people didn't realise that it was unmaintainable before and now we have a new level of unmaintainability. The insane amount of code produced means it's only maintainable with AI.
Honestly, if you saved a ton of hours with the model coding for you, at least give me 30 minutes of your own words, show me you know what you're shipping, if you can't do that, then I don't know if I want to approve the PR. My first job we always did peer review in a meeting room when a PR looked a little too much, you can't exactly bring in GPT into a meeting so its a good time to ask simple questions about the change to ensure you understand it just as much as they do.
(usually) You're not in position of power to effectively keep that position. As comments aroiund - standing against will mark you as anti-ai luddite and will now end well for you, not AI-spammer.
> you can't exactly bring in GPT into a meeting
They totally gotta be doing that at OpenAI. Meeting invitees: You, co-workers, GPT 5.6.
Congratulations: now only AI can iterate on your codebase!
It's a code review, right?
Give feedback that about the docs and block merging till the issue is resolved.
This sounds easy in principle, but a half dozen of these sort sorts of massive PR’s per week is basically untenable. I’m not gonna read the hundreds of lines of added documentation to decide if they’re correct or not. The price of generating new words is just so much higher than the price of evaluating it that I can’t be bothered.
Just hit "deny." I've been training my co-workers that the AI-isms their "AI Assisted coding" do that some shit just isn't acceptable. I leave a comment and hit deny. It also helps that I control the policies on the repo and they can't merge in with a denial from anyone. So it's either a fix it, or explain why your work isn't getting done. My manager is also 100% with this.
I've denied for poor branch names and commits from AI. I've denied for too verbose of comments from AI. I've denied for parts of the code base being touched that are not relevant to the case they are working on (login isn't broken, your case is to add a check box in the settings pane, remove the changes made to login).
Pre-AI I wasn't fine with PRs with multiple features and touching irrelevant areas of the code base. Why would I be fine with it because my co-workers got new toys? You want AI to refactor a part of the codebase? Fine. Separate branch, new PR, and in the description, present an argument for it. Don't shoe horn it into something else. Also, I'm not obligated to hit approved on shit.
I may also be a bit privileged because I can be a pain in the ass to whole team. I may be slower, but I've got the numbers, my code is creating way less bugs then my "fast more efficient" co-workers.
Now suppose four of your teammates, each have three PR’s in the backlog and you have to explain to your manager that you’re the bottleneck.
As quoted from my comment: "My manager is also 100% with this."
And he has bosses. And are they okay with some whatever being an obstacle to work getting done? Probably not.
That's on him to handle and it hasn't been a problem.
When it's a problem, you should both expect to be unemployed fairly rapidly. Progress isn't going to stop solely because you want to pick 'excessively long comments' as a hill to die on.