Managing AI Coding Costs at Scale
databricks.com193 points by moonikakiss 10 hours ago
193 points by moonikakiss 10 hours ago
I would be really curious to hear from devs at Databricks what the experience of development is like internally. I work at a small startup with essentially unlimited AI spend budget - the entire point is that I should be turning to it at every opportunity since our human labor is so expensive relative to tokens. So generally it's like:
- Spend most time prioritizing/discussing what to do.
- Once that's agreed, use Fable 5 High + 5.6 Sol XHigh come up with a design + plan. Agree on the high level plan. (Usually this just comes down to choosing where the change belongs on the spectrum between minimal patch <-> full redesign)
- Use Opus 5 or Sol Med to execute
- Auto-fix bugs and CI until green + thermonuclear review skill x3.
- Manual interrogation of change/nits
- Come up with QA plan and have Codex Computer Use execute on it
- Manually spot check the final result (usually a sizable diff, thousands of lines, complete feature E2E, etc)
I probably spend like $80 a day at least but I produce the output of 3 or 4 2022 engineers and probably at better quality. So it's easily worth it. Would I save money by switching to GLM 5.2 and such...perhaps? IDK. At our scale it's not worth the time spent building the eval harness to actually understand the performance tradeoff.
In our team's experience, the product of agents is generally The Homer (1). It does work, but it's vastly overengineered.
When I personally want tight code, I have to spend a considerable amount of time adjusting it manually:
- It needs to be trimmed down. In my experience, at least one agent I use struggles to produce minimalist designs, and it's very frustrating
- I need to consider whether there are solutions based on higher-level assumptions, that AIs typically miss
- I need to check whether there are off-the-shelf solutions - AIs like to reinvent the wheel
IMO, software production has become a mass-produced commodity in every sense - it's much more expensive to produce software manually, but the quality is not the same.
It took me a few seconds of deliberating if The Homer was a reference to baseball or "The Odyssey" and then realized there was a footnote
> software production has become a mass-produced commodity in every sense - it's much more expensive to produce software manually, but the quality is not the same.
Agreed.
I believe “The Homer” is a reference to Season 2, Episode 15 of The Simpsons, “Oh Brother, Where Art Thou?”
Homer reunites with his long lost brother, who runs car company Powell Motors. Homer is ultimately tasked by his brother with helping to design a car for the “average man” that ultimately bankrupts the company for being wildly overengineered and costing too much ($82,000 in 1991-money).
Here’s the car: https://simpsons.fandom.com/wiki/The_Homer
If software has become a mass-produced commodity then seems to me the software business will become a much more finance focused business
You will really have to weigh the cost of making the software against the expected revenue.
Since software is still a winner-takes-all market, the mass-production property of software doesn't really matter.
In such markets, what you produce is either worth nothing or worth millions of dollars. For as long as it's the case that well-constructed code (with or without LLM help) is more likely to be in the latter category, the economics of software don't really change.
Even before LLMs, you could've commissioned a half-assed clone of any app you wanted from a 3rd world consultancy for a few thousand dollars. LLMs are basically Bangalore-as-API.
I think part of the reason software was winner take most was the difficulty of making software.
I remember hearing a story that in the past movies were so technically difficult to make that any movie that got made had a good chance to be a profitable hit. But as movies got cheaper to make, more movies got made. Nowadays movie studio execs have to really calculate out the audience and expected revenue for any new movie and balance that against the budget and the cost of the studio's failed movies.
I think a similar dynamic may happen in software
> I think part of the reason software was winner take most was the difficulty of making software.
That might be part of it, but I think it also has to do with the reality of replicating and scaling. Hardware or physical goods simply don’t scale like digital goods. There can be hundreds of knock-off physical products that have lower quality and lower cost but serve 90% of the same purpose, because physical capacity for raw materials, construction, labor, shipping, etc. have scaling limits in each market and economy. Digital goods are just so much easier to replicate and scale, so it often doesn’t make sense to buy software at lower quality and lower price if it doesn’t do most of the job. There are still limits of course, and different from physical goods, but I think this is a key reason why software is seen as winner-take-all.
That honestly sounds a lot healthier than “just ship what the CTO/Product team wants” with as much hand waving as is necessary to very roughly estimate ROI and then pray it hits with the market. In anything that’s not a startup operating in a new industry, the “old way” is a hard way to run a business
> Since software is still a winner-takes-all market, the mass-production property of software doesn't really matter.
Fairly sure software dev was always an iceberg. Most software and most software devs aren't working on horizontal software, but on vertical software, in cost centers. Sales for that kind of software don't scale as much.
No surprise, LLM companies optimize for waste. More tokens, and more prompts means more revenue. Reminds of Google’s Prabhakar Raghavan story: deliberately making search worse [1]
[1]: https://pluralistic.net/2024/04/24/naming-names/#prabhakar-r...
Or, more likely, it's that concise code requires a much deeper, wholistic, understanding that these models just are capable of yet.
Same with a junior dev. They don't write long form spaghetti because they're trying to write more LOC. They do it because not doing it is hard, literally above their pay grade.
I use LLM every day, but they're still completely awful at architecture. I don't think this clear lack of ability is some conspiracy.
As a business user, the same thing is true for non-code documents. The biggest exertion is reducing the excessive slop down to concise, clear points.
> software production has become a mass-produced commodity
For who?
The public? The public has never liked buying software at any price.
Businesses? Businesses need higher quality software when it's relevant to their core competencies, so they hire people instead. Buying competing SaaS or depending too much on AI is throwing the baby out with the bathwater.
This was more true a few months ago but Fable has improved the situation considerably.
Also just remember - minimalist code looks and feels great but customers do not read your code. I have caught myself many times providing "corrections" to abstractions that were already ~fine, just not perfect. The average SWE costs $200/hr. Careful you don't burn $50 worrying about code that will likely be rewritten or can be better abstracted when that's actually needed.
> The average SWE costs $200/hr
This is a pointless quibble but the hourly rate claim is not true--it's like ~$60 in the USA [0]. Maybe you meant at a specific Org but this is important context when comparing "pricing" between human and AI.
[0] https://www.salaryexpert.com/salary/job/software-developer/u...
How is this not true? Taking a Senior SWE @ ~$200K, even just the base salary cost / 2080 working hours is $100/hr. Fully loaded employer cost + accounting for non-coding time gets you to upper 100s easily.
Even for a junior making $100K, I have a hard time believe their time is worth less than $75/hr or so.
Edit: Fine, "Senior" is not "Average". But naive salary is not the true numerator.
I have a hard time believe their time is worth less than $75/hr or so.
Western Europe is mostly consultancy, and the rate paid by client is usually higher, and doesn't matter if it's eastern Europe, Portugal or even India.
It is. And the quality is on par with any us eng. People here forget that the big comp packages are a minority even in the US. The cost tho is much higher than just salary.
Company time != Pay rate, if you're working somewhere that's publicly traded check out "revenue per employee" metrics sometime.
I hire contractors for a large enterprise in the US. The going rate is typically $85-$100/hr for a senior dev, depending on specialization. Lead-level maybe $120 for the right skill set.
Of course, the SWEs making that much (over 200k) are not representative of the broader field. That's the point.
Pay hits a ceiling, and that ceiling is moving lower regardless of experience. That has nothing to do with AI, but what the market will bear. Hiring counts of humans must increase no matter what. Moving some of the spend to AI reduces the risk of hiring less qualified employees they might have rejected a decade ago.
Wages at the top end are stagnating to subsidize this. That's undeniable.
An MBA's rule of thumb is that a full time employee's hourly cost to a business is at least 1.5x to 2x times their salary depending on employer taxes, benefits, offices, travel, training, hardware, perks, etc.
Minimalist code is necessary to keep AI agents working well for longer than a month on a system IME. At a certain point, their own machinations overwhelm them and they both slow down, and make worse and worse decisions.
Also, you can probably think about it like this:
"Will I benefit from this code being minimalist before [date]", where [date] is whenever you think the agent will be good enough to come back and make the corrections you would make today.
I'd caution that some corrections become harder to make over time, rather than easier. A bad architecture now can become much harder to fix once other things have grown up around it.
Even as a self-contained unit, you can't step in the same river twice, and on [date] some important details may have seriously faded, both in terms of text that can be mined and also in terms of human "why did we do that" and "what was the reason we did it this way and not that way" etc.
> The average SWE costs $200/hr.
And this is how I find out I'm woefully underpaid.
That "cost" includes all the overhead provided by the company: benefits, rent for offices, utilities, equipment, etc. The average SWE is not taking home anything close to that, outside of Silicon Valley and a few other limited areas.
Whatever you are making this year as SWE you'll be making less next year if the current trend in improvement of AI coding aids is going to be sustained. Think about it : programmers used to derive a lot of their value from the fact that it was a hard skill to acquire. My kids can now 'vibe code' stuff faster (and better looking) than what I could come up with as the beginnings of a design plan. And then I still need to implement it.
There's a massive difference between your kids vibe coding something and an engineer using AI to implement something. If you're unable to discern the difference, that's something to reflect on :)
It doesn't matter if GP is able to discern the difference, it matters if your CEO is forced to care about the difference.
CEOs will always want someone who knows to implement so we're safe from kids vibe coding their way in but in a short while it becomes AI who knows who is managing less expensive AI.
The average SWE makes $400k a year? Are you being serious?
Costs, not makes. That includes employer taxes, benefits, offices, travel, training, hardware, any perks.
I'm also at a startup. My workflow is similar but I have Fable 5 xhigh drive the whole thing: it gets Codex CLI installed in its environment with an API key, and it's instructed to delegate ~everything to Codex and review its work, especially for code quality/conciseness. Fable delegates to Sol or Luna (fast mode) xhigh/max depending on the task - I think Luna xhigh on fast mode is basically a Pareto improvement over Sol medium.
IME this works until it does not. This approach works well at the beginning of a greenfield project, but at the same time because it is so easy to add features, you will likely ship something that is way too over engineered. And that complexity will not amortize over next increments and will more likely lead to the entire project being a black box only fully understood by AI. However a more careful use of AI for targeted surgical changes is far more ”productive” in the long term IMO.
I disagree, the approach works well in a legacy project, since there are structures and standards that already exist, which them model can draw from (if you aren't more explicit about it in AGENTS.md)
Disagree. I operate this way inside a multi-million line legacy codebase.
> I work at a small startup
How does a “small startup” end up with a multi million line “legacy” codebase? Something not mathing
> How does a “small startup” end up with a multi million line “legacy” codebase?
Easy! The output of 6 months ago Opus! Which seemed so wonderful at the time.
This! I don't think folks understand how easy it is to go from greenfield to brownfield with these tools, esp if your organization is only valuing velocity. Meaning your doing full agentic development on large features, barely reviewing any code, and shipping without much refinement. It's insane, but this appears to be the status quo in SF startups.
Have you worked at many startups?
Something isn't clear about the size of your codebase here and the level of reliability your customers expect, as a reader of your comments. Clarity there will help.
My observation has been:
- Initial greenfield work by an LLM is fast and very effective with minimal or no human oversight.
- Subsequent work ends up being over engineered and very verbose. Assumptions are made that aren't suited to the problem at hand (for example I find Fable is extremely regex happy where structured data would work much better from a readability perspective.)
- Once code bloats beyond a certain point due to unguided LLM usage, complexity is high enough that only LLMs can operate on the codebase with any economical amount of time.
- Rinse repeat and your code ends up unclear about any state that's not explicitly being tested and verified in QA loops
For some of our products this has been fine, for others it's been problematic. An understanding of your size and reliability requirements will help make the conversation more productive.
> unguided LLM usage
Why aren't you guiding your LLM usage? Is that what I said - to spam it and not guide anything? Or to have a careful workflow where you agree on design and maximize your human judgement/leverage?
> any state that's not explicitly being tested and verified in QA loops
As opposed to before, when engineers perfectly reasoned about code behavior from first principals and QA was unnecessary?
> Why aren't you guiding your LLM usage? Is that what I said - to spam it and not guide anything? Or to have a careful workflow where you agree on design and maximize your human judgement/leverage?
You didn't say anything positively or negatively regarding this so I made an assumption that you were using the LLM relatively unguided (e.g. a bit of oversight, not the kind of thing that heavy code reviews used to involve pre-agents.) Feel free to add clarity on your actual usage loop.
> As opposed to before, when engineers perfectly reasoned about code behavior from first principals and QA was unnecessary?
In my experience, most engineers are quite good at reasoning about code behavior for non-QAed code paths. Obviously things fall through the cracks. But I've been in the ground floor of plenty of Big Techs in their early stages before agents and, yes, a lot of initial development had spotty test coverage and yet most of the engineers had good mental models of what was happening. It used to be a very valuable skill to wrap your head around a torrid piece of code with few or no tests but was nonetheless a core piece of your application. Conversely, agentic development can bring cognitive debt [1].
===
This isn't a fight. We aren't sparring over what's right and wrong. I'm just curious how other people use agents in their work as someone who is also now in a startup that uses LLM agents heavily and has no limitations on spend.
> You didn't say anything positively or negatively regarding this so I made an assumption that you were using the LLM relatively unguided
I feel like this statement betrays your lack of advanced experience coding with LLMs.
OP's elaboration of the steps they are going through (planning, agreeing on plan, getting one LLM to draft execution plan, approving it, then executing with a separate LLM, then reviewing/testing) made it super obvious to me that they are guiding their LLMs quite considerably as part of their work.
Anyone making blanket statements about LLMs producing garbage is just telling on themselves about not having proper SDLC practices in place.