How to Migrate from OpenAI to Cerebrium for Cost-Predictable AI Inference
ritza.co43 points by sixhobbits 13 hours ago
43 points by sixhobbits 13 hours ago
This isn't really about cost savings, it's about control. Self-hosting makes sense when you need data privacy, custom fine-tuning, specialized models, or predictable costs at scale. For most use cases requiring GPT-4o-mini quality, you'll pay more for self-hosting until you reach significant volume.
The "not optimized" self hosted deployment is 3x slower and costs 34x the price using the cheapest GPU / a weak model.
I don't see the point in self hosting unless you deploy a gpu in your own datacenter where you really have control. But that costs usually more for most use cases.
Is there actually some scale magic that allows the 34x cost saving (over 100x when you include performance), or is it just insane investment allowing these companies to heavily subsidise cost to gain market share?
Calculating without energy costs: The A10 Gpu itself costs 3200$. With a 3 year usage that is 0,002$ per minute. From the blog post the cost per minute is charged at 0,02$, so a premium of 10x. So with energy if you can load the GPU at minimum 15-20% self hosted becomes cheaper. But you need to take care of your own infrastructure.
With larger purchases the GPU prices also drop so that is the scaling logic.
> I don't see the point in self hosting unless you deploy a gpu in your own datacenter where you really have control. But that costs usually more for most use cases.
Not wanting to send tons of private data to a company who's foundation is exploiting data it didn't have permission to use?
How to move from one service that is out of your control to another service that is out of your control.
Having the ABILITY to move seamlessly and without significant cost is absolutely critical.
It gives you flexibility if the provider isn't keeping pace with the market and it prevents the provider from jacking prices relative to its competitors.
Vendor lockin is awful. Hypothetically, imagine how stuffed you'd be if your core virtualisation provider jacked prices 500%! You'd be really hurting.
...ohwait.
You're not really locked in in any meaningful way currently, you just switch the API you're using. Rather like is being demonstrated here.
I don't understand - what do they mean when they say you can run things on your own infrastructure then?
To people from Cerebrium: why should I use your services when Runpod is cheaper? I mean, why did you decide to set your prices higher than an established company with significant user base?
(Not affiliated with Cerebrium, just did a bit of looking into this a little while back).
Runpod outsources much of their infrastructure to small players that own GPUs. They have recently added some requirements on security and reliability (eg: some level of security audit such as SOC 2, has to be hosted in a real DC, has to be in a locked rack), but fundamentally they are leaning on small shops that slap some GPUs in a server at a colocation facility. This personally would make me nervous about any sensitive workloads.
My impression is that Cerebrium either owns their own GPU servers or they're outsourcing to one of the big players. They certainly don't have the "partner program" advertised on their site like Runpod does.
Hey! Founder of Cerebrium here.
- Runpod is one of the cheapest but it comes at the price of reliability (critical for businesses) - We have more performant cold start performance with something special launching soon here - Iterating on your application using CPUs/GPUs in the cloud takes just 2–10 seconds, compared to several minutes with Runpod due to Docker push/pull. - Allow you to deploy in multiple regions globally for lower latency and data residency compliance - We provide a lot of software abstractions (fire and forget jobs, websockets, batching, etc) where as Runpod just deploys your docker image. - SOC 2 and GDPR compliant
With that all being said - we are working on optimisations to bring down pricing
Why? Honestly, there are already tons of Model-as-a-Service (MaaS) platforms out there—big names like AWS Bedrock and Azure AI Foundry, plus a bunch of startups like Groq and fireflies.ai. I’m just not seeing what makes Cerebrium stand out from the crowd.
Well, they are announcing their $8.5m seed round and hope to attract the maximum number of users by giving away $30 in credits.
Do you forecast costs or just provide more visibility?
Is this article just saying openai is orders of magnitude cheaper than cerebrium?
It's a demo project using the free tier hardware from Cerebrum, demonstrating how to migrate with a few lines of code from OpenAI. The cost is never going to beat OpenAI on an A10, there are more powerful options available.
I’m trying to figure out the cost predictability angle here. It seems like they still have a cost per input/output tokens, so how is it any different? Also, do I have to assume one gpu instance will scale automatically as traffic goes up?
LLM pricing is pretty intense if you’re using anything beyond a 8b model, at least that’s what I’m noticing on OpenRouter. 3-4 calls can approach eating up a $1 with bigger models, and certainly on frontier ones.
Serverless setups (like Cerebrium) charge per second the model is running, its not token based.
You're still paying more than the GPU typically costs on an hourly basis to take advantage of their per-second billing... and if you don't have enough utilization to saturate an hourly rental then your users are going to be constantly running into cold starts which tend to be brutal for larger models.
Their A100 80GB is going more than what I pay to rent H100s: if you really want to save money, getting the cheapest hourly rentals possible is the only way you have any hope of saving money vs major providers.
I think people vastly underestimate how much companies like OpenAI can do with inference efficiency between large nodes, large batch sizes, and hyper optimized inference stacks.
I'll echo one of my original concerns, which is how is this supposed to scale? Am I responsible for that?
How is what supposed to scale?
If you mean the serverless GPU offering, typically you set a cap for how many requests a single instance is meant to serve. Past that cap they'll spin up more instances.
But if you mean rentals, scaling is on you. With LLM inference there's a regime where the model responses will slow down on a per-user basis while overall throughput goes up, but eventually you'll run out of headroom and need more servers.
Another reason why generally speaking it's hard to compete with major providers on cost effectiveness.
Past that cap they'll spin up more instances.
Thank you, this is what I wanted to know.
typically you set a cap for how many requests a single instance is meant to serve
If this is on us then we'd have to make sure whatever caps we set beat api providers. I don't know how easy that cap is to figure out.
If you're making the effort-cost tradeoff like this, you typically choose a model, test a few inference stacks with prompts that are representative lengths for your use case, then benchmark.
To benchmark you identify a maximum time to first token your users will accept, and minimum tokens per second they'll accept, then test how many concurrent requests you can handle before you exceed either limit.
I can tell you, in my case the only reason why the pricing is somewhat competitive for self-hosting is that I'm aggressively seeking cheap rentals, have a use-case that requires very long prompts with few cache hits, and I've used extensive (and expensive) post-training to deploy smaller models than I'd otherwise need.
[dead]