Tell HN: DigitalOcean's managed services broke each other after update

Usually those agreements either just give you credits for the same service, pay way less than you lost or basically everything falls under force majeure.

If it works for you that's great, but when the actual shit hits the fan I don't think you should expect actual compensation.

At our scale I doubt if we can get any cloud provider to write custom contracts. But if I had negotiating power, I completely agree.

> could you do better yourself

For this particular failure mode absolutely - this is amateur-level stuff that shouldn't have happened.

You know how to make something that works keep working? Not messing with it. Of course, this doesn't pay salaries if your entire career is based on "fixing" things that work until they don't.

There is no reason to hurry a Postgres upgrade - the thing shouldn't be internet accessible anyway, so no risk of security issues.

If you do want to update, it's best to test the update on a test/staging system. Which I'm sure they would have if they didn't have to pay a 10-90x markup on the compute price.

Finally, when you do the update, you'd do it manually during a time where you are present and outside of business hours to further minimize the impact of something going wrong, instead of the upgrade happening out of the blue at a random time.

> but vendors are just as fallible as the rest of us

Isn’t the point that they shouldn’t be. They should have specialists dedicated to running these kind of things, test upgrades before rolling out, et c., while for the rest of us it’s just one of many things we try to handle.

> vendors are just as fallible as the rest of us

One of the issues I have with this is the insane markups they're charging for services that ultimately aren't any better than what you can do yourself.

If they aren't any better at least save yourself some money.

> though it probably goes down less than something you self-manage, unless you're a full-time SRE with the experience to back it.

I wonder how true that is. This went down because of a bad update, which is probably like 99.99% of outages. The other 0.01% is cosmic rays causing hardware failures.

My server was up for 3.5 years with no outages because I just didn't touch it. I had to take it offline a couple days ago to move it which made me a little sad. Took a snapshot and moved it to a new droplet, brought it back up as-is and it's running great again.

Anyway, emergencies are less emergy if things go down while you're upgrading and shuffling things around yourself. You expect hiccups if you're the one causing the hiccups. It's when someone else is tinkering on the other side of the country/planet and blows something up that suddenly you have an emergency.

I've read a few horror stories, but I always thought it wouldn't happen to me :)

> It's not my problem, I opened a ticket, now I'm going to get lunch, hope it's back up soon.

That's a good way of thinking about it.

Bonus point is that with such a simple stack you don't need to phone them often because the thing just works.

Most cloud outages are self-inflicted with the endless churn trying to reinvent things, not actual hardware failure. Just not touching the working system would boost their reliability and uptime, but then a lot of people would lose justification for their salaries so it can't happen.

Wait, customer support with a competent sysadmin? You're not making this up? It sounds ethereal.

This is the way... you live in operator heaven.

I've had an RDS instance fail in an equally-weird way that required manual AWS operator intervention 24 hours later, until which we were effectively locked out of our data and had to restore from a recent backup & rebuild the missing data from logs to bring the service back up in the meantime.

All the supposed "savings" of using managed services to save on staff costs evaporated immediately. No refund from the provider obviously despite it being an edge-case in their implementation.

We were on AWS for a while. The complexity was way higher than what our team could manage. DOKS is simpler, and this is the first major issue we've hit in many months.

AWS frequently has outages. us-east-1 anyone

The font color implies this comment is downvoted, but I earnestly encourage readers to take very seriously the difference in SLOs and SLAs between high-cost vendors like AWS and GCP and low-cost vendors like DigitalOcean. Read their docs; do not assume DO is "the same, but lower cost."

As the sibling comment already mentioned, k8s is not much more complexity once you're past the learning curve. I used to host with ec2 + scripts earlier. K8s actually solves a lot of problems that you will have to solve yourself anyway.

As a solo dev who just started his second cluster a few days ago... I like it.

Upfront costs a little higher than I'd like. I'm paying $24 for a droplet + $12 for a load balancer, plus maybe $1 for a volume.

I could probably run my current workload on a $12 droplet but apparently Cilium is a memory hog and that makes the smaller droplet infeasible, and it seems not practical to not run a load balancer.

But now I can run several distinct apps running different frameworks and versions of php, node, bun, nginx, whatever and spin them up and tear them down in minutes and I kind of love that. And if I ever get any significant amount of users I can press a button and scale up or horizontally.

I don't have to muck about with pm2 or supervisord or cronjobs, that's built in. I don't have to muck about with SSL certs/certbot, that's built in.

I have SSO across all my subdomains. That was a little annoying to get running, took a day and a half to figure out but it was a one time thing and the config is all committed in YAML so if I ever forget how it works I have something to reference instead of trying to remember 100 shell commands I randomly ran on a naked VPS.

Upgrades are easy. Can upgrade the distro or whatever package easily.

Downsides are deploys take a minute or two instead of sub-second.

It took weeks of tinkering to get a good DX going, but I've happily settled on DevSpace. Again it takes a couple minutes to start up and probably oodles of RAM instead of milliseconds but I can maintain 10 different projects without trying to keep my dev machine in sync with everything.

So some trade-offs but I've decided it's a net win after you're over the initial learning hump.

Running Kubernetes in a managed environment like DO is no harder than using docker compose.

Your grand total of one submission is a link to Razer.com. When you actually contribute something to this community perhaps then you can make a statement like that. Still, probably not the best.