Active incident at Dreamhost forcing password reset
dreamhoststatus.com165 points by greenchair 3 days ago
165 points by greenchair 3 days ago
Reading the tea leaves: given that the outage started with "connectivity issues affecting the File Manager", which was an install of Monsta FTP [1] - a commercial PHP application - I suspect the file manager got popped by one of the recently published PHP vulnerabilities, and users' FTP credentials may have been exfiltrated.
20 year customer here. This morning I asked their support chatbot if this is related to the PHP vulnerabilities and it assured me that's not the case. I'm skeptical.
These status updates from the PHP incident would also have us believe that they were never directly affected: https://www.dreamhoststatus.com/pages/incident/575f0f6068263...
They're being awfully vague about this issue so far. 8 hours ago I submitted a ticket requesting more information and they've not responded yet. I guess that they're still in the thick of it.
Why would their support chatbot know?
Having worked in support for years quite often higher layers of the company will give T1 support almost no information on what happened intentionally. That's pretty standard corporate CYA policy in litigious societies.
I didn't think that it would know but their support UX funnels you through it so that's where I started. I followed it up with a ticket.
I would never expect first tier support to know, let alone ever admit, something like that.
Asking the chatbot is as useful as asking the pet parrot.
You're being unkind to the parrot. Parrots are known for mimicking others in their environment.
One might reasonably expect it to scream/shout and swear if one mentions PHP.
hopefully their hashed/salted passwords, right? not plaintext?
Internally, sure. But the application that was (probably) compromised was an FTP client, so it would have received the plain text of those passwords from users as they logged in.
> the application that was (probably) compromised was an FTP client
Thank you, this actually makes some sense. It's disturbing to get these emails with zero explanation what's going on, and very frustrating that user community has to guess. But PHP vuln in a web client intercepting user logins is the only thing that makes sense, otherwise that would mean passwords in the clear or some other 1990s-era nonsense.
For those customers who are still logging in with FTP…
As a precaution they reset everyone’s email passwords, presumably from an assumption people use the same ones for both.
I’m surprise they still let people use FTP.
I came across such a thing recently. The counterparty was a bank, and they point blank refused to update their processes so the people I talked to had no choice but to continue to use something they knew was hopelessly outdated. SFTP was released what, 3 decades ago or so? There is a good chance that whatever was on the other side of that upload was not capable of any kind of secure connection.
I dont really see a problem with ftp over TLS. The protocol might be dated, but it still works fine.
This has been a real annoyance for me lately since I run a lot of my music production business out of my website lexwhite.net hosted on Dreamhost and Dreamhost VPS. I love and trust Dreamhost a lot, though, as they have been an exceptionally solid company that I have worked with for, dang, nearly 20 years now I think! All of this new AI stuff has been exposing vulnerabilites at a rate that is hard to keep up with for anyone. We just have to really harden our cybersecurity as much as possible now.
I got the email about this and realized I’ve been using them for just shy of 20 years. It was one of the original places I hosted things and where I hosted projects for other people. I haven’t actively used it years but migrating the last few things off has never been a priority. This might be the push I need to re-evaluate everything hosted there and shut down or migrate it all.
Yeah same here! I dont pay them any money but I still have an account there. For awhile I let my domains sit there until I realized that it was cheaper to move them to Cloudflare
Dreamhost - gosh that brings back memories…
000webshost anyone? Also .tk domains.
Hey, I used both of those! Those were fun days. When DotTK moved to "Freenom" though, that's when it really started to go to shit.
Super displeased with DreamHost's handling of this. I have multiple master accounts. One of them got the email around 12pm today (over 48 hours after the incident started) saying...
"As a precaution, we have expired the passwords for some email and FTP/SFTP users on your DreamHost account. Their old passwords no longer work, and no new password has been set."
But none of those email passwords were actually expired - all were working. This is confusing - did they only expire passwords that were under threat? If a password wasn't expired, should it be trusted?
The same email suggests you change all email passwords you control.
Another account received no email warning from DreamHost so I assumed it was fine, but 7 hours later it also received the same generic email.
My complaints:
- sucks that it took them 48 hours to identify the breach - perhaps understandable
- but once the breach was identified, how the fuck does it take them over 7 rolling hours to email their customers? That's terrible.
- confusing messaging: DreamHost are claiming that passwords are expired, but in testing multiple accounts, all of them are still working. Do I need to rotate these? (I probably will just for safety).
I'm a 25+ year DreamHost customer and generally satisfied, but this feels like amateur hour.
For emails over 7 rolling hours, unless you're e.g. Google or Microsoft, you kinda have to, otherwise those systems will automatically shadowban (send-to-spam) you for excessive/abnormal email sending rates. It's not great.
It also doesn't help that a lot of scammers use this, or fake cyber incidents, for phishing; so it's not like they have an easy solution for exempting "security emails" either.
Sure, but it also took them 7 hours to email their own customers on the email infrastructure run by DreamHost themselves. Since they run the IMAP and SMTP servers, seems like they could have done that in seven seconds.
Of course, if they are nuking everyone's email password that doesn't help for notification purposes -- but as of this morning, all my email logins are still working fine, so it seems they didn't actually do that? Confusing.
Yeah. And if it's not all passwords under the account, the email should've had a list of the affected ones. Instead Dreamhost says both that they've already deleted the at risk passwords, and also that customers need to reset all passwords. Maybe when they mean "reset all" they meant "reset all [the deleted] passwords", but being so unclear about scope and remediation steps is a painful mistake.
A single sentence like, "If a password wasn't reset as of (timestamp), we don't believe it could have been at risk." would go a long way to fixing these incident comms, if that's the case.
Agree. And this is all made worse by the fact that there seems to be a bug in iCloud keychain syncing - I can change the password on my mac, verify it works, but when I go over to iPhone or iPad, the old password is still being used. Not sure if there's a way to force the sync process on both ends?
I migrated all my stuff from dreamhost about 5 years ago, got this email today and felt validated from having made that decision.
Dreamhost is no longer the hosting company that I originally signed up for. I'm also looking for a different host.
I might have been willing to stick it out, but their panel/website-management is so slow.
I love their shared hosting; I have a $100/year account with dozens of little low traffic apps in their own subdomain/user isolation. They maintain the OS, I maintain the app, I can pull code with git and even compile a python interpreter if I need.
I love that I have infinite/unmetered email accounts. In practice I run 3-4 accounts on 2-4 domains, but they don't nickel and dime you on it, I've literally setup 1200 forwarders in the past. They have a good sending reputation.
What other provider does this as well?
have you tried digital ocean? some friends I have swear by them as good
Yes, they're in my rotation as well (I never want to have just one provider.) But I liked that DH would maintain the OS, the shared hosting performance never bit me (my sites are all highly performant, no wordpress dogs) so all of my time would be in patching vulns. DH giving me no-root access to a hardened, performant box is ideal all around.
> I migrated all my stuff from dreamhost about 5 years ago
May I ask to where you migrated?
I re-architected most of what I had to fit in a serverless deployment model on AWS and now they run well within the free-tier in AWS + Neon DB
What's going on with Dreamhost lately? Is it falling apart?
I havent seen anyone talk about it for the last 10-15 yrs. I would have guessed it fell apart already
They abandoned all support for email lists a couple of months back. Will likely move after 20+ years on DH.
Chat support works well. Email is nonexistent.
I don't mean "support for using mailing lists". I meant "they no longer provide the services (i.e. mailman) to run mailing lists"
Nobody wants to run a service like that anymore.
i wouldn't be so sure. I know I move in privacy and "anti big tech" circles, but I (and a lot of people I know) am moving a lot of my personal/recreational internet time to rss feeds of personal websites, mailing lists, and Mastodon. And there's a fair bit of discussion and small but growing use of private and very small group federated Mastodon hosts.
Having said that, very little of that is being done by the sort of people who'd host it on a service like Dreamhost.
Mailing lists are inherently dangerous as you can be marked as a spammer.
Everyone (and I mean everyone) is a spammer by default these days, so this is no problem in practice.
No, they’re still doing well.
Support is primarily via chat these days.
Pricing wise they don’t beat Herzner anymore but are somewhat competitive with Digital Ocean.
I am gradually moving everyone to my own self hosting mostly for supportability reasons. We want to react in seconds, not hours, when a customer has an issue.
I've been using Contabo, and so far, the service has been quite stable and professional. Updates are handled automatically, and I haven't encountered any major issues DigitalOcean seems a bit slower and more expensive in comparison, especially when Contabo offers a VPS with 4 CPU and 8 GB of RAM for just 4.50 euro/month.
I'm still at DH (have been for 20 years), but taking a peek at Hetzner... seems compelling at 30% cheaper than I pay now.
Just be warned Hetzner will delete your account at the slightest sign of controversy. It's fine for Minecraft but I wouldn't want to host a political website there.
I used them for entirely too long, and only continued to do so because a classmate in high school used my referral code to sign up for the account that would become his career of selling and managing websites for local churches.
I could not believe what the hell they were charging for a year of hosting with a dedicated ip (which they at the time required if you wanted https, real scumfuck move), so I didn't pay.
I guess I hope they actually deleted my shit because I sure didn't get this email
How long ago was that because at the beginning you did have to dedicate an IP for https prior to SNI
Ah, I noticed this yesterday, figured it was just me.
Now that's a blast to the past. What's next, Rapidshare.de is having issues?
Remember when they put that 25GB daily limit vs MU unlimited, that was the start of its death.
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
I've been a user for more than 20 years. With their shared hosting plan, which was about $5/mo. when I joined and is now $13/mo, I've been able to host my personal blog and professional site, as well as probably 3 dozen other sites over the years, including several that have hit the HN front page and never got hugged by the squeeze of death.
Honestly, compared to some of the professional environments I've worked in, it's a joy to have an idea, create a few PHP files, and in a matter of seconds have that idea live.
A shitload of small business sites that were built years ago and haven't been touched since.
If they are static sites that shouldn't be a problem. Many are probably not though.
My suspicion is at least half are unpatched WordPress installs.
It's cheap and reliable with predictable costs. It hasn't caused me any problems in a decade. Most of the websites I want to host don't need cloud anything, they're statically generated.