Security Disclosure
blog.railway.com361 points by passenger 6 hours ago
361 points by passenger 6 hours ago
It's 5:20 AM here in Europe and I just rotated a bunch of keys.
Only to read,
> Because deployments are paused, you will not be able to redeploy your services to pick up the new credentials until builds are restored. We will post here when builds resume with any additional details.
So now my service does not work. Which is worse: a stolen key or a broken product? I'm genuinely regretting following their advice on rotating keys, because logs showed no sign of incorrect usage, and our service would at least have remained online.
I hope that they automatically redeploy their customer services. I just worry that I won't have tested it. I mean: 5AM. Keys changed. Nothing more I can do. Should go back to sleep. No timeline to wake up to check deployments, so... I... wait?
The customers of my company are angry. It is 9.45 AM here in SE Asia. Things are bad.
Yeah, I had to rotate my data store keys ASAP. Now I'm stuck with a broken application. Best option, horrible option.
Same situation. Frankly unbelievable - how do you not lead with this? You can't tell customers to rotate their keys and not inform them in the same breath them that actually, environment variable updates currently cannot deploy.
They're actively firefighting an incident, and are likely going to make some mistakes along the way. In this case, they've erred on the side of security over availability, which in some cases is the better outcome. I know not in all, but some.
They did.
No. They did not. From https://blog.railway.com/p/security-disclosure-wednesday-oct... :
> What you should do: As such, we strongly recommend that you immediately roll all environment variables, including any API keys, stored in Railway, as well as any private registry credentials, and review logs for those services for unusual activity.
> All deployments are paused while we investigate. Until builds are restored, you will not be able to roll out new versions. We apologize for this inconvenience.
They lead with rotating keys.
Next, I did not realize that this meant not being able to restart an existing deployment to bring up new env vars. They also speak of builds. I don’t need a new build. I need my existing deployment to work.
It was not at all clear ‘paused’ mean this; maybe I should have realized, but I had interpreted it as doing new deployments or builds. New services. Code changes. New builds as they even say. It’s fully reasonable to pause that in this sort of situation and so I had interpreted their statement that way.
If there is a clear impact from following advice — ‘do what we recommend and your service will not function’ — those who provide that advice need to make that clear.
Their first message where they recommended rotating keys did not make that clear (though obviously if you cannot deploy, rotating keys means you will go down)
Crazy day for cybersecurity: PHP RCE, eMudhra certificates, and now Railway.
And it’s all just getting started imo.
I think the pingcap hack might be the biggest of them all, somehow it has gone under the radar.
I guess I picked the wrong week to stop sniffing glue.
Related ongoing thread:
PingCap TiDB Leaks - https://news.ycombinator.com/item?id=50173716 - Oct 2026 (32 comments)
(It had only spent 3 hours on the frontpage so I re-upped it.)
I find it wild how much they have downplayed this in their statement. https://www.pingcap.com/trust-hub/security/security-update-o...
”An external incident response firm reviewed PingCAP’s production systems and found no evidence that the attacker ran commands or maintained persistent access in the systems reviewed”
hmm, I don't think that's true.
"Found no evidence" doesn't mean it didn't happen, it means the incident response firm didn't find it.
The evidence is in the blog post explaining how they did it. However seeing at how bad their operations were I doubt they have evidence of anything themselves
That was a single company, the cert problem hit something like 70% of all websites and devices.
Jesus that one is bad
> We even stood up a replication feed of their own control-plane database using credentials committed in their own repositories — GCP service-account keys and an Azure service principal — and let their infrastructure deliver it to us. Every six hours, for the last twenty-two days of the campaign, from 8 September to 30 September, that pipeline faithfully re-exported every table of the operational heart of TiDB Cloud — tenants, users, API keys, cluster credentials, changefeeds, migrations — and pushed it out. Eighty-eight consecutive exports, not one detection event.
Also Shopify, they have been delisting a lot of apps without giving a reason, most lickely a supply chain attacked that spoofed/stol checkout data
Postiz founder posted on X that his service is being brought down, his keys were already used and Railway was quiet about this until he spoke up:
Why is a 300k mrr company (his tiwtter bio) running on raleway?
You may think that sounds like a very nice salary but $4MM annual gross is a very small business. I can’t imagine how Railway would be in business if that were considered a large customer.
Sure, I guess there are a ton of dumb people. But please name a few sites that aren't slop ecom fly by nights... I cant think of any serious business that runs on raleway.
I dont understand using any PaaS that essentially functions by arbitraging aws to people that are scared of an aws dashboard at a 3x markup.
Seems like a decent platform for toy projects not for enterprise.
Maybe I'm finally old, but I don't see how it's surprising that Raleway is at all successful. Heroku built the same business ~15 years ago and had plenty of "enterprise" clients. I don't really understand if there's that much more value in Raleway vs Vercel, and Vercel claims to have OpenAI and Meta as clients.
> But please name a few sites that aren't slop ecom fly by nights.
tripadvisor and bilt, maybe?
It's pretty common to have "toy" infrastructure in a non-tech business
300K ARR means you get 300K in annual recurring revenue.
Subtract costs for employees. Which, given current economy and inflation, that gets you a founder scraping on their savings, one or two other people (remember that if you're paying someone unless it's totally offshored through a house which has it's own inflation, you have other factors) and then your own taxes... yeah... much more than that and you are in the black.
As a general factor I keep in mind, when I worked at a 'normal' but toxic place before I was an actual software engineer, my manager's rules were something like '40 percent L2R' (might have been lower) for in-house employees and 70- 80% L2R for contractors.
IOW, for every dollar billed for an actual employee, we paid the employee 40 cents. For every dollar billed for a contractor, we paid them 70-80 cents.
I've heard similar figures from others (e.x. 'in house employee has a real cost of 1.5-2x their actual pay').
And yes it sucks and yes the structure at least here in the US is a factor.
If your overhead for an employee is 60% you need to seriously rethink some things
Yup, got my OpenRouter key. They managed to run up a whole $2 on Opus 5.5 and Fable. Well, time for a new hosting provider I guess...
Incidentally, I'm testing Muse as an email triage agent today - it buried "oh also, railway and anthropic say yours keys have been leaked and you need to rotate asap" under 50 routine deploy / admin emails notifications. Oh to live through such an AI powered software renaissance!
Right just as I take my first day off (for the entire year!) from work to fly out for a vacation. The mother of all security bombs starts dropping all around stacks and services that I am responsible for. How else could have taken time off go for me…
Brother do not cancel anything for these people, they do not care about you, it's their fault for not considering bus factor in their org chart. Why can't the person above you handle that?
Prioritize your mental health, go enjoy your vacation.
Thanks, yes I will. Left without the laptop(s). Let the chips fall where they lay. If they need me back so urgently, they will have to go through the full cancellation of booked flights and hotels first. Lets enjoy the time off.
I hope you don't check in on hackernews after this post here!
What if they do. Sec analyst terminated for being on holiday posting on a forum being on vacation without laptop, not cancelling his trip ?
I didn't read it as that, more as, "You might hear about other similar work-related bullshit here while you're trying to chill and not think about it".
Not your problem. Enjoy your vacation. Bus factor of 1 is poor planning of leadership.
Either a) your company will reimburse you for your costs incurred and give you a different week off, or b) they don’t need you there that badly.
I say this as a Railway customer, go take your vacation. The only time you should even reconsider is a massive bonus plus more time off for staying.
Correct me if I am wrong - if the API keys are compromised, then a large bill could be racked up?
Oh you're right, that does sound like a good reason for the original commenter to not take his vacation. Lucky the organisation he works for made him the single point of failure in circumstances such as these.
I had assumed the original commenter OWNED the company. Yes, I agree with you that if he is an employee then take the time off.
Ah yep, that would be a different story if that's the case (from reading his original comment I still think he is an employee though)
Seen multiple folks in the Railway Discord confirming that their secrets stored in Railway have been used by third parties, so it definitely sounds like the attackers were able to read decrypted environment variables.
Environment variables are a terrible place to store secrets.
I'm just going to say it, since having said it non-stop for over 20 years, nobody seems to listen and we keep circling this drain.
The 12 Factor manifesto was 100% wrong on this point.
Use environment variables to point to where secrets are located, not to store their values.
Or we could just use our operating systems for the things we do well, and have a service to send our API calls to that inserts the appropriate token.
I don't have to enter my Mac's password every time I open an app.
> Use environment variables to point to where secrets are located, not to store their values.
Now your apps need to be able to read the secret that is pointed to, as opposed to the task runner populating the secret and the app would not know about the details.Not to mention the changes it entails to development.
> Not to mention the changes it entails to development.
Oh no, guess they'll have to hire developers that can adapt!
It doesn't sound like they got them via a shell environment, they just hacked into the place that stored them. If the place you store secrets can be hacked into, it doesn't really matter if they are injected into environment variables or not.
How does this help? If your container gets compromised it's trivial to exfiltrate the secrets.
Environment variables are generally fine for secrets. Environment files are problematic. Modern systems that inject secrets into the environment launch the program itself.
(pre-caveat thing to not be massively annoyingly pedantic throughout, I'm not a security buff and I know who it is I'm talking to, I'm not trying to correct them)
I'm not sure I get the difference - the issue I would have thought was having the system code have access to the secrets themselves? Env file vs env vars injected at runtime seems the same there - either can be exfiltrated. The difference would be having it injected when used.
Is the problem with files about what happens outside that, being committed, etc?
Everything can be exfiltrated. If you never store the environment in a durable file, it's not meaningfully easier to exfiltrate from the environment than from process memory. Most of the wisdom about the insecurity of environment variables falls out of people setting them in .profile or in .envrc files --- that's very bad, yes, as are configuration files (the thing 12 Factor Apps was responding to).
The headline here is that there's no reason to believe "environment variables" were the reason this data was breached; presumably, the database those variables came from was what actually got hit.
You can say "everything can be exfiltrated" but capturing my PC's password is about nigh unto impossible. The only way it can be captured is if a program has active control over the keyboard controller/associated drivers when I type in the password to unlock it.
These problems were all solved a long time ago.
I have a bad gut feeling that every day is going to look a bit like today with endless urgent patches and breaches.
Fwiw it sounds similar to the hack over the weekend that opencode suffered.
The security team at my company is working with a frontier lab to scan all our code/systems. They are in a borderline panic to close every possible issue, even innocuous ones that are buried under defense in depth. Whatever they are seeing from our partner has them spooked -- as in "acceptable risk" is no longer a thing anymore.
I fully believe the only sustainable future is that hot patching will need to itself become fully automated. Hot patch now with a minimal allowed amount of line changes to head off the AIs, review it in office hours for auditing and the team is actually awake.
My second belief is that environment variables as a development concept will need to be retired. If it's something that causes a security breach if it by its mere existence on a public git repo, it is information that by design cannot ever pass to a filesystem out of a chain of trust, not by mere development conventions people choose to adhere to.
> hot patching will need to itself become fully automated.
Great, even less control over the things we use and more power ceded to corporations.
Due to the small nature of hot patching (eg. bump this dependency, rotate these keys, turn off that config, follow the runbook) it will unlikely need to be a frontier model and is likely able to be done LocalLLM/Jev style decision making rather than having to hand all the keys to frontier model operators.
One more step to just turning it all over to Skynet. It's hilarious how prophetic that movie was.
Hot patching is itself a liability
Not reacting is a liability. Blue-Green deployment is a liability. Code is itself a liability. Choosing this career at all is a liability.
For the record, there was a bunch of non-functional vibe coded fixes flying around during this PHP bug. If you were just running the first thing AI made because you were worried about "not reacting," you would likely end up running a crap script.
AI is quite good at finding vulnerabilities, but it's been rough for patching them. The old adage "defenders have to be right every time, attackers only have to be right once" applies to AI security patches.
Great, but the fact remains that easy hot patching which skips normal review is going to open you up to many of the same liabilities you're trying to minimize by introducing it. The target just shifts, and people will attack that infrastructure.
The actual solution largely involves the same technologies but in reverse: leveraging AI for live security monitoring, better correctness validation and security testing before code goes live, etc. And of course more pressure to use battle-tested widely-adopted libraries and systems, even though of course that comes with its own tradeoffs.
Importantly, as an engineer, when someone tells you something has liabilities or tradeoffs, the response is not "Everything is a liability, existing is a liability"; that is unprofessional. The appropriate response is to engage with them on these tradeoffs and working through the best ones, because that's your job. Otherwise defer to security engineers who are ready to back up their arguments. It could be as simple as following up with, "But do you think those liabilities outweigh the benefits of hot patching without proper review? What are some of the main liabilities you see, and what methods/technologies do you think properly address them?"
With AI, in the long term open source code might actually become more secure than obfuscated closed source, since it's had so many more (AI) eyes looking at it.
The internet is over. We are going back to static htmls for public pages, and Point to Point VPN bases connections for B2B service provision.
B2C? too risky to publish services online haha.
>We are going back to static htmls for public pages...
God yes, please. I backpack and mountaineer in the PNW and have been trying to research trip ideas up in British Columbia, and as a result I discovered https://bivouac.com.
Here's a website that's still frequently used by a large number of people as a valuable resource, is regularly updated and maintained, and yet looks like it's straight out of 1995.
Every pages loads instantaneously, everything is simple yet packed with detail, and it's such a basic, straightforward, delightfully pleasant experience to just browse through the site. I was so thrilled after just 5 minutes on the site that I threw them $25 for a year's full access.
Sorry for the tangent, but oh man, it's such a refreshingly stark contrast to the rest of the bullshit that fills the web these days.
Still some industries where you cable colocated servers in a shared DC together where neither server has internet access.
I think the only way out is to go back to on-prem for everything, like in the 00s.
LAN cables will go up in price soon.
Ha, Cyberpunk 2077 is becoming reality. We're proposing the Blackwall.
That was always going to happen. The future of the internet is AIs scamming you and your grandmother, ransomwaring you, draining your back accounts, etc. There will be no reason or rhyme, it's simply a function of available compute and ease of automation, both of which have reached a critical threshold.
I think we'll see a reduction in complexity and an increase in transparency, rather than just on-prem.
One medium-sized server can now easily serve the entire banking system of a large country. So you don't need clusters of servers with careful tradeoffs between consistency and availability because of the CAP theorem.
Just get a big honking host, slap Postres on it, and run your application server in a VM on the same host. For backup, export the WAL logs from Postgres onto your favorite off-site blob storage. UI for it? There's Pigsty.
It still makes sense to host this in a cloud provider, just because you probably don't want to deal with actual hardware. And maybe get a second standby server for redundancy.
That's it.
You'll need a bunch of external dependencies for email, 2FA, observability, LLMs, and so on. But they should all be outgoing dependencies that should not allow access back into your system.
Real loads are too spiky for that.
A PostgreSQL server on an NVMe can handle several _million_ simple requests per second. This is bonkers.
Do you think you can hit these load levels? For compute, a 64-core CPU can saturate a 10G network with simple template rendering. If the code is written in something reasonably fast (Go, Rust, Java).
Of course, if you're on a scale of Amazon or Stripe, you'll need more than that.
How does on-prem help if your product is a SaaS.
You put the SaaS into a Box your customers can install into their datacenter!
What a brave new concept!
Sometimes I do miss owning the hardware to run a thing.
What I don't miss was having all of those appliances run a different base distro of Linux, each with varying levels of update support, while the security team keeps reading security scans to me.
Train wreck. People were writing on discord 3 hours ago, 10 hours after the incident started, that keys created just for railway were running up usage. Meanwhile builds have been down for more than 12 hours. Only NOW did they just send out an email to all customers. I wouldn't have found out until then if I wasn't checking the discord periodically.
Hi Angelo from Railway here. This is very much me. https://x.com/ngeloxyz/status/2113077263386239309?s=20
We have gotten a lot of comments about why didn't we just message the breach sooner when people have noticed compromise with their accounts. I think that crit is fair and justified, and although we have been up for days on end- I don't blame you if you don't have sympathy for us.
Our thought process was that we didn't want to recommend that people roll keys when we couldn't possibly confirm that we confidently plugged the breach and rolled the mitigation(s). I am comfortable to publicly say that this was a highly motivated attacker. Meanwhile, our threat team mitigates DDoSes, CVEs, while been patching everything immediately as dealt with exploits.
As for this breach in particular, we tried to balance a duty to warn, which we take very seriously... However, it took a significant amount of time on the order of hours to verify, reviewing, and keep up with the attacker as they attempted to move laterally around, and briefly, within the system. We didn't want message anything beyond a shadow of a doubt that there were no further exploit paths.
We take the view that we would like to "aviate, navigate, then communicate" when we deal with incidents like these then we prepared communications to the public and all users. While it was ongoing, we were in contact with external partners and vendors to revoke keys (the source of some online chatter) with investigators that required time and coordination.
On the incident side of the house, it's a suite of bad options, we're still in the middle of it making sure customers are safe. But, hopefully this offers a view into our thought process.
Formal postmortem after we fully restore service and complete the investigation. Apologies to all, I personally I feel like I let everyone down.
> and although we have been up for days on end- I don't blame you if you don't have sympathy for us.
From my timezone, this started last night and is ongoing into today. Has this been going for longer than the status page and email indicated? If so, why was notification delayed so long?
I love your product, have an incredible amount of sympathy for you as humans (having been in similar sleep-consuming nightmares) and was willing to forgive the GCP-triggered outage as an understandable teething pain, but sadly this response has really knocked my confidence.
I understand the hard reality of security issues -- "urgently rotate your keys, they're already compromised" is an important message to share -- but "sorry, that will also require your site going down because we've blocked deploys" (presumably as a response to the incident...) is an incredibly frustrating situation to find ourselves in.
You know... I get it.
Railway / OpenAI: I had a key compromised this morning. It hit our limit, so I revoked the key... and figured, well, we'll have to track down the leak.
Then this security notice came out, telling all of their customers to rotate all their keys... As the guy whose spent the entire evening rotating keys and etc. doing
this is... it's a huge message to send.
The cost and implications go really far.
We have many production services that we've taken offline or disabled because we can't deploy to fix the credentials. because of the railway deploy problem...
Railway was stuck between some pretty nasty tradeoffs. I get that the few hour delay in communication caused some of us more cost, issues, and potential breach... and maybe larger suff.
But I also realize that rolling the keys while things may still be compromised, with deployments blocked, could make thins even more sticky
leaves only very bad options. Also we don't know how much they we're spending time actively stopping the attack. the time and focus could be very much consumed. if it was 5% of exposure and they prevented the 95% exposure.. i'm sure glad they aviated instead of communicated.
We make the best decision we can in these moments.. and they all just suck."aviate, navigate, then communicate" only matters if you are the one in control of outcomes.
Customers could have simply revoked keys and not re-launched services with rolled credentials (that from your post, may have been re-stolen), but instead you let this fester.
What a disaster for Railway. I hope you get through this stressful moment, but this should be the end of Railway.
IMO "aviate, navigate, then communicate" does not work in this situation. There is more than one pilot. The moment you detected a breach, hell, even knowing that an attacker was inside your system, you should have pointed to anyone. Hell, even AI, to draft a disclosure with more info coming.
Anyone can help?
I changed he api key in the env var as openrouter revoke the previous one based on this incident. However, as the redeploy cannot be done in my region, the env cannot be updated and now the product cannot be used. Any command in railway to fix this?
How is it possible rotate keys if you disable deployments? You're basically telling us to take our services offline...
What about buckets? I tried to regenerate my secrets but deploys are paused. I don't know if they actually rotated or not because the old keys seem to still be working! Dear attackers, please don't read or mutate my files while my app is offline :)
I've already taken my applications offline and will be migrating to a new host...
In general, the new host will have its own problems, otherwise you would've used them to begin with.
Yep. What an incredibly frustrating notice. I like Railway and I guess their hands are tied here, but wow.
I rotated mine. If you just change the keys and hit deploy it works
It appears to work in the UI, but go test your product. Mine didn't receive the new keys.
Saw someone on Twitter indicating that they did this, and new keys got compromised.
Jeez, what a rough past 24 hours for all security folk. Hang in there
It's going to be a rough decade.
Yeah, the future of our jobs in this industry... for the type of work we do. are concerning. Explosure has whole new meaning
I really don't know how we are going to deal with accelerated exploit generation via LLMs — it blows up the disclosure processes we've evolved.
The only thing I can think of is hunkering down with static HTML to reduce attack surface, but that's laughably incompatible with many many business requirements.
In new software I write these days I limit the number of external dependencies and mostly vibe code anything I need besides the most critical foundational libraries (increasingly I am replacing these as well).
I think this, the supply chain attacks and binary reverse engineering will cause us to go back to thin clients, html over the wire and vibe coded libs instead of external open source deps.
Vibe coding deps is security through obscurity, with all of its weaknesses (and strengths). It's not that such deps have fewer security bugs than the open source reference implementations from which they are derived, it's just that those bugs are currently less discoverable.
In the near to medium term, the current open source regime has issues because bugs have become extremely discoverable and easy to exploit. But I am skeptical that obscurity will provide vibe coded deps sufficient protection over the long term. As soon as the source code for those vibe-coded deps leaks, you are screwed.
It's not just obscurity, it's increasingly difficult to trust the governance and quality of code of any outside library. A lot of deps in code are unnecessary you use maybe 1-10% of the api surface. But since building and maintaining software can be cheaper now it can be better to maintain a 1-5k LOC vibe coded version that meets your security bar than to trust an external party's governance & quality. This has the additional benefit of reconsidering if you even need the dep or if you can reuse something you already have (since you may not want to maintain the lib).
> it's increasingly difficult to trust the governance and quality of code of any outside library
The value of 'code written before 2024' is going to be substantially higher than of code written afterwards.
Sort of like a 'low background steel', a comparison someone made on HN before which I think was pretty good.
Limiting the number of dependencies is always good, within reason. Especially with eco-systems that want to pull in everything and the kitchen sink. That's just asking for a supply chain attack. I try to limit myself to what well reputed distros provide and what I can review. That's still no guarantee but it cuts down the risk substantially.
I think it will kill open source to some degree. I found it already very hard to imagine how the endless queues of issues were being dealt with so diligently, but at least there was a human on the other side. The torrent that is unleashed now is not manageable on a human scale. It's a highly automated arms race with a bunch of good-willing humans caught in the crossfire between large companies and criminals duking it out.
Could open source activity transition towards provably correct software, with far more modest feature sets?
I don't have experience with proof based tools, so I'm not sure what's feasible.
A lot of people in the open source world actually enjoy the act of programming. Not all of them will enjoy prompting to the same degree, but to keep pace with the bad guys you will have to either join or find something else to do. That's going to be a big loss.
It's funny, that's true although I personally can't relate — I enjoy having built something robust, but the act of building isn't something I love.
I can imagine prompting an LLM to build provably correct software. Aren't we already doing something similar already when we remind it to write unit tests and account for all edge cases?
Actually writing provably correct software from scratch sounds terribly tedious, which is why I've never explored it. But maybe an LLM could make that task more approachable.
I switch between the two modes. I love algorithmic stuff and I really don't like working on boilerplate and plumbing. So I use AI for the second and read every line and all the other stuff I roll by hand. The speedup isn't super like that but the amount of control is pretty good.
I just gave this prompt to duck.ai (5.6 Terra under the hood): "How would one go about writing a provably correct algorithm to count UTF-8 code points?"
It gave me something which looked pretty legit. Of course, the generated output very likely to be subtly wrong, but as starting input to a formal checked specification/proof process it seemed like a good start.
I wrote some UTF-8 based string handling software back in the day. It doesn't seem unreasonable to write up a set of requirements and edge cases and then have a formal mechanism prove correctness for common UTF-8 string manipulation operations.
[PEAK HN ON] Now if we could just write provably correct exception handling, we could solve one of the two PHP vulns from today. [PEAK HN OFF]