Rubygems.org AWS Root Access Event – September 2025
rubycentral.org278 points by ilikepi 4 days ago
278 points by ilikepi 4 days ago
AWS account root access on a language package registry for 11 days. Not EC2 root - AWS account root. Complete control over IAM, S3, CloudTrail, every-damn-thing.
They're claiming "no evidence of compromise" based on CloudTrail logs that AWS root could have deleted or modified. They even admit they "Enabled AWS CloudTrail" after regaining control - meaning CloudTrail wasn't running during the compromise window.
You cannot verify supply chain integrity from logs on a system where root was compromised, and you definitely can't verify it when the logs didn't exist (they enabled them during remediation?).
So basically, somebody correct me here if I'm wrong but ... Every gem published Sept 19-30 is suspect. Production Ruby applications running code from that window have no way to verify it wasn't backdoored. The correct response is to freeze publishing, rebuild from scratch (including re-publishing any packages published at the time? Ugh I don't even know how to do this! ) , and verify against offline backups. Instead they rotated passwords and called it done.
Isn't the subtext of this post pretty clearly that the unauthorized actor was Andre Arko, who had until days prior all the same access to RubyGems.org already?
The impression I have reading this is that they're going out of their way to make it clear they believe it was him, but aren't naming him because doing so would be accusing him of a criminal act.
Let's say that they are 100% correct, we parse the subtext as text, it was totally him.
We still do not know the critical details of how (and when) he stored the root password he copied out of their password manager (encrypted in his own password manager? on his pwned laptop? in dropbox? we'll never know!) therefore the whole chain of custody is still broken.
The leading contender to replace RubyGems has Andre Arko as a charter member, so this all seems very salient.
Right but that speaks more to Andre's character, IMO.
Why are you copying a password out of a shared vault that should only be used in break-glass type scenarios? It's that's not planning for possible malicious action in the future, I don't know what is.
You can try and excuse it as having your own break-glass for the break-glass, but that's on the spectrum between irresponsible and incompetent.
Again, if the accusation is true, removing him was justifiable from any possible perspective you might have.
The other subtext is that they literally have no idea how to run rubygems securely... And what to do in case of a security incident...
I'm addressing the question of whether we all had better assume all the RubyGems published after this incident were compromised, and my response is "that is probably not rational since the actor in this scenario had all this access legitimately just days beforehand". The rest, I don't care.
Look, it's enough to know that Rubygems did not require 2FA before August 2022. There were gems with millions of downloads with owners without 2FA on their accounts. I think your initial assumption is pretty safe even without the ongoing fiasco.
It seems to me like the inherent trust in open source software is a big problem. Reliance on software maintained by strangers, sometimes just one individual, and not reading/understanding the code before running it.
The other other subtext is that this sure is an effective distraction from their governance problems, and muddies the waters. Given the utter lack of trust I have for anything the Ruby Central folks say at this point, given the amount of spin and misinformation they've spread already, my default assumption is that this is an excuse to malign someone who may well have had legitimate access, in the process of claiming that you're locking things down, which was always the excuse being made for kicking people out.
Update: https://andre.arko.net/2025/10/09/the-rubygems-security-inci... is pretty much exactly the kind of thing I expected here. Person with legitimate access doing their job, organization flailing around in the process of kicking people out that should never have been kicked out in the first place.
He changed the AWS root account password; RC implies they had to go through a reset flow to recover the account. This apparently went on for more than a week. I don't know how to reconcile what Arko is claiming with what RC is claiming.
Arko believed he was in the right to do so, and while he probably should've reached out sooner to notify them of the "precaution" he was taking, the fact that they didn't notice for almost two weeks shows how unserious they are about security
At this point, it looks like everyone involved, not just RubyCentral, contributed to the governance problems over many years https://archive.md/SEzoV
> Regarding Arko’s blog post about his removal, McQuaid [Homebrew Maintainer] told me it’s good that Arko is crediting other people for their contribution and that he’s following open source principles of community and transparency, but that “his ‘transparency’ here has been selective to things that benefit him/his narrative, he seems unwilling or unable to admit that he failed as a leader in being unwilling or unable to introduce a formal governance process long before this all went down or appoint a meaningful successor and step down amicably.”
CloudTrail logs for the last 90 days are enabled by default, cannot be turned off, and are immutable, even by root. If you view this “event” as starting when Arko was supposed to have their access terminated, that’s within the 90 day window and you can indeed trust the logs from that period.
CloudTrail's 90-day immutable Event History only logs management events (IAM changes, instance launches, bucket creation). It does NOT log:
* S3 object reads/writes (GetObject, PutObject) - these are "data events" requiring explicit configuration[0]
* SSH/RDP to EC2 instances - CloudTrail only captures AWS API calls, not OS-level activity[1]
With root access for 11 days, someone could modify gem files in S3, backdoor packages, SSH into build servers - none of it would appear in the logs they reviewed. Correct?
[0] https://docs.aws.amazon.com/awscloudtrail/latest/userguide/l...
[1] https://repost.aws/questions/QUVsPRWwclS0KbWOYXvSla3w/cloud-...
SSH is totally irrelevant here. Having AWS root account access doesn’t give you any ability to SSH to or otherwise access running instances. You could access data on those instances by cloning the EBS volumes or modifying build pipelines or changing network access or similar, but these would all show up in CloudTrail even without data events enabled.
For S3 objects, you don’t necessarily need data events to identify if tampering happened. S3 objects are immutable as well, so if any changed you would see that reflected in the creation date and new hashes that S3 attaches as tags, which you can correlate with application logs to see if they match up or not. It’s not as simple as data logging, sure.
But you’re also missing the key component here that they did not say they only just enabled CloudTrail logs, they’re saying they just now enabled CloudTrail log alerting. We don’t have any idea if data events were enabled or not, or if things like flow logs were enabled or not, or what other investigation tools they have running at the application layer. However, even if none of existed, there’s still a lot more audit-ability of events that happen in an AWS account than you’re implying, even the root account.
I’m not sure this is true. The EC2 web console terminal drops me right into root on any of my instances.
Ahh you’re right, there are some that just initiate a connection via something like Session Manager, but those connections where AWS initiates the connection for you are logged in CloudTrail, even without data events, and root doesn’t give you any ability to directly SSH into an instance outside of those methods (you cannot, for example, use root to find out what the private keys are for logging into an instance) so we’re back to the fact that any such access would be auditable.
> Having AWS root account access doesn’t give you any ability to SSH to or otherwise access running instances.
It does if access credentials to those instances are stored in any AWS service in the account (such as in Secrets Manager).
Even without data events, there are clues you could infer from management events that somebody is going through buckets. ListBuckets and GetBucketLocation are management events and are seen often by someone enumerating buckets to exfil data from. The fact that this actor was someone very familiar with the account might make this a moot point though as they would know exactly which bucket to go to and what region its in.
Thinking about this a bit more... it sure is interesting that around the time of a competing project launch that something just happens which might reasonably completely compromise trust in the previous incumbent, isn't it? Odd!
That was intentional according the Joel Drapper who leaked this incident, he wanted to make Ruby Central look bad
https://www.reddit.com/r/ruby/comments/1o2bxol/comment/ninly...
>> Why did Joel give so little time of advance notice before publishing his post revealing Andre’s production access? That struck me as irresponsible disclosure, but I may have missed something.
> I decided to publish when I did because I knew that Ruby Central had been informed and I wanted the world to be informed about how sloppy Ruby Central were with security, despite their security posturing as an excuse to take over open source projects.
> What I revealed changed nothing about Ruby Central’s security, since André had access whether I revealed that he did or not. When you have security information that impacts lots of people, you publish it so they can take precautions. That is responsible disclosure.
how can you trust gem.coop isn't already mining request logs + IPs to try and monetize lists of companies using specific packages & versions — besides the privacy/ethical concerns it is super useful data for hackers looking for vulnerable apps
no single person should have Github owner + AWS root password for a major language's package manager and ecosystem just sitting around on their laptop while they fly around to different conferences (as Andre seems to have done while showing off he still had the login to rubygem's AWS root account while in Japan)
Not going to bother reading the article, but will chime in here that the recommendation from AWS is to have a separate security account within your organization that only holds your CloudTrail logs. This does potentially double your cost, as you only get one CloudTrail for free, and it's very useful to have an in-account trail for debugging purposes.
Organizations are also useful because you can attach SCPs to your accounts that deny broad classes of activities even to the root user.
You are actually right, I didn't think of this before.
How can they ensure that nobody else did any tampering?
It seems RubyCentral did not think this through completely.
> It seems RubyCentral did not think this through completely.
this is the problem when you fire all the maintainers who do anything
You can set up EC2 instances in a way that that just having AWS root access doesn't give you ssh/console access to the instances. You can still do things like Run Command but that leaves a very obvious trail (although even this is preventable with enough effort).
Also you can enable cloudtrail log validation which can ensure you know if you're looking at tampered logs or not.
Really it all depends on how their accounts are set up. Unless you know the operational details you can't make a call here.
I've run a multi-million dollar/year AWS Org for the last decade or so and setting things up this way is kind of brass tacks.
So you almost certainly know that a lot of IaC tooling has terrible defaults like enable_log_file_validation being set to false. Based on the quality of their credential management and what else you can read from this blog post, how much would you wanna wager they did it right?
I assume everyone is professional until they prove otherwise.
Based on how things have been described on both sides, it actually sounds like they do a pretty good job. Oversights happen -- we're all human -- and this access was already limited to a small single-digit number of people. Given the history, it's reasonable that Arko would have had this high of a level of access and the oversight was in forgetting that when removing him.
Also it's reasonable to assume that people with that access wouldn't do something criminal/malicious, and if they did, while annoying, the situation is very easily recoverable. Especially if you're using IaC tooling as you mentioned.
If you're already taking the position that Ruby Central are "the bad guys" it's easy to assume that they're doing everything wrong, but that would be a mistake.
I believe this is a scenario where AWS recommends multiple accounts.
1. Create another "management" AWS account, and make your other AWS account a child to that.
2. Ensure no one ever logs in to the "management" account, as there shouldn't be any business purpose in doing so. For example, you should require a hardware key to log in.
3. Configure the "management" account to force children account to enable AWS Config, AWS CloudTrail, etc. Also force them to duplicate logs to the "management" account.
Step 2 is important. At the end of the day, an organization can always find a way to render their security measures useless.
2) Surely, someone needs access to the account. How do you prevent those with access from using it? Security feels like turtles all the way down where you ultimately have to trust a few people to do the right thing.
The only reason someone would need access to the management account would be maintaining child accounts and IAM roles or reviewing logs, none of which should need root.
IMO the only way to avoid doing a total rebuild is to have Andre Arko:
1. Admit that he was the unauthorized actor (which means he's probably admitting to a crime?) 2. Have him attest he didn't exfil or modify the integrity of service while committing a crime.
If I was Ruby Central I would give clemency on #1 in exchange for #2 and I think #2 helps Andre Arko.
So you would expect people to accept that the entire root chain of custody for the Ruby supply chain is attested by ... A guy saying he didn't do anything bad? I have a cool cryptocurrency you might wanna check out that I definitely don't have a backdoor to!
If Andre doing that was criminal, it seems quite possible that their original takeover of the github organization was also criminal?
I have been waiting to hear if there would be any civil action on it since it's not at all clear they had any rights to do most of what they did.
No, that's not at all clear. Ruby Central owns the AWS account for which Arko is (pretty clearly) being accused of changing the AWS root account password after having his access revoked.
I don't think for a second Arko will be charged, but there isn't a "nuh-uh, you did this gross thing in our open source community" defense for 18 USC 1030.
I didn't say it was clear, and I never said there was a defense. I implied that the wronged party in one case might want to be careful about raising the specter of liability or criminality.
Ruby Central isn't capable of giving clemency. They could refuse to testify in any prosecution, but they don't get to pick whether a relevant attorney general or district attorney decides to prosecute.
> They could refuse to testify in any prosecution
Legally, they can state a preference not to testify, but they couldn't (legally) refuse to if issued a subpoena. (And, even more emphatically, they couldn't accept a good or service from the person who might be charged in exchange for not reporting the crime, or for refusing testimony.)
In the US, at least, private parties can not grant immunity from prosecution for a crime (only public prosecutors of the jurisdiction against whose laws the crime was committed can do that), and they may face legal jeopardy in agreeing, or even offering, not to report a crime in exchange for some good or service of value, as that is the definition of blackmail.
I hate to ask; but
- Account created 14 hours ago.
- Posts article crammed full of accusations
- Has a strong well formed opinion about "it's a crime", but didn't? read the content where the subject of the accusations has.... Already disclosed they had access in both private and public.
My account is also very new, because I have opted to discard my previous ones. I have used it to comment predominantly on this topic, as I sympathise with the maintainers.
So in the interests of making a similar disclosure is there any chance you are affiliated with RubyCentral through a business relationship with them, their legal counsel, a marketing or PR agency or anything of that nature?
Given the context of the post, it seems like "Enabled AWS CloudTrail, GuardDuty, and DataDog alerting" means "enabled alerts via CloudTrail, GuardDuty, and Datadog", not "enabled Cloudtrail logging". Otherwise the comment about reviewing Cloudtrail wouldn't make sense.
So the attacker turns logging off (was log file validation enabled? usually isn't in Terraform ) which does not fire an alert because there is no alerting. Then does their bad stuff ... Then modifies the logs (which are in an S3 bucket on the compromised account, remember!) Then they turn logging on? The whole point is alerts go outside AWS. They go to like, your inbox or pagerduty or whatever. If they had no alerts then what use are their logs, which could have been modified? Do you think they set up cross-account logging or had enable_log_file_validation set to true?
You can't enable or disable AWS Cloud Trail as far I know?
You can enable the persistent storage of trails. But you can always access 90 days of events regardless of that being enabled
This was my understanding as well, but earlier I couldn't find any documentation to prove this so I never wrote a comment.
CloudTrail can be configured to save logs to S3 or CloudWatch Logs, but I think that even if you were to disable, delete, or tamper with these logs, you can still search and download unaltered logs directly from AWS using the CloudTrail Events page.
They buried the lede...
Arko wanted a copy of the HTTP Access logs from rubygems.org so his consultancy could monetize the data, after RC determined they didn't really have the budget for secondary on-call.
Then after they removed him as a maintainer he logged in and changed the AWS root password.
What a truly wild situation.
In a certain sense this post justifies why RC wanted so badly to take ownership - I mean, here you have a maintainer who clearly has a desire to sell user data to make a buck - but the way it all played out with terrible communication and rookie mistakes on revoking access undermines faith in RC's ability to secure the service going forward.
Not to mention no explanation here of who legally "owned" the rubygems repo (not just the infra) and why they thought they had the right to claim it, which is something disputed by the "other" side.
Just a mess all around, nobody comes off looking very good here!
I can give benefit of the doubt that making a proposal to monetize user data is a poorly-considered, bottom-scraping effort to find a replacement funding source for the on call work. Most of us would not consider it, but I think it should be ok to occasionally pitch some bad ideas, all else being equal and lacking full context.
But messing with the credentials crosses an ethical line that isn't excused no matter how much you disagree with the other party's actions.
I totally agree, assuming all this is accurate he immediately proved that RC was right all along to be concerned about him!
I can only assume it is silly revenge seeking behavior. Look at how symmetrical it is:
1. RC takes over GitHub Repository and locks everyone out
2. Arko takes over RubyGems server and locks everyone out.
EDIT: Missed their email notification revoking his production access. Yeah looks like they could have a legal basis.
This is not legal advice, but if you get fired and then break into the office later, the police won’t care that you used your spare key.
Seems someone took it a step further and changed the locks too.
really disappointing. it's such a huge security concern and privacy/ethical lapse, i am super disappointed in him, despite his contributions to the world of Ruby package management
he's now started a competing gem.coop package manager, and while they haven't released a privacy policy it does make me suspicious about how they were planning to fund it
no single person should have Github owner + AWS root password for a major language's package manager and ecosystem just sitting around on their laptop while they fly around to different conferences in Japan e.g. (as Andre did while hacking rubygem's AWS root account to show off)
[citation needed]
From the article:
“Following these budget adjustments, Mr. Arko’s consultancy, which had been receiving approximately $50,000 per year for providing the secondary on-call service, submitted a proposal offering to provide secondary on-call services at no cost in exchange for access to production HTTP access logs, containing IP addresses and other personally identifiable information (PII).”
I'd recommend to people to wait for a response - RubyCentral spins up a gazillion accusations right now and has been in the last days (and, it is also incomplete, because why did they fire every dev here and placed Marty Haught in charge specifically? They never were able to logically explain this; plus, why didn't they release this write-up before? It feels very strange to wait here; they could have clarified things before, but to me it seems they kind of waited and then tried to come up with some explanation that, to me, makes no real sense).
I also highly recommend to not accept RubyCentral's current strategy to post very isolated emails and insinuate that "this is the ultimate, final proof". We all know that email conversation often requires lots of emails. So doing a piecemail release really feels strange. Plus, there also were in-person meetings - why does RubyCentral not release what was discussed here? Was there a conflict of interest due to financial pressure?
Also, as was already pointed out, RubyCentral went lawyering up already - see discussions on reddit. Is this really the transparency we as users and developers want to see? This is blowing up by the day and no matter from which side you want to look at it, RubyCentral sits at the center; or, at the very least, made numerous mistakes, tries to cover past mistakes by ... making more mistakes. I think it would be better to dissolve RubyCentral. Let's start from a clean state here; let's find rules of engagement that doesn't put rich corporations atop the whole ecosystem.
Last but not least - this tactical slandering is really annoying. If they have factual evidence, they need to bring the matter to a court; if they don't, they need to stop slandering people. To my knowledge RubyCentral hasn't yet started a court case, and I have a slight suspicious that they also will not, because we, as the general public, would then demand COMPLETE transparency, including ALL of RubyCentral's members and their activities here. So my recommendation is: wait for a while, let those accused respond.
Yeah, this is incredibly confusing. The stance that Ruby Central has stated since the takeover of the RubyGems (offline) tooling on Github was that it was necessary for supply chain security, but if this happened literally within a couple of weeks of when they tried (and apparently failed?) to remove all of the previous maintainers, how does this add any amount of confidence in their ability to secure things going forward? If they can't even properly remove the people they already knew had access that they went out of their way to try to remove, it's hard to feel like consolidating their ownership over all of the tooling is going to be an improvement.
> I'd recommend to people to wait for a response
https://andre.arko.net/2025/10/09/the-rubygems-security-inci...
This makes Ruby Central look even worse. TFA is only concerned with the root user, and the timeline ends at September 30, but Arko was able to confirm as late as October 5 that he had access to _other_ accounts with production access. Ruby Central doesn't seem interested in the article to mention that even after being notified about unauthorized access they still hadn't rotated all relevant credentials almost a week later.
Welp, now that there is confirmation that lawyers are involved, the chances there will be any of sort of open and transparent reconciliation process have plummeted.
> let those accused respond.
Literally all we've heard so far is from the other side...
> If they have factual evidence, they need to bring the matter to a court
I'd be surprised if they aren't. This post feels very much like the amount of disclosure a lawyer would recommend to reassure stakeholders.
> rules of engagement that doesn't put rich corporations atop the whole ecosystem
Right now the only thing stopping us all from being held hostage by rogue maintainers is a rich corporation.
The rogue maintainers have apparently been been successful enough with their stewardship for years to the point that people still use and care about the tools they had maintained today. On the other hand, the new maintainers sponsored by the rich corporation have managed to draw scrutiny immediately about how they became the new maintainers and apparently failed to effectively protect their new assets from a major breach within two weeks of acquiring them despite security being their main argument for why they should be in charge in the first place.
This is a pretty hilarious and long-winded way to say "we have no idea how to lock someone out of a web service:"
> 1. While Ruby Central correctly removed access to shared credentials through its enterprise password manager prior to the incident, our staff did not consider the possibility that this credential may have been copied or exfiltrated to other password managers outside of Ruby Central’s visibility or control.
> 2. Ruby Central failed to rotate the AWS root account credentials (password and MFA) after the departure of personnel with access to the shared vault.
Right?! Did nobody there think to actually disable the accounts? These are the people who are harping about "security" being the reason for the ham-fisted takeover of the source repos, but they didn't secure the production infrastructure?
It didn't occur to them that he might have written the password down? That's wild.
No matter how you slice it this is miserable root password security. Why do maintainers need root access? No one in my org has root access but me and all those creds are tied to hardware MFA locked in my MDF.
Or more realistically, accessed the accounts via IAM token and/or service account.
Something they also failed to consider, reading between the lines.
If they really have ethical concerns regarding sharing data with third parties, maybe they should update their privacy policies accordingly?
"We collect information related to web traffic such as IP addresses and geolocation data for security-relevant events and to analyze how and where RubyGems.org is used."
(https://rubygems.org/policies/privacy)
"We may share aggregate or de-identified information with third parties for research, marketing, analytics, and other purposes, provided such information does not identify a particular individual."
> “Following these budget adjustments, Mr. Arko’s consultancy, (…), submitted a proposal offering to provide secondary on-call services at no cost in exchange for access to production HTTP access logs, containing IP addresses and other personally identifiable information (PII). The offer would have given Mr. Arko’s consultancy access to that data, so that they could monetize it by analyzing access patterns and potentially sharing it with unrelated third-parties.”
WTF. This is the same guy that is launched gems.coop, a competing index for Ruby gems recently.
On the other hand, RubyCentral actions were truly incompetent, I don’t know anymore who is worse
> failed to rotate the AWS root account credentials ... stored in a shared enterprise password manager
Unfortunately, many enterprises follow the poor practice of storing shared credentials in a shared password manager without rotating them when an employee with prior access leaves the company.
You might be surprised/horrified at the number of shops I run into that use shared creds without a password manager, still use creds from ex-employees because changing them smells too much like work, and ask "why would I do that?" when you ask about rotation.
Presuming, as a group full of security peers kibitzing about this in a chat right now all do, that the "unauthorized actor" here is Andre Arko, this is Ruby Central pretty directly accusing Arko of having hacked Rubygems.org; it depicts what seems to be a black letter 18 USC 1030 violation.
Any part of this narrative could be false, but I don't see a way to read it and take it as true where Arko's actions would be OK.
Putting myself in Arko’s shoes, I can imagine (charitably!) the following choice, realizing that I still have access and shouldn’t:
1. Try to get in touch, quickly, with someone with the power to fix it and explain what needs to be rotated.
2. Absent 1, especially if it cannot be done quickly, rotate the credentials personally to get them back to a controlled state (by someone who actually understands the security implications) with the intent to hand them off. Especially if you still _think_ of yourself as responsible for the infrastructure, this is a no-brainer compared to letting anyone else who might be in the same “should have lost access but didn’t, due to negligence” maintain access.
Not a legal defense, but let’s not be too hasty to judge.
I hadn't yet seen it when I wrote this, but 2 is pretty much exactly what Arko says:
> Worried about the possibility of hacked accounts or some sort of social engineering, I took action as the primary on-call engineer to lock down the AWS account and prevent any actions by possible attackers.
https://andre.arko.net/2025/10/09/the-rubygems-security-inci...
Even if Arko wasn't the unauthorized access, this absolutely tarnishes his reputation for the rest of his career, damning email notwithstanding.
Horrible time to own/run a consultancy. Can't imagine what his other customers are thinking right now.
So is this a smear of Arko (and by extension Ruby Gems' sloppy security) but dressed up like a Security disclosure?
If I'm reading it right, it seems quite petty (and a bit cowardly). Arko was a maintainer was he not? How is that a breach? Presumably his credentials were not misbegotten, or is that the accusation?
After Arko's direct access was revoked, Arko retained access via possession of the root password (which RC should have rotated at the same time). Arko then changed the root password, locking RC out of their AWS account, waited a couple weeks, and then Joel Drapper blogged about the situation with proof that the now-fired Arko controlled the account, in order to make RC look bad.
one assumes he copied the AWS root password out of the RC-provided enterprise password manager / vault onto his own personally controlled password manager before he was locked out, which might be forgivable if it wasn't the root login for a major language's package registry
It’s only a smear if all of the public comments thus far against Shopify and Ruby Central are also smears.
Update from Andre Arko: https://andre.arko.net/2025/10/09/the-rubygems-security-inci...
Discussion: https://news.ycombinator.com/item?id=45535149
Not a good look for Ruby Central IMO.
Does anyone know if Arko can have legal troubles (like being sued) if it can be proven he "removed authorized users" from RubyCentral AWS account?
https://www.justice.gov/jm/jm-9-48000-computer-fraud
at least a misdemeanor. most of the time its prosecuted, a felony.
RubyCentral is a legal entity and they paid him for his work so they definitely could take action against him if he was shown to have harmed users or the company in any way.
Legal recourse is an interesting omission from their update, especially considering this was surely approved by their attorney(s).
The most notable thing about this post is the great lengths to which they are willing to go to detail every step of this process, something they never came close to doing in the fallout from the original stripping of rights from the longtime volunteers.
I've read both sides of the story and it does not make sense why he would change the root password and not communicate this with RubyCentral or make any attempt to contact them about the change. That in itself suggests malice.
They do seem to have found the perfect scapegoat to point everyone towards to deflect from all the other issues with their actions
Not only that but they get to use an individual who they have philosophical differences with. You can say it was 'good security practice', tarnish his reputation, and get to switch the narrative to something sympathetic to yourself all in one go. Very convenient for them.
I think they make a lot of overly strong claims here, even though there are plenty of alternative explanations possible. The mere fact that 3 people had AWS root access during this period but they only identify one and never question that it could have been one of the others is telling. They reallllly want you to just take it as obvious that 1) all these actions were taken by 1 individual and 2) that individual was malicious. Then they sprinkle in enough nasty sounding activities and info about Andre to get you to draw the conclusion that he is bad, and did bad things, and they had to do these things the way they did.
Using what reads like a business strategy email as a 'nefarious backstory' is so bad faith. I bet if you got access to all the board's emails you would see a ton of proposals for ways to support RubyGems that may not all sound great in isolation. They are being just transparent enough to bad mouth Andre while hiding any motivations from their end as purely 'security' related.
Well this certainly clarifies a lot. I'm not sure I'm confident that they know nothing was compromised further or the extent of any data exfil, it sounds like a lot of logging was enabled after the incident occurred.
Ethical and legal boundaries? RubyGems Privacy Notice already tells you that they share information with a number of large firms and notably ClickHouse... for "Customer Data Processing."
All this proposal does is request from one of the maintainers/on-call providers? another entry in this Privacy Notice as a part of a payment deal.
This is a mess, but it also unnecessary smears both sides. It calls out that RubyCentral had poor cloud management in place, and it trashes an on-call provider.
This is a terrible postmortem and all it does is advertise to users that RubyCentral doesn't know what it's doing.
[flagged]
I have no doubt a certain contingent who tend to flock on a certain former bird-themed network are happily spreading their votes around.
All my comments on the previous thread has been retroactively flagged so I can attest to this.
I think the point being that whatever was going on with access controls here was blatant gross negligence, because what is this guy doing with access who doesn't literally own the organization or the intellectual property?
Set that aside, which obviously stinks, but then why is said obviously incompetent organization sharing confidential corporate emails with the public, saying this guy proposing a corporate data access plan in exchange for consulting fees is crossing "ethical and legal boundaries?"
Are you serious? This reeks of someone or a number of people who have no idea what they're doing. Have these people never worked with a business before? Never spoken with a software firm? A marketing firm? Any consultancy whatsoever?
Who owns the registry? What is going on here? This is insane.
It makes the entire Ruby ecosystem look like it's run by children.
If only the Ruby ecosystem was really run by children, that sounds way more for than whatever this is. Joking aside though I think you are very right to question the professionalism behind this post. The narrative spin here is so toxic to good faith arguments around how we could change the 'supply chain' of Gems for the better. Wasn't that what this was all about Ruby Central? Can we get an update on that instead of this attack on your former admin?
Apparently the guy isn't even just a former admin, he actually seems to be a pivotal person around the whole formation of the RubyGems ecosystem, which is even weirder.
This isn't some random consultant who happened upon an admin position.
This is an absolute disaster for Ruby and the integrity of the Ruby ecosystem...
I dont know if you can build a product with Ruby beyond this without basically having your own highly restricted gem cache.
"The root account credentials, essentially the highest level of administrative control, are stored in a shared enterprise password manager in a shared vault to which only three individuals had access: two current Ruby Central staff members and one former maintainer, André Arko"
I am wondering. Did they at least have MFA enabled on the root login or not ?
Yes because they state under the section "Root Cause Analysis"
> Ruby Central failed to rotate the AWS root account credentials (password and MFA) after the departure of personnel with access to the shared vault.
If both password and MFA are stored in the same shared vault then MFA's purpose is compromised. Anyone getting access to that shared vault has the full keys to the kingdom the same as if MFA wasn't enabled.
Also in this day and age, there's no reason to have the root account creds in a shared vault, no-one should ever need to access the root account, everyone should have IAM accounts with only the necessary permissions.
> If both password and MFA are stored in the same shared vault then MFA's purpose is compromised. Anyone getting access to that shared vault has the full keys to the kingdom the same as if MFA wasn't enabled.
absolutely
> no-one should ever need to access the root account
someone has to be able to access it (rarely)
if you're a micro-org having three people with the ability to get it doesn't seem that bad
everything else they did is however terrible practice
So they failed to properly protect their credentials?
This sure doesn't reflect all this supposed professionalism and improvements RC was supposed to make.
Years ago, I decided with all the DHH drama, that using Rails was too much of a liability and this shit just makes the whole Ruby ecosystem a liability to anything build in that ecosystem.
Honestly I think everyone involved here, and I know most of them, needs to step back and grow up. Despite the formal tone, and the potential legal issues at play here, this situation seems to be descending into needless childishness with potential life damaging side effects. All sides are being disrespectful.
In 2025 there's no reason for anyone to be logging into an AWS account via the root credentials and this should have been addressed in the preventative measures.
There's no actual control improvements here, just "we'll follow our procedures better next time" which imo is effectively doing nothing.
Also this is really lacking in detail about how it was determined that no PII was accessed. What audit logs were checked? Where was this data stored?
Overall this is a super disappointing postmortem...
> In 2025 there's no reason for anyone to be logging into an AWS account via the root credentials and this should have been addressed in the preventative measures.
I am curious what preventative measures you expect in this situation? To my knowledge it is not actually possible to disable the root account. They also had it restricted to only 3 people with MFA which also seems pretty reasonable.
It is not unheard of that there could be a situation where your ability to login through normal means (like lets say it relies on Okta and Okta goes down) and you need to get into the account, root may be your only option in a disaster situation. Given this was specifically for oncall someone having that makes sense.
Not saying there were not failures because there clearly are, but there have been times I have had to use root when I had no other option to get into an account.
You don't need the root account, unless you need to bypass all policies. In such a scenario, you a use the root access reset flow instead, reducing standing access.
As for other flows (break glass, non-SSO etc), that can all be handled using IAM users. You'd normally use SAML to assume a role, but when SSO is down you'd use your fallback IAM user and then assume the role you need.
As for how you disable the root account: solo accounts can't, but you can still prevent use/mis-use by setting a random long password and not writing it down anywhere. In an Org, the org can disable root on member accounts.
To me that sounds like security by obscurity not actual security.
If you have the ability to go through the reset flow than then why is that much different than the username and password being available to a limited sets of users. That would not have prevented this from happening if the determination was made that all 3 of these users need the ability to possibly get into root.
As far as having an IAM user, I fail to see how that is actually that much better. You still have a user sitting there with long running credentials that need to be saved somewhere that is outside of how you normally access AWS. Meaning it is also something that could be easily missed if someone left.
Sure yes you could argue that the root user and that IAM user would have drastically different permissions, but the core problem would still exist.
But then you are adding another account(s) on top of the root account that must exist that you now need to worry about.
Regardless of the option you take, the root of the problem they had was 2fold. Not only did they not have alerts on the usage of the root account (which they would still need if they switched to having long running IAM users instead, but now they would also need to monitor root since that reset flow exists) and their offboarding workflow did not properly rotate that password, which a similar problem would also exist with a long running IAM user to delete that account.
At the end of the day there is not a perfect solution to this problem, but I think just saying that you would never use root is ignoring several other issues that don't go away just by not using root.
Not using root means not bypassing policies. There is no way to not bypass all policies. So yes, never using root makes that issue go away completely.
As for all the other stuff: what it does is it creates distinct identities with distinct credentials and distinct policies. It means that there is no multi-party rotation requires, you can nuke the identity and credentials of a specific person and be done with it. So again, a real solution to a real problem.
It solved “a” problem buy not “the” problem.
It depends on what the goal of all of this was, which is unclear. If the goal was simply to get the data that they originally wanted it does not solve that problem and it would have just happened a different way.
According to the article there was 11 days between the first actions taken and them finding out it happened.
If instead of a root account you have a long running IAM user that you can then assume into the role you normally use through SSO. If you also do not monitor that account with proper alerts and proper offboarding procedures than they could have logged into that account and retrieved the data they wanted.
Which again is the reason I am saying they just saying not using root is not a magic bullet that would have avoided problems. Maybe the situation would have been different but they still could have done a lot in 11 days.
The problem was that the user's credentials were revoked but because the root account was a shared credential it wasn't revoked. Was the break-glass account also a user-specific account, it would have fit in with any 'revoke anything for user XYZ' workflow instead of being a root account edge-case.
So, in short, this would likely have prevented this, as the normal off boarding for user-bound credentials worked out fine already.
Resetting the root password requires proving access to the email address associated with the root account. It also leaves a massive papertrail.
By "massive papertrail" do you mean "a pair of emails to the associated address"?
No, a massive amount of CloudTrail logs.
Does it? Pretty sure that logging in as root generates one cloudtrail per action, regardless of whether or not you did it with a saved password or you reset the password. Resetting the password doesn't generate a cloudtrail event as far as I've seen.
I'm now questioning my sanity but I thought you could disable login for the root account in AWS.
Since there are certain operations that can only be done with the root account, there is no way to disable access to it.
Since 2024 you can disable the root credentials on all accounts except the Organization management account: https://aws.amazon.com/blogs/aws/centrally-managing-root-acc...
I don't think the post mortem details whether the root access was on the org management account or an org member account.
Sometimes I log into the root account to see the billing information.
I created an "administrator" account, but apparently it can't see the billing information, including the very-important amount of remaining cloud credits.
Maybe I could spend time fiddling with IAM to get the right privileges, but I have more pressing tasks. And besides, on my personal AWS account I only log in with the root account.
So we have DHH with his unhinged posts on one side, and Arko wanting to sell PII on the other. Great!
I think we need an f-droid-like project for Rubygems that builds the gems from source, and takes care of signing, and is backed by a non-profit that is independent from Rails/Shopify
Gem can pull in gems from any repository, even straight from a git server like GitHub. And most of the time gems are built from scratch on your computer, Nokogiri is the only one I can think of that isn't.
The problem, as with every package manager, is transitive dependencies. It's all well and good to set up direct dependencies to only pull from git repositories, but bundler still needs a way to resolve those gems' dependencies.
You could pre-resolve every dependency in your chain to a git repository, even to a fork under your own control, but that will end up being a maintenance nightmare.
Can't a middle compromise happen as it happens in something like golang?
Can some vps/serverless provider not do this like fly.io as an recent example with kurt got got? or hetzner?
I think that golang's model can actually be sort of cheaper/ more cost effective for servers as compared to how ruby might be doing it right now and so cheaper might mean that a new non profit can be created which can work on less money/outside funding/drama overall
Retrofitting Go's dependency model into Ruby is not trivial. Go has used full URLs for dependencies from the jump, making a central package repository irrelevant. Ruby doesn't have that. At best you might have a source code URL in the gem source that you can access from a gem server, but that doesn't really anything. Someone still has to provide the index.
> I think that golang's model can actually be sort of cheaper/ more cost effective for servers as compared to how ruby might be doing it right now and so cheaper might mean that a new non profit can be created which can work on less money/outside funding/drama overall
It also means no code signing and the natural capture of most of the ecosystem by Microsoft (due to devs preferring to host their code on github, a bundler that lacks package hosting will be entirely at the whim of MS)
If you are worried about github/MS capture... Then my suggestion is to just create mirrors of golang projects you like on gitlab/codeberg
But this is so so much better than having arko or somebody having your PII.
Like I hate github but I am pretty sure that people there aren't actively looking for my PII when I download go projects or that a single person couldn't really access it I suppose
I am not really familiar but if I remember the heads project related to coreboot isn't there a way to sign your github repository with your ssh key or something related (I can be wrong, I usually am)
Like I know it could be a pain in the ass but if you are so worried about github, what if we could optionally have everything be gpg'd via ssh keys & the project could only work if someone shares a ssh key
And something like rubygems could just have a name <-> github mapping <-> gpg mapping and it might require some additional software right now but I am just giving ideas maybe for new languages as well I am not sure
What are your thoughts? And what do you think the ideal way could be. I have heard from many people (like primagen) that golang is the best package model and I also resonate with that statement but yeah github is a bit of menace/threat to open source
All the more reason to use something like codeberg!
Yes!
Not even sure why you are being downvoted, this is such a great idea actually.
F-droid has been so professional and they are just so professional
There was this developer (axet) who recently accused f-droid of somehow convincing the users "maliciously" that the funds are going to the the creator and f-droid when in reality it was going to f-droid and he name called them and what not..
Do you know what f-droid team still said?
They said that they can help him in the donation process and remove theirs and they actually took some feedback from what I know...
They clarified that the donations in their about page that the money that you donate through f-droid in their website's homepage donate goes to f-droid only which should be obvious but for some it wasn't
they also had f-droid donate in the website links of apps and I am not sure when they stopped it but they also stopped it and I deeply deeply respect it.
Like, okay maybe mistakes happen but f-droid is seriously good corporation. We might need something like that for sure. I genuinely think that out of thinking about open source so much, I realized that we need to have priorities to share things about open source.
F-droid is on the top of the list, its just that great, then there is signal/grapheneos or maybe all 3 are on top...
F-droid as an organization is something that I deeply appreciate and its a shame of google's attestation. I genuinely love f-droid nowadays.
> Not even sure why you are being downvoted, this is such a great idea actually.
Expressing negative opinions about DHH is not well-received here.
Oddly enough the Ruby community includes both the most thoughtful and gentle people and the biggest assholes I know... I refuse to believe the latter are not fringe.
Yes I absolutely hate DHH as well
and in fact I was using omarchy but then migrated over to cachyos hyprland
https://jakelazaroff.com/words/dhh-is-way-worse-than-i-thoug...
DHH is not a good guy but the hype around him made me feel so. He's weird and racist and fascist.
Stop the hype around dhh and everyone please read the article everybody here's DHH reality
Let’s ditch the superlatives and review David’s post objectively:
He thinks that even if you were born in the UK, you only count as British if you’re white.
He wouldn’t consider living in London specifically because it has too many people of color.
He uses racist tropes to accuse Asian men of being dangerous predators who attack white women.
He pushes debunked conspiracy theories about immigrants replacing white people.
He finds a march where speakers called for banning all non-Christian religions and ethnically cleansing immigrants “heartwarming”.
Finally — and maybe most alarmingly — he argues that all of the above is normal and not extreme.
The above lines were from the article
This guy shouldn't remotely be talked about in a good light imo, yes I appreciate open source but I can't seperate the art from the artist.
I genuinely don't know why they defend this guy.
HN literally flaged this post in literal minutes when it had come out but I was lucky to have read it and I will continue to spread this word because HN's moderators seem to flag anything like this and its kinda sick and enabling behaviour really
They will allow the post that cf is sponsoring omarchy promotion thing or omarchy links in general but not a dhh-is-way-worse-than-i-thought/and I was surprised by how quickly they deleted the post that after I had read that post, it got flagged and I couldn't even write a comment.
A little bit Shocking if I can be honest.
I was on omarchy but now I am on cachyos hyprland and I learnt some custom live iso stuff too, I might make an article about it... I edited this because maybe I got a little angry towards DHH but I genuinely don't like the guy. I genuinely admired him as a person untill I found about it and I have strong opinions on him.
I think its the paradox of tolerance, should we as a society be tolerant to the intolerant people?
You’re taking the interpretation of the author of that article as the words that DHH said.
Do you legitimately believe DHH would say those are his beliefs?
I hope that you have read the article but they have given a sound reasoning behind it.
DHH mentions 39% or something which was the population of native white and not native british as an example...
Please read the article link and they have given a proper sound reasoning...
>Do you legitimately believe DHH would say those are his beliefs?
Yes, I mean, DHH wrote it in his own blog post. There is still an argument to be made that DHH is far right but even he knows that it is bad and somehow tries to normalize it...
DHH might not say that these are his beliefs but his words in his blogs logically point to this conclusion. Why do you think that DHH said those words in his blogpost if he doesn't believe in such similar far right ideologies? Nobody forced him to write a blog post but himself...
Why do you think such things are not what DHH believes in? Do you have any evidence as the author of the article provides for their reasoning/interpretation?
> Why do you think such things are not what DHH believes in?
Because the one thing DHH doesn't do is shut up.
He hasn't really been the kind of person who minces words, he says what he thinks, and he is pretty unafraid of pissing people off. If he wanted to make explicit racist statements, it feels like he would make them.
My read is that this is broader commentary on civic/cultural integration. There are, fundamentally, immigration challenges that present themselves in modern society, especially when cultural values are different. I don't know that it's wise to believe that everyone will eventually see the world the same way, and we are then left with a question of how to reconcile that as we develop our societies and cultures.
Tommy Robinson is a violent anti-Islamic voice -- But, we've unfortunately found ourselves with those voices being the ones willing to speak to the problem that a growing portion of the population feels is unaddressed.
I don't think these are easy problems. I also cringe at calling Robinson's march "heartwarming" without qualification.
Yet I still am not willing to, without much stronger and more explicit evidence, read into DHH's words and label him a `far-right racist` -- Because I think that label loses effect when it's applied to every person that we disagree with on certain policy issues.
RC seems to be incompetent and malicious, the original hostile take over is explained as a security measure. But in reality, you can simply just take away the ability to deploy and leave access to contribute to the repo untouched for people outside of the RC organization.
But claiming that you care about security while missing a basic step of rotating credentials once a member has moved away is pathetic. God help Rubygems.org and the users.
That email screenshot is pretty bad for Arko. It clearly shows intent to sell PII data to a third party during a time when Ruby Central had diminished funds and needed help affording basic services.
What the fuck.
Why do they need money? What happened to their funding?
It was after a big sponsor pulled out and presumably before Shopify stepped in...
One of the major sources of funding was cut because they sided with the devil...
The dastardly devil who made the whole thing popular in the first place! Quite a devil indeed.
they were funded by a company run by a guy who read DHH's blog and cut funding unless they excommunicated him
interesting timing, isnt it?!
Lol and in previous posts everyone was acting like Arko is above board...
I brought up multiple times that his actions were suspicious, was downvoted. Now proof of that plus an email trying to low-key extort RubyCentral into allowing him to sell user data...
It's not extortion, because refusing his offer would have left them no worse off than if he hadn't gotten involved in the first place. You can still object on other grounds, of course.
At least it’s not npm. Or cargo.
Props to Ruby Central for taking all of the smears and reputational damage on the chin silently while they mitigated an actual security incident, made absolutely sure and wrote up a proper post-mortem. All of that in line with their original statement that their actions taken were in the interest of security/integrity of their platform.
If there's any evidence that you need to know who the proper stewards of Ruby's gems are, it's this.
An entity that promised security had a security incident due of their incompetence to properly secure their production environment root access?
If somebody is going to abuse their accidentally-retained access after being removed from my organization, than the incompetence was in having that person in my organization in the first place. It turns out they were perfectly justified in removing him!
First of all, it's criminal, and second of all, it absolutely lights a torch to any credibility they have. I expect people don't want to become unhireable.
I've had access/credentials to organizations that I've left and never abused them even once.
it's not that clear cut, because there is no "rubygems company" or clear ownership of any of this stuff
it would be quite easy to argue that ruby central had never had a right to remove these people at all
> I've had access/credentials to organizations that I've left and never abused them even once.
yes, likewise
and if I was Andre I wouldn't have even have ATTEMPTED to do this, as it looks terrible regardless of the eventual legal determination
odd timing for such an incident..
Not really. Someone abusing their access after being removed is exactly when such events occur.
Your post suggests conspiratorial thinking when there shouldn't be.
my post suggests critical thinking when there should be..your post is just defamatory. all of this looks like a "problem, action, solution" scheme. what was the takeover then, a honest and transparent move by RC? you must be kidding. looked pretty much conspiratorial.
I mean I'd much rather it be people who remember to rotate their passwords after firing high profile staff