Ask HN: GitHub employees what's going on? Why?

The fools thought we needed data centers for agents, but we actually needed data centers for Github!

GitHub’s reliability issues started earlier than the boom in agentic coding they talk about. I have comments on here from 2025 talking about massive outages with actions and issues.

I want to believe that it's justification, but looking at the historical uptime [0] (early data is actually overly positive on this graph since it had to scrape historical data instead of realtime), I can't be convinced of this anymore. They do have more load - that's for sure. I just don't believe it's an excuse for their god awful uptime.

[0] https://damrnelson.github.io/github-historical-uptime/

I would take what C-level execs say with a grain of salt. Remember these folks hire people with plenty of system design experience. They likely also have the money for hardware resources.

If load was an issue these things would have been solved already. Or they are purposely causing millions of dollars of productivity loss daily by not solving this.

Something more serious is at play. And I want to hear from the actual underlings.

That is a good point. And this issue looks analogous to what is happening in the research publication world (arxiv, peer-review, etc). In particular, I wonder whether code pushed is of lesser quality, CI pipelines break more often too, and this saturates github servers.

I'm curios what is the bottleneck then? Because github(ms) has enough resources to hire competent people and provision needed hw to solve scaling the issues.

My sense is that a lot of this activity isn't even legitimate use. As one example: my company has a GitHub app, and last night some bot added my app to 1700 repos. Then the bot immediately started rapid-firing commits which each affected hundreds of files, triggering a deluge of GitHub webhooks to my servers.

The repo names all matched other GitHub apps, so my sense is this bot had added 1700 GitHub apps to 1700 repos, with that number rapidly increasing at the upper bound of whatever GitHub's rate limit is.

My systems caught it quickly and auto-booted the bot, but the whole situation is ridiculous.

> Github is struggling because AI-boosted coding increased the number of commits 14x in the past year, and the pace is still accelerating.

Sure, but how is OpenAI, and Claude handling the increased usage? They are handling it a lot better than Github. Just because you have fast growth does not mean you can not handle it.

Instead you have to realize that you are entering into a high growth phase and design around that.

It has been clear that we are in a high growth phase for around a year now and Github still hasn't adapted to that properly.

At some point you have to admit that Github ready for the moment.

Turns out that the cloud doesn’t scale that easily by itself after all?

The obvious answer is to end free unlimited private repos. Source is no longer a specific thing. GitHub is just a free S3 or Azure provider.

I can't believe they've been asleep all these months, not foreseeing that AI would generate more commits and Git usage and, consequently, not preparing themselves to scale.

Github is owned by Microsoft, its not like its some boostrapped startup who has a closet of servers that are melting.

Nobody at Microsoft could've predicted AI was going to be so popular!

If that's the superficial cause, then the root cause is a lack of limits. Any production system in the world will go down from too much load if you don't limit the load. You have to do performance testing to determine what the system's peak utilization is, and put limits in place to prevent exceeding that limit.

There are multiple kinds of limits you need. First you need the raw i/o limits on things like networking, load balancer requests per second, etc so you don't slow to a crawl on high load; you can throttle max i/o so demanding users don't steal performance for others, or even throttle network traffic to the client IP and internal services. Then you need additional limits at app layer, like number of requests you can process that make DB calls (as most DBs have a connection and request limit), or limit API requests to X/minute. Then you need more business logic limits, like max number of concurrent logged-in users, max number of repos a user can have, max number of requests for free users, priority for paid requests, etc.

When you hit a limit, you have a couple choices. You can shed load (dropping calls immediately, which results in 503 errors for users, or even network disconnects), but it makes a very poor user experience, and can cause larger issues/conflicts in a distributed system. You can instead keep a max number of active sessions, and adjust that number over time, to ensure that (on average) your users have a stable experience; this requires implementing a "waiting room" for new sessions when old ones expire or log out; still not great for users, but it prevents everyone from getting random errors, which means the majority of users have a good experience.

There are way more kinds of limits to implement. Each business has to analyze its entire system to identify performance bottlenecks and implement limits to ensure the majority of users continue to have good service. Without those limits, you get a shit experience for everyone, your brand is tarnished, and customers run for the hills.

Finally, the performance testing helps you plan how much infrastructure you need to serve a given number of users. If you plan properly, you know X infra allows for Y number of users. Once you exceed Y users, you need to start throttling or stopping new user signup (and requests per minute) until you can increase capacity. Continuing to allow signups and additional usage, without the capacity to support it, results in what we see today.

But competent engineers know all this. If they had competent engineers, and it gets this bad anyway, the problem is management. Microsoft likely already made back what they acquired GitHub for. They probably figured they could keep riding the gravy train until the train went off the rails, and either eventually sell GitHub or write it off. Such is business.

I think that data is a bit dated and misleading, but I would love to see if AI slop is a contributing factor here once the dust settles and how Github will address it. Looking at the return to mean in https://areyougoingexponential.rhys.dev/kdaigle makes me think the COO spike could be a Karpathy wiki or an agent too.

> Github is struggling because AI-boosted coding increased the number of commits 14x in the past year,

OK so then use the additional revenue to add more resources and employees? What the f*k are they doing over there?

Are these disproportionately in the free tier? Or is the increased storage load similar across the paid tiers?

Might be worth considering a /GB/MO fee. Let's face it, VCS is a (glorified) storage service.

It is just an excuse. If they can't support paid member or don't have infra they should start charging these slop projects who are committing relentlessly.

In all fairness, the features expanded substantially after the acquisition so there's are a lot more components that can experience downtime. Pre-acquisition, github was a fairly simple code repository with issues bolted on. Actions, Dependabot, Codespaces, Advanced Security, Package Registry, Copilot, etc. all came after the acquisition and probably involve much more overhead than what github was prior to the acquisition. It basically went from code hosting platform where machines are sitting idle when pushes and pulls aren't happening to a fully integrated developer ecosystem. Any one of those things experiencing degraded performance is going to show up on an uptime chart.

To be fair, before the aquisition, GitHub was super stable, but also it felt like they rarely changed or added anything.

After Microsoft acquired them, they started rolling out new features at an incredible pace. And if you look at the breakdown of the downtime, most of the downtime is in features that didn't exist before the acquisition.

Though not all of the downtime is in new features. The core is also less stable than ever.

I’ve noticed the same thing with computers, whenever I install MS windows on them uptime tanks ;)

Note that most of the downtime in that graph comes from Github Actions, which simply didn't exist before the Microsoft acquisition.

The downtime also starts happening in 2020, which is a while after the acquisition. 2020 was the year many people started learning to code while being stuck at home.

Microsoft acquiring GitHub is not a valid root cause analysis explaining Github’s degraded SLA.

On the other hand I think usage increased and GitHub operates now at a different scale.

Microsoft acquires GitHub and subsequently moves all infrastructure to Azure

No one hates msft more than me, but there’s no way you think this one data point is useful to explain anything at all

So perhaps we should be asking Azure engineers, not Guthub engineers?

The reality is that it's probably a skill issue. Scaling a mature platform 10x can be a really, really hard problem, they obviously don't have their arms around a solution, and are probably spending a majority of their time on ops to keep the bleeding down. In the old days you would have Jeff Dean come down from heaven and invent a new database for you or something like that. It doesn't really seem like Github has that kind of technical ability, so they're probably trying to cobble together Azure ops with internal bandaids while everyone internally is cranking out AI code and it's just not going to be enough.

I mean they may be working around the clock, but after 5 or 6 layoffs in 3 years… half their engineers are probably still learning the code base.

Frankly yes. Do it in a way that doesn’t violate outright or cover your tracks. Customers deserve to know the truth of the type of company they are dealing with.

The alternatives exist but aren't exactly stellar. Otherwise I think this thread would be filled with recommendations.

AI definitely does look at git commit history in my experience

I pick Azure but not Microsoft (?)

Also the fact that its written in Ruby on Rails probably doesn’t make it easier, performance wise.