GitHub Is Experiencing Difficulties
githubstatus.com110 points by Jeaye 4 hours ago
110 points by Jeaye 4 hours ago
I don't think this portents anything great for software in general.
We're a good year+ into the use LLMs for all major bits of software that we all rely upon and GitHub here is down to one 9 of uptime. I've been using GitHub for a _long_ time, my first commits there go back to August 2009!, and I honestly don't recall GitHub going down as much as it has in the last year.
I'm sure there's other things happening in the background, but I can not help but believe that this is directly correlated with the increase of LLM usage.
Though I would love to hear someone else's pet theory how a rock of the internet went from four+ nines of uptime to maybe one.
Are these outages caused by introduced bugs, though, or by load issues?
As someone who has spent many years working in high load environments, this is not an uncommon pattern.
You design a system and it works great. It can handle failures, load spikes, it is horizontally scalable, things are great. You think you figured it out.
And then load keeps increasing and you suddenly hit a tipping point where everything keeps failing, and you cant keep up. The things that you thought were perfectly horizontally scalable turn out to have a bottleneck you didn’t even think about until you got to a truly massive scale. Your systems suddenly don’t have the excess capacity to handle load spikes or catchup work, so suddenly any failure cascades and recovery is more and more difficult. You can’t solve the problem with additional hardware, and your perfect scalable design actually can’t scale any more.
This doesn’t have to be about GitHub using LLMs in their code to still be related to LLMs. GitHub gets a lot more commits now because of LLMs and probably get a lot more reads because of LLMs as well.
It could be that the extra usage just pushed them past one of those capacity thresholds.
To me this correlates more to them being bought by Microsoft, a company known for being seemingly incapable of creating quality software to the point that it's not even funny anymore, and also known for sloppifying all the products they touch.
I don't think so. GitHub was bought by Microsoft 8 years ago and people have only started complaining about its uptime in the last year or so - exactly correlating with the surge in LLM use.
I don't think so. I've seen those complaints for more than a year.
I have an Ops background and I strongly suspect they were given a stupid timeline for the Azure migration.
I've got to believe Microsoft have decent Ops people but the management wanted to move faster than was reasonable and screwed it up. Move one thing at a time and double check it all works and you can do a migration like this.
https://damrnelson.github.io/github-historical-uptime/
Seems pretty conclusive. Very similar story when they bought skype.
Didn't github also migrate at azure recently? That certainly can't help.
I guess github is kind of a shared garden. Interesting that, like in game theory, if everyone is using it too much, no one gets to use it.
It's probably not attributable to AI in the way that you're thinking - Github has been absorbing an exponential increase in usage, and that increase is mostly due to new AI-related projects being created and worked on.
Though I'm sure some of the blame can go to internal slop code.
For the longest time, I didn't appreciate the "AI-induced traffic" excuse. But seriously, I checked the rough github egress for our lab versus an old log from 2024, and there's an order of magnitude or two difference. From asking around, it seems people all have the gh cli tool and let it loose with parallel tool calls and e.g LLM's polling Actions in a background bash while loop with sleep $TOO_FEW_SECONDS. Some people use a variety of skills where the agent makes a commit every few code changes, and uses Issues for its memory/log. And they have O(5) sessions at the same time doing all kinds of crap. It's the same with PR checks/PRs. Recently we also saw continued usage throughout the night as well, which did not exist pre coding agents. Loops or whatever they call cron jobs in the harnesses these days is the reason. It must be adding up.
At this point maybe they should consider sending out announcements when it is working instead
I once saw a sign at a railway station in the UK that actually did say "normal service will be operating between the 12th of July and 18th of August" (or something like that). Made me laugh. At the time Anglia Railways had a rail replacement bus literally every weekend.
I thought they should rename to Anglia Busways and have bus replacement trains instead.
It would seem like GitHub is in a precarious situation.
We have many agents per employee working in parallel pushing way more commits than was humanly possible before AI, triggering GitHub actions a lot more than the workflows were built for, causing Actions costs to escalate (they really aren't cheap if you compare to hosting it yourself), meanwhile working with YAML workflows is just a pain, and just writing code would be so much more fun and AI compatible[1].
At the same time, GitHub has about ~3 different PR review UIs? And they're all half-bad? Any decently sized PR triggers their "optimized for large PRs" UI which jumps around randomly in my experience. If you don't get that UI and keep the scrolling one (there's an old and a new one btw) then god forbid you click a line number because at some point your browser will randomly scroll back to that line and it won't unstick. Now Linear[2] (and others) is replacing the PR review experience for the agentic era.
I'd love to see a solid AI first Git + CI + reviews.
[1] Cloudflare CI https://blog.cloudflare.com/ci-workflows/
[2] Linear PR reviews https://linear.app/changelog/2025-01-23-pull-request-reviews
Given the recent history of outages, I wonder how many are seriously considering moving off GitHub for anything other than code hosting. I mean actions / workflows.
To offer a data point (not a tribalist argument): GitHub Actions being down this often is still less expensive than the cost of switching everything somewhere else. Would rather have my team walk away from the PRs and do other tasks than have engineering effort, meetings, design pages, scheduling, etc. for switching over.
This is annoying and I'm here because it's down. But it would have to be far worse to come close to actually being worth changing.
A case study in vendor lock-in.
Pretty much, eh?
If I could right now:
1. go sign-up elsewhere 2. Log into GitHub and point Actions to that new host 3. All my actions files immediately worked without question
I'd probably give it a spin and make a wiki page explaining how to swap back and forth. No meetings. No design issues. No scheduling. Just a flip switch on who to pay for computers.
For what it's worth, we switched some of our actions to a self-hosted Woodpecker instance, and although there were a few kinks to iron out, it works better overall (for example, because of better caching on that single instance, our docker images build faster).
We've started writing some new workflows against argo-workflows for the last like 9months or so. Especially for use cases where the actions are like self-service type automation.
I think people were so excited to move away from jenkins to something 'managed' just because of how much a dinosaur jenkins is and how much a pain in the ass it is to upgrade it... but now we are seeing how managed can bite you in the ass if the manager is incompetent.
GHA still beats Circle CI or trying to run your own Jenkins - even with all the downtime. I'd move in a heartbeat if there were good alternatives.
I think I agree if you don't have someone dedicated to running and maintaining Jenkins. If you have someone at your org that knows what they are doing though, I'd still take Jenkins over GHA any day.
gitlab self hosted though? no outage. i remember equivalent functionality at my last job taking this path
That's significantly more difficult to set up and it cost money. Github's big issues that it's free, Microsoft doesn't want to allocate enough budget to keep the thing running properly.
But for people who either don't pay anything at all or phenomenal amount one 9 of up time is all you need.
If you're actually trying to run a business I guess you can call and gitlab and get an Enterprise contract
Our deployments are triggered by GitHub Actions, and it is a pain to deploy without it (specifically collecting the credentials and putting them into variables the shell will read).
This is multiple times this month that this has been a problem.
Has GitHub completed it's internal migration to Azure yet? Or is it still ongoing? None of our devs want to switch away from GH, but we will have to at this point.
What would you guys recommend as an alternative? Both hosted and/or self hosted
We have set up a new enterprise and are migrating to GHE. For us this was a no-brainer; we get our own isolated environment, Enterprise Managed Users with Entra OIDC provisioning and Github Copilot in EU data residency. https://eu.githubstatus.com/posts/dashboard looks pretty good to me. We will still keep our github.com enterprise around for public repositories. I’m genuinely curious to know why a company would prefer github.com over GHE?
We self-host gitlab and it’s great. It’s not cheap, but it’s also never gone down.
Here's what I did.
Step one - migrate my build workflows to Docker.
My Github actions are now basically: "checkout / set env vars from secrets / docker-compose builder run make".
I used large machine runners to run full the Docker (Podman actually) on Github first to avoid dealing with docker-in-docker complications. This step also provided some very nice robustness advantages, as I can now trigger deployments from my laptop if needed.
Step two:
Migrate to self-hosted runners. I used my former homelab server to set up a build machine. It has 16Tb of fast NVMe SSDs and thanks to Podman container layer caching, my entire lint workflow now takes 30 seconds. Faster than just one "npm install" on Github before.
And Github's self-hosted runners are actually surprisingly easy to set up and use. They are also somewhat more robust.
Step three:
Swap Github for something else.
WoodpeckerCI for DIY, Drone.io for paid support. Self-hosted, OAuth login, repo-permission-based authorization, container-native design, easy web UI, optional RDBMS, supports many platforms. All the core features needed for scalable CI, deploy as few/many as you want for multiple teams, small enough to easily run on one box for one repo. Nothing else is as simple, easy, powerful, compatible.
I would conjecture that the long tail of GH users are not dependent on any of the CI stuff in the first place.
I seriously want to. I'm just waiting for the day where I have enough extra energy to simply bite the bullet and do it.
how hard could it be these days for a mid to large size eng company to have their own gitlab/hub type solution hosted in aws
Until someone makes a better PR experience you guys are stuck with GitHub.
Nobody cares about ATProto or whether your commits are a damn NFT or some bs just literally improve upon the experience.
That’s it.
It’s as if no company is focusing on the product experience or anybody’s experience anymore. It’s all ooo look what I got I got this I can do that too me me me but nobody will ever buy that.
Say what you want about huge companies like Microsoft or Walmart but they spend a lot of energy understanding the human experience to sell products and less on their own perceived self-aggrandizement.
GitHub is the best version control online and it’s not even close.
Forgejo has a better PR UI. Github has degraded so much that its UI can't even _show_ _the_ _fucking_ _diffs_ without clicking on "Load Diffs".
Github is just the laziest default. It's not _terrible_, but it's also not great.
My company moved to GH Actions 6 months ago even after I pushed back with a "Are you sure given their reliability issues of late?"
Now I'm stuck twiddling my thumbs with PR checks stuck/failing...
Dealing with an incident on prod is extra difficult when your pipelines are not running.
Ever since Microsoft took over, and then when AI Skynet took control, things started to decay. AI companies owe all of use a lot of money. By the way, why do I have to pay for increasing RAM prices here? Why are we so dependent on a few greedy corporations, anyway?
I knew that when the build actions started failing en masse and my newly submitted PRs did not trigger the build actions at all (besides the mandatory one in my organization), it seemed logical to me to look at HN first..
Especially troublesome in the middle of trying to fix a high score security vulnerability when the release vehicle is Github.
Self-hosting is king.
I’ve been self-hosting lore and gitea the past 6 months or so and it’s been a breeze. For anything I really care to share I throw up on Tangled.
Except anything I'd self host would be down much more often. I've hardly ever actually experienced down time issues with github when I needed it.
I've never understood this argument. Even if self-hosted things were offline more often (which I've never found to be the case, I've had Forgejo + runners running for a year now with no downtime), the real benefit is that you yourself can work to bring it back online when it does, rather than waiting on a large, slow-moving organization to figure out what slopped PR caused their global service serving ungodly amounts of RPS to go offline again.
I wrote my reply before yours appeared but it's basically agreeing. A simple Forgejo hasn't given me any grief. It really seems simpler.
Maybe they're hosting in us-east-1 though :)
Then you would be exceptionally bad at selfhosting as Github has like one nine of uptime.
It still has 2 9s according to the status page, but it is approaching Claude levels of downtime. :(
I noticed my CI throwing errors all of a sudden. I sure wish they would become more active in alerting folks or build it into these tools - especially as these occurrences are becoming more frequent. Possibly an API-accessible services status. Then, at least we could build in our own checks when we hit errors and not have to hunt down what all is broken.
I had tried downloading release assets of two different OSS that I use, and all requests failed.
It was before it became a news and a trend in X.
Really frustrating experience.
This seems persistent and endemic, have they said anything about why it's been so bad and what they are going to donto improve going forward?
It's really crazy to me they can just be down for hours and can't recover their own systems. It shows they don't have the capability or infrastructure to roll back disastrous changes. These kinds of things are a tell on the organization and operational excellence (or not). As soon as a viable alternative surfaces for Github, I'm moving off and will advise all my clients to do so as well.
As I said 6 years ago. [0] You would be better off self-hosting than using GitHub or GitHub actions. No CEO of GitHub exists and now it is falling over again.
There is no better time to self-host.
It's Thursday.
so... what happened?
using github in the big 26 is self ownage
[dead]
Azure