Fast and observable background job processing for .NET
github.com71 points by mikasjp 3 days ago
71 points by mikasjp 3 days ago
This seems to be a thin abstraction over Channels, I'd probably prefer to use Channels directly in almost all cases.
I really like Channels for in-memory queues like this, they're very nice and it's easy to make everything parallel. But if you need more than this you usually need persistence, and then you need something else entirely.
It is a thin abstraction over Channels, and that’s by design. What it adds is graceful shutdown handling, OpenTelemetry integration, timeout support, and simple configuration. I kept needing those pieces in almost every project, so wrapping them up into a small reusable library felt worthwhile.
Out of curiosity, why do you use an explicit semaphore here? I usually used Parallel.ForEachAsync together with ReadAllAsync from the Channel. Avoiding the manual semaphore handling was one of the things I really liked about Channels.
I’m not entirely sure what the practical advantage of that approach would be compared to the explicit semaphore, but I’d be happy to learn more. If you think it’s a better solution, you’re more than welcome to open a pull request
The entire main loop that queues jobs is then a single ForEachAsync that draws from the channel. And it just works, there are no real mistakes you can make with it.
Semaphores work well of course, as long as you don't make mistakes. Probably not an issue in your current version, but can easily happen if the code is more complex or especially when different developers later modify code like this. For example, you release the semaphore in a different class than the point where you acquire it, which makes this a bit less obvious than I'd like. If any developer later adds code that takes a different path this might break, and those kinds of bugs can be very annoying.
It's not really a problem with a simple case like this, but in general I don't use low-level concurrency primitives if there is a higher-level abstraction I can use that fits my problem.
A front-page HN post about .NET!! Let’s go!! Want more of these.
If you are running a web service, does this provide any advantages to BackgroundService? https://learn.microsoft.com/en-us/aspnet/core/fundamentals/h...
Under the hood, BusyBee consists of an in-memory queue built on Channels and a job processor that dequeues and executes tasks. The processor is essentially a BackgroundService, but BusyBee wires everything together for you, so you don’t have to manually set up the queue, parallel processing, timeouts and errors handling and observability for OpenTelemetry.
I'm trying to find the actual value-add here. This feels like TPL but with more steps.
Why do we need to serialize the jobs through a Channel<T>? Couldn't we just do Task.Run and let the runtime take care of scheduling our work?
You have no control over concurrency/scheduling, have to manage scoping, error handling, etc. TPL/Threads add to much low level noise to the logic.
Like you could easily blow up the thread pool depending on what you are doing, where a channel based implementation would just deal with spillover and not affect the other threads. You can easily capture scoped services that are disposed by the time the thread executes - but you never catch it in dev because you get executed immediately and request takes long enough, etc.
Spawning threads will work but I can see the use in a small abstraction like this lib for sure. Not sure I would use a lib for this - would probably hand roll something since I don't like adding unvetted external deps by default.
This is exactly why I built BusyBee. My team found ourselves hand‑rolling something similar almost every time we started a new project. If we kept needing it, I figured there are probably more people in the same situation. So instead of duplicating the same background queue logic across multiple codebases, I decided to build it once, add proper OpenTelemetry support, and keep it simple without extra bloat. That way we can just reuse it and reduce the amount of similar code we have to maintain.
> You have no control over concurrency/scheduling
https://learn.microsoft.com/en-us/dotnet/api/system.threadin...
I don't get what you are trying to say - are you going to create a task scheduler/thread pool as a singleton for background jobs ?
Channels are a built-in feature from Microsoft, and they do all the hard work here.
Yeah thats what I am saying - I would just hand roll my background worker with channels probably
I wish TPL Dataflow was more known.
Not only, most people really aren't aware of all batteries that come with .NET, and Java, more so than Python or Go.
BusyBee is a high-performance .NET background processing library built on native channels. It provides a simple, configurable, and observable solution for handling background tasks with built-in OpenTelemetry support and flexible queue management.
I use Postgres as a queue with SELECT FOR UPDATE SKIP LOCKED. You can easily spin up whatever state around the jobs you want to.
How is it different from hangfire?
Hangfire is primarily a job scheduler. It is designed for running jobs at specific times or intervals, and it persists jobs in a database so they survive restarts. It comes with a dashboard, retries, and a lot of infrastructure around long‑term job management. That makes it powerful, but also heavier in terms of setup and overhead.
BusyBee is focused on lightweight background processing. Everything is in‑memory, with no external storage required. It is designed for scenarios where you want to enqueue a task and have it executed immediately in the background, without scheduling or persistence.
A practical example: if you are building an API that accepts file uploads and you want to process the file asynchronously after the request returns, BusyBee is a good fit. You just enqueue the job and it runs in the background right away. If instead you need to schedule a nightly cleanup job or ensure jobs survive application restarts, Hangfire would be the better choice.
So what happens if the application shuts down before the file is processed? I get the appeal of a simple solution, but I don't know if I can ever recall a situation where "push work to the background and no one cares what happens to it" is ok.
How is Hangfire different from the Azure DurableTask library?
- hangfire has very user friendly admin UI
- developer experience - hangfire simpler to use for simple scenarios (in many cases you can run an existing method in background with line of code)
- hangfire has some support for batches/delay/task dependency but DurableTask is way more full featured in my opinion
- hangfire has .net 4.8 framework support :(