A first look at Django's new background tasks
roam.be60 points by roam 4 hours ago
60 points by roam 4 hours ago
Assuming you're fine with keeping the queue in postgres, I've used Procrastinate and it's great:
https://procrastinate.readthedocs.io/en/stable/index.html
Core is not Django-specific, but it has an optional integration. Sync and async, retries/cancellation/etc., very extensible, and IMO super clean architecture and well tested.
IIRC think the codebase is like one-tenth that of Celery.
This is an exciting development. I imagine I'll continue using Celery in most cases but being able to transparently swap back-ends for testing, CI, etc. is very compelling.
I haven't looked into this in any detail but I wonder if the API or defaults will shave off some of the rough edges in Celery, like early ACKs and retries.
How is the typing support? We just had downtime because a change to a celery task didn't trigger mypy to complain for all call sites until runtime. Too many python decorators aren't written with pretty weak typing support.
With regards to args and kwargs? None. Your callable is basically replaced with a Task instance that’s not callable. You need to invoke its enqueue(*args, **kwargs) method and yeah… that’s of course not typed.
Static analysis will never be fully robust in Python. As a simple example, you can define a function that only exists at runtime, so even in principle it wouldn’t be possible to type check that statically, or even know what the call path of the functions is, without actually running the code in trace/profiler mode.
You probably want something like pydantic’s @validate_call decorator.
> you can define a function that only exists at runtime, so even in principle it wouldn’t be possible to type check that statically
Can you say more, maybe with with an example, about a function which can't be typed? Are you talking about generating bytecode at runtime, defining functions with lambda expressions, or something else?
I’ve been using the django-tasks library in production for about a year. The database backend and simple interface have been great. It definitely isn’t intended to replace all of celery, but for a simple task queue that doesn’t require additional infrastructure it works quite well.
This is great! The prev recommendation was usually a lib called celery that I wasn't able to get working. I don't remember the details, but it had high friction points or compatibility barriers I wasn't able to overcome. This integration fits Django's batteries included approach.
I've been handling this, so far, with separate standalone scripts that hook into Django's models and ORM. You have to use certain incantations in an explicit order at the top of the module to make this happen.
> scripts that hook into Django's models and ORM
Django has management commands for that [1].
When you use Django over time, you experience this pleasant surprise over and over when you need something, “oh, Django already has that”
[1] https://docs.djangoproject.com/en/5.2/howto/custom-managemen...
Django this is about 10 years too late. It's frustrating because we use all manner of hacks to work around this being part of the builtin story.
[dead]