Scheduling Tasks in Django Django.How

Author avatar wrote on 27/05/2022

Purpose

There are long running tasks that you want to hand off to a background process without it blocking the page from loading
While we wait for it to complete.
Like sending emails and generating huge files and graphics.

Way 1: django-background-tasks

1. install

pip install django-background-tasks

2. Add to INSTALLED_APPS:\

INSTALLED_APPS = (
# …
‘background_task’,
# …
)

3. Migrate your database

python manage.py migrate

4. Creating tasks
from background_task import background
from django.contrib.auth.models import User

@background(schedule=60)
def notify_user(user_id):
    # user_id and any other parameters must all be serializable as JSON
    user = User.objects.get(pk=user_id)
    user.email_user('Here is a notification', 'You have been notified')

5. registering (calling) tasks

notify_user(user.id) # default time will be taking from the decorator
notify_user(user.id, schedule=90) # 90 seconds from now
notify_user(user.id, schedule=timedelta(minutes=20)) # 20 minutes from now
notify_user(user.id, schedule=timezone.now()) # at a specific time
notify_user.now(user.id)   # launch a notify_user function and wait for it
notify_user = notify_user.now   # revert task function back to normal function. Useful for testing.
notify_user(user.id, verbose_name="Notify user", creator=user) # specify a verbose name and a creator when scheduling a task

6. Running tasks

python manage.py process_tasks

Resources:

  • https://django-background-tasks.readthedocs.io/en/latest/
  • https://stackoverflow.com/questions/30816134/how-to-use-django-background-tasks
  • https://medium.com/@robinttt333/running-background-tasks-in-django-f4c1d3f6f06e

    • Way 2: Django Management command and cron tab

    https://django.cowhite.com/blog/scheduling-taks-in-django/
    https://devcenter.heroku.com/articles/scheduling-custom-django-management-commands

    Way 3: task as a regular Python thread

    
def start_new_thread(function):
    def decorator(*args, **kwargs):
        t = Thread(target = function, args=args, kwargs=kwargs)
        t.daemon = True
        t.start()
    return decorator

@start_new_thread
def foo():

    do stuff

    https://stackoverflow.com/questions/18420699/multithreading-for-python-django

    Way 4: Celery Most Advanced

    a well maintained and popular Python package that can be used in Django without any Django-specific extensions.

    Notes:

  • Celery is excellent, I use it my self , but it has a steep learning for setting up tasks
  • also you will be need to use message broker to send tasks to celery and additional utility to run celery in daemon mode.
  • AWS Elastic Beanstalk, answer should be yes
  • Celery is a bit more structured and wants you to store these tasks in a tasks.py file in the application
  • Celery and Django-Q both have extensive documentation
  • Require broker like (Redis)
  • use decorators with the task so all your async imports and logic are within the tasks.py file.
  • Celery has it’s own logging functionality which is imported and appears to work just fine.

    • https://testdriven.io/blog/django-and-celery/

  • https://stuartm.com/2020/04/basic-django-celery-example/

    • Way 5: Django-Q

    a task queue, scheduler and worker application designed specifically for Django.

  • Require broker like (Redis)
  • Django-Q and Django-RQ both just use standard Python logging which I preferred for my use case.
  • Celery and Django-Q both have extensive documentation

    • Resources:

  • https://stuartm.com/2020/05/basic-django-q-example/

    • Way 6: Django-RQ

    a Django extension for the popular RQ (Redis Queue) python library

  • Require broker like (Redis)
  • use decorators with the task so all your async imports and logic are within the tasks.py file.
  • Django-Q and Django-RQ both just use standard Python logging which I preferred for my use case.
  • No extensive documentation

    • Resources:

  • https://stuartm.com/2020/05/basic-django-rq-example/
  • https://stuartm.com/2020/05/advanced-django-rq-example/

    • WAY of the future

    All of the above may become irrelevant in the future when Django implements full view-based support for async code.

    How are Django channels different than celery?

    https://stackoverflow.com/questions/38620969/how-are-django-channels-different-than-celery

    Channels:

  • Channels in Django is for WebSocket, long-poll HTTP.
  • In Django are meant for asynchronous handling of requests.
  • The standard model Django uses is Request-Response but that has significant limitations. We cannot do anything outside the restrictions of that model.Channels came about to allow Web Socket support and building complex applications around Web Sockets, so that we can send multiple messages, manage sessions, etc.
  • Django channels does support background processing. I think the question is more on, how django channels background processing is different from celery workers.
  • Django channels gives to Django the ability to handle more than just plain HTTP requests, including Websockets and HTTP2. Think of this as 2 way duplex communication that happens asynchronously No browser refreshing. Multiple clients can send and receive data via websocket and django channels orchestrates this intercommunication example a group chat with simultaneously clients accessing at the same time. You can achieve background processing of long running code simliar to that of a celery to a certain extent, but the application of channels is different to that of celery.
  • Passes messages between client and server (Full duplex connection)
  • Handle HTTP and Web-sockets requests
  • Asynchronous

    • (Example cases)

  • Real time chat application
  • Update social feeds
  • Multiplayer game
  • Sending Notifications

    • Celery:

  • Celery is for background task, queue.
  • is a completely different thing, it is an asynchronous task queue/job queue based on distributed message passing. It is primarily for queuing tasks and scheduling them to run at specific intervals.
  • Celery is an asynchronous task queue/job queue based on distributed message passing. As well as scheduling. In layman's terms, I want to fire and run a task in the background or I want to have a periodical task that fires and runs in the back on a set interval. You can also fire task in a synchronous way as well fire and wait until complete and continue. So the key difference is in the use case they serve and objectives of the frameworks.
  • Perform long running background tasks
  • Perform periodic tasks
  • Asynchronous

    • (Example cases)

    Processing Videos/images
    Sending bulk emails

    Both can

  • Channels & Celery can both do asynchronous pooled tasks in common.
  • Channels and Celery both use a backend for messages and worker daemon(s). So the same kind of thing could be implemented with both.

    • Resources:

  • https://www.oddbird.net/2017/04/17/async-notifications/
  • https://stackoverflow.com/questions/38620969/how-are-django-channels-different-than-celery
  • Resource https://stackoverflow.com/questions/63543603/scheduling-tasks-in-django
  • https://stuartm.com/2020/05/django-async/
  • More options https://djangopackages.org/grids/g/workers-queues-tasks/