Django, Celery, and Flower: Creating and Managing Asynchronous Tasks

Introduction to Asynchronous tasks in Django

If you are reading this, chances are you're familiar with the Django framework.  If not feel free to check out some of our Beginner's Guide to Django Web Apps.  In this post, we discuss how to set up asynchronous background tasks using Celery, Redis, and a monitoring tool called Flower using the Django framework.  Asynchronous tasks signify that multiple tasks are processed in parallel whereas synchronous tasks are only executed one by one.  Celery is a task queue written in Python that allows work to be distributed amongst workers, thus enabling tasks to be executed asynchronously.  However, Celery requires a message broker that acts as an intermediary between the Django application and the Celery task queue.  For this tutorial, we will use Redis as our message broker.  Lastly, to view our workers and see the tasks we complete, we will use Flower.  Below we list a summary of the various features and official websites for each piece of software.  

Celery Features

• High availability
• Easy to use and maintain
• Fast - millions of tasks can be processed in a minuted
• Supports multiple brokers: RabbitMQ, Redis, Amazon SQS, and more
• Schedule tasks using the Python datetime module
• Official Docs: https://docs.celeryproject.org/en/stable/index.html

Redis Features

• Supports a wide range of data structures: strings, hashes, lists, sets, bitmaps, geospatial indexes, and many more
• Can be used as a database, cache, and message broker
• Can store 512 MB in a single key and value
• Pub/sub messaging system for high-performance messaging 
• High read and write speed
• Interactive tutorial to understand how Redis stores data: https://try.redis.io/
• Official Docs: https://redis.io/documentation

Flower Features

• Real-time monitoring with task progress, start time, arguments, and statistics
• Control Celery workers remotely - restart, revoke, or terminate worker instances
• Monitor broker
• Simple and straightforward dashboard
• Official Docs: https://flower.readthedocs.io/en/latest/

Flower Preview:

Django Celery Redis Tutorial:

For this tutorial, we will simply be creating a background task that takes in an argument and prints a string containing the argument when the task is executed.  Of course, background tasks have many other use cases, such as sending emails, converting images to smaller thumbnails, and scheduling periodic tasks.

• Create a virtual environment, activate the environment, install Django, and set up a project (Note: the following commands are for Windows 10)

• C:\Users\Owner\Desktop\code>py -m venv celery-demo
  
  C:\Users\Owner\Desktop\code>cd celery-demo
  
  C:\Users\Owner\Desktop\code\celery-demo>ls
  Include  Lib  Scripts  pyvenv.cfg
  
  C:\Users\Owner\Desktop\code\celery-demo>Scripts\activate
  
  (celery-demo) C:\Users\Owner\Desktop\code\celery-demo>pip install django
  
  (celery-demo) C:\Users\Owner\Desktop\code\celery-demo>django-admin startproject mysite

• Change the directory to mysite and install Celery with the Redis broker.  Obviously, you can use a different broker besides Redis here

• (celery-demo) C:\Users\Owner\Desktop\code\celery-demo\mysite>pip install "celery[redis]"

• In the mysite/mysite folder (with settings.py) create a new file called celery.py
• First, we need to define the Celery instance.  In celery.py add the following code:

• import os
  
  from celery import Celery
  
  # set the default Django settings module for the 'celery' program.
  os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'mysite.settings')
  
  app = Celery('mysite', backend='redis', broker='redis://localhost:6379')
  
  # Using a string here means the worker doesn't have to serialize
  # the configuration object to child processes.
  # - namespace='CELERY' means all celery-related configuration keys
  #   should have a `CELERY_` prefix.
  app.config_from_object('django.conf:settings', namespace='CELERY')
  
  # Load task modules from all registered Django app configs.
  app.autodiscover_tasks()
  
  
  @app.task(bind=True)
  def debug_task(self):
      print('Request: {0!r}'.format(self.request))

• Next, we need Celery to run when Django starts.  In mysite/mysite/init.py add the following:

• from __future__ import absolute_import, unicode_literals
  
  # This will make sure the app is always imported when
  # Django starts so that shared_task will use this app.
  from .celery import app as celery_app
  
  __all__ = ('celery_app',)

• In mysite/mysite/settings.py add the following:

• BROKER_URL = 'redis://localhost:6379'
  CELERY_RESULT_BACKEND = 'redis://localhost:6379'
  CELERY_ACCEPT_CONTENT = ['application/json']
  CELERY_TASK_SERIALIZER = 'json'
  CELERY_RESULT_SERIALIZER = 'json'
  
  INSTALLED_APPS = [
      # others app
      'celery',
  
  ]
  
  

• Now we need to add a task.  To do so, we'll create a Django app then create a new file in the app called tasks.py.  The previous function app.autodiscover_tasks() in celery.py searches for "tasks.py" in all the apps we create, in which case, you can create multiple apps that hold different tasks as long as they are stored in a tasks.py file.

• (celery-demo) C:\Users\Owner\Desktop\code\celery-demo\mysite>py manage.py startapp main

• Notice all tasks must use the @shared_task tag. Create tasks.py in the main app and add the following task:

  • #mysite/main/tasks.py
    
    from __future__ import absolute_import
    from celery import shared_task
    
    
    @shared_task
    def test(param):
        return 'The tasks executed with the following parameter: "%s" '

• Create urls.py in the main app:

  • #mysite/main/urls.py
    
    from django.contrib import admin
    from django.urls import path, include
    from . import views
    
    urlpatterns = [
        path('', views.home ),
    ]
    

• Add a view to connect the newly added task to a slug.  Notice how we add delay when we call the task to execute the function in the background.

  • #mysite/main/views.py
    
    from django.shortcuts import render
    from django.http import HttpResponse
    from .tasks import test
    
    # Create your views here.
    def home(request):
    	test.delay(15)
    	return HttpResponse("Hey there!")

• Time to install Redis:
  
  • On Windows, since Redis does not have an official release, visit https://github.com/microsoftarchive/redis/releases and download the latest release.
    • Extract the zipped file to the mysite project folder. 
    • In command prompt, run redis-server.exe
    • In a new command prompt, go to the project directory and run redis-client.exe
    • Type in "ping" and you should receive a "PONG" response
    • Close command prompt
  • On Mac, feel free to use Homebrew and enter the following commands in the mysite directory
    • brew install redis
    • Run redis-server in terminal
    • In a new terminal, go to the project directory and run redis-client
    • Type in "ping" and you should receive a "PONG" response
• Keep your Redis server running in one terminal/command prompt, in the other, stop running redis-client and enter the following command to install flower
  • pip install flower
• Now, make sure to have 4 terminals/command prompts for the following commands:
  • 1) To run the redis server 
  • 2) To run: py/python3 manage.py runserver
  • 3) To run celery: celery -A mysite worker -l info --pool=solo

  • 4) To run flower: flower -A mysite
• After running flower, you will see an address to view the dashboard, in this case, at port 5555

  • (celery-demo) C:\Users\Owner\Desktop\code\celery-demo\mysite>flower -A mysite
    [I 200914 08:03:54 command:140] Visit me at http://localhost:5555
    [I 200914 08:03:54 command:145] Broker: redis://localhost:6379//
• Visit http://127.0.0.1:8000/ to run your task and then view the Flower dashboard again
• If the dashboard shows that the task succeeded, then you should be able to view the executed task in the terminal/command prompt that was running celery:

  • [2020-09-21 12:12:23,328: INFO/MainProcess] Connected to redis://localhost:6379//
    [2020-09-21 12:12:23,338: INFO/MainProcess] mingle: searching for neighbors
    [2020-09-21 12:12:24,367: INFO/MainProcess] mingle: all alone
    [2020-09-21 12:12:24,376: WARNING/MainProcess] c:\users\owner\desktop\code\celery-demo\lib\site-packages\celery\fixups\django.py:206: UserWarning: Using settings.DEBUG leads to a memory
                leak, never use this setting in production environments!
      leak, never use this setting in production environments!''')
    [2020-09-21 12:12:24,377: INFO/MainProcess] celery@DESKTOP-9ERS2HA ready.
    [2020-09-21 12:12:37,121: INFO/MainProcess] Events of group {task} enabled by remote.
    [2020-09-21 12:12:40,825: INFO/MainProcess] Received task: main.tasks.test[4b88a8e4-9efa-4d7c-82ff-bfffa5969c4c]
    [2020-09-21 12:12:40,829: INFO/MainProcess] Task main.tasks.test[4b88a8e4-9efa-4d7c-82ff-bfffa5969c4c] succeeded in 0.0s: 'The tasks executed with the following parameter:15'

Conclusion: Benefits of Asynchronous Tasks

While this was a rather simple tutorial on scheduling background/asynchronous tasks, there are plenty of opportunities moving forwards.  Think about Twitter or Instagram and how users rarely need to reload a given page.  Instead, these apps "feel alive" as likes, retweets, and shares update nearly in real-time.  Asynchronous tasks allow applications to provide smoother user experiences since users can constantly engage with updated content.