Python 3.8.3 : A brief introduction to the Celery python package.

The development team tells us:

Celery is a simple, flexible, and reliable distributed system to process vast amounts of messages, while providing operations with the tools required to maintain such a system.

To do that Celery uses tasks.

These task queues are used as a mechanism to distribute work across threads or machines using brokers.

A task a client puts a message on the queue and the broker then delivers the message to a worker.

The install of this python package is simple:

Celery can run on a single machine, on multiple machines, or even across datacenters.

pip3 install -U Celery
Collecting Celery
...
Installing collected packages: pytz, billiard, vine, amqp, kombu, Celery
Successfully installed Celery-4.4.2 amqp-2.5.2 billiard-3.6.3.0 kombu-4.6.8 pytz-2020.1 vine-1.3.0

The Celery library must be instantiated before use like an application.

First, I tested with a simple example application from the official website.

from celery import Celery
app = Celery()

@app.task
def add(x, y): return x + y

if __name__ == '__main__':
    app.worker_main()

When I run it this application the output is this:

python.exe .\example_tasks.py
 
 -------------- celery@DESKTOP-9DSLSMF v4.4.2 (cliffs)
--- ***** ----- 
-- ******* ---- Windows-10-10.0.18362-SP0 2020-05-24 18:30:04
- *** --- * --- 
- ** ---------- [config]
- ** ---------- .> app:         __main__:0x1c697187940
- ** ---------- .> transport:   amqp://guest:**@localhost:5672//
- ** ---------- .> results:     disabled://
- *** --- * --- .> concurrency: 6 (prefork)
                .> celery           exchange=celery(direct) key=celery


[2020-05-24 18:30:06,273: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//:
[WinError 10061] No connection could be made because the target machine actively refused it.
Trying again in 2.00 seconds... (1/100)

[2020-05-24 18:30:10,296: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//:
[WinError 10061] No connection could be made because the target machine actively refused it.
Trying again in 4.00 seconds... (2/100)

[2020-05-24 18:30:16,323: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//:
[WinError 10061] No connection could be made because the target machine actively refused it.
Trying again in 6.00 seconds... (3/100)


worker: Hitting Ctrl+C again will terminate all running tasks!

worker: Warm shutdown (MainProcess)

This error is show because I trying to connect to a local instance of default broker named RabbitMQ .

RabbitMQ is most popular open source message broker and can be download it from the official webpage.

If you want to install RabbitMQ, then you need to install Erlang, see screenshot:

This is not the only one option, you have many features like for example Redis.

The development team try experimental support for a myriad of other solutions, including using SQLite for local development.

A full list with all supported brokers can be found here.

Celery is easy to integrate with web frameworks, some of them even have integration packages:

• Pyramid  with pyramid_celery;
• Pylons with celery-pylons;
• Flask not needed;
• web2py with web2py-celery;
• Tornado with tornado-celery;
• Tryton with celery_tryton;
• and Django also no not needed because works well.
  

Maybe I will make another tutorial about this python module and how works with Redis or Django.