What is a task queue?
Taskiq is an asynchronous distributed task queue
for this, we need a broker. Broker is an object that communicate to worker using distributed queues
Redis, RabbitMQ, Kafka are the queues and there are brokers for these queues.
There are 4 terms in it: message, broker, task, queue, worker. Which one works when ?
Does broker add the message in the queue and the worker takes up the message from the queue wheneve it wants? Ans : Client adds the task to the task queue , this task queue is built in the broker (Redis, RabbitMQ). The worker continuously polls the broker and takes up the tasks from the task queue whenever there is a task in it.
If there are 4 workers then each worker can take up the message from the queue and execute it. That is, at a time 4 messages can be executed? Yes, if there are 4 workers then each can take up the task and execute it in parallel.
Does each worker operate asynchronously. Suppose there is only one worker, but the number of messages in the queue is 4. The worker executes the first message, the first message has an I/O operation. As the worker is aynchronous then it will start executing the next task while the first task is doing its I/O operation. is it possible that a single worker takes up all the 4 tasks.
Taskiq will do but celery will not.
Read about GIL Asyncio events
import asyncio
from taskiq_nats import JetStreamBroker
broker = JetStreamBroker(“nats://localhost:4222”, queue=“my_queue2”)
@broker.task async def my_task(a: int, b: int) -> None: print(“AB”, a + b)
async def main(): await broker.startup()
await my_task.kiq(1, 2)
await broker.shutdown()if name == “main”: asyncio.run(main())
In this code, what is broker, queue, task ,message and worker.
queue is Redis. Task is addition operation.
(1,2) is the message which is queue in the Redis queue.
Worker is a process that executes the addition Task.
Taskiq is prefered when the background processing mainly involves API calling.For asynchronous project, Taskiq is prefered.
Celery is prefered when background task is CPU intensive. For fully synchronous project, celery is prefered.
redis commander to view messages in redis