asyncio Tasks — Concurrent Execution with gather, Task, and Queue

Building on async def and await, this article covers how to run multiple coroutines concurrently. asyncio.gather for "fire all and await all", asyncio.create_task for "fire now, await later", and asyncio.Queue for producer / consumer — these three cover almost any async pattern you'll write in real projects.

About running code here

async / await is all about timing, but this site's runner buffers print output and shows it all at once after the script finishes. Real-time output and elapsed-time feel won't match a real Python environment. The article uses diagrams to make the internal behavior clear, but if you want to see print flow live or feel the actual timing, run it in your local Python environment with asyncio.run.

asyncio.gather — run multiple coroutines concurrently

"Fire several APIs concurrently and only move on once every response is in" — a textbook async use case. Synchronous code adds up the response times, but gather finishes in the time of the slowest single one.

asyncio.gather(coro1, coro2, ...) starts every passed coroutine at once and returns a list of results once they all finish. The list comes back in the order you passed them in — not completion order — so input and output stay aligned.

import asyncio

async def fetch(name):
    await asyncio.sleep(1)        # Simulate an API call with a 1-sec wait
    return f"{name} done"

# Fire 3 in parallel → all results in 1 sec (vs 3 sec sequential)
results = await asyncio.gather(
    fetch("A"),
    fetch("B"),
    fetch("C"),
)
print(results)                    # ['A done', 'B done', 'C done']  ← input order

What happens on exceptions

By default, if any coroutine inside gather raises, the whole call raises and aborts. To collect exceptions instead, pass asyncio.gather(..., return_exceptions=True) — exception objects then come back as list elements so you can check types after the fact. Useful when you want to hit multiple APIs and tolerate partial failures.

Run 3 coroutines concurrently with asyncio.gather and confirm that completion order can differ but the return value stays in input order. Each task waits a different amount of time so you can watch completion order (B → A → C) vs return order ([A, B, C]) drift apart.

① Add import asyncio.

② Define async def task(name, secs): — await asyncio.sleep(secs) then return f"{name} done".

③ Run with await asyncio.gather(task("A", 0.3), task("B", 0.1), task("C", 0.5)) and store in results.

④ Print print("results:", results).

⑤ Print print("count:", len(results)).

(Run successfully and the explanation will appear.)

create_task and wait_for — fire now, await later, with timeout

asyncio.create_task(coro) wraps a coroutine in a "Task" object and starts it immediately. Unlike gather's "fire all and await all", this is the classic async pattern of "fire now, do other work, then await task to collect the result later".

asyncio.wait_for(awaitable, timeout=N) is a safety net: "raise TimeoutError if it doesn't finish in N seconds". It's standard to combine it with Tasks as a fail-safe when a Web API doesn't respond.

import asyncio

async def slow_api():
    await asyncio.sleep(2)
    return "response"

# Fire it with create_task (the Task starts running right away)
task = asyncio.create_task(slow_api())

# Other work can happen while the task runs in the background
print("task fired, doing other work...")

# Collect the result with await when you need it
result = await task
print(result)                       # response

# wait_for adds a timeout (give up after 1 sec)
try:
    result = await asyncio.wait_for(slow_api(), timeout=1.0)
except asyncio.TimeoutError:
    print("timeout!")               # 2 sec response vs 1 sec budget → here

Three ways to reach "done" — return / exception / cancel

There are 3 paths to done: (1) normal completion — return produced a value, (2) exception — something raised inside, (3) cancel — task.cancel() interrupted it. task.done() returns True for all three paths, and task.exception() extracts the exception if there is one.

Useful Task object methods

The Task object returned by create_task supports useful operations: task.cancel() to interrupt, task.done() to check completion, task.result() to fetch the result of a finished Task (raises if not done), and task.exception() to fetch any raised exception. Handy for controlling long-running background work.

Fire two Tasks with create_task, do other work in between, then collect their results.

① Add import asyncio.

② Define async def task(name): — await asyncio.sleep(0) to switch out, then return f"{name} done".

③ Use asyncio.create_task(task("A")) and asyncio.create_task(task("B")) to fire 2 Tasks, storing them in t1 and t2.

④ While the Tasks run, print tasks fired.

⑤ Use await t1 and await t2 to collect each result and print as A: ◯ / B: ◯.

Use asyncio.wait_for to set a timeout and try both within-budget and over-budget cases.

① Add import asyncio.

② Define async def slow_task(): — await asyncio.sleep(0.5) to wait 0.5 sec, then return "response done".

③ await asyncio.wait_for(slow_task(), timeout=1.0) — finishes inside 1 sec, so it succeeds. Print as success: ◯.

④ try: / except asyncio.TimeoutError: around await asyncio.wait_for(slow_task(), timeout=0.1) — 0.1 sec isn't enough, so TimeoutError fires. Catch it and print timeout!.

asyncio.Queue — producer / consumer

"I want one coroutine to feed values in and another to consume them" — a frequent pattern in scraping, job processing, stream handling, and other situations where two loops with different speeds need to mesh.

asyncio.Queue is an async queue for passing values between coroutines (a FIFO = First In First Out — values come out in the order you put them in). Use await queue.put(value) to insert and await queue.get() to take — and when the queue is empty / full, it automatically switches to another task and waits, keeping things simple.

import asyncio

queue = asyncio.Queue()

# Put values in
await queue.put("item-1")
await queue.put("item-2")

# Get them out (FIFO = first in, first out)
print(await queue.get())            # item-1
print(await queue.get())            # item-2

# get() on an empty queue switches to another task and waits for a value
# print(await queue.get())          # ← pauses here until someone puts

Stop cleanly with a sentinel

On the consumer side, while True: item = await queue.get() waits forever for something to arrive. The pattern is to have the producer push a termination marker at the end (typically None, or a custom sentinel = a dedicated guard object you can tell apart from real data). The consumer breaks out of the loop the moment it sees the marker.

Try put and get inside one coroutine to confirm Queue basics and FIFO (first in, first out).

① Add import asyncio and create an empty asyncio.Queue.

② await queue.put("a"), "b", "c" — insert 3 values in order.

③ Call await queue.get() 3 times and gather the values in a list, then print as pulled: ◯.

Run producer / consumer concurrently with gather

Queue really pays off when multiple coroutines pass values back and forth. Split "feeding" and "consuming" into separate coroutines and run them concurrently with gather — await put / await get act as switching points, and the two halves mesh naturally.

import asyncio

async def producer(queue):
    for i in range(3):
        await queue.put(f"item-{i}")
    await queue.put(None)           # termination marker

async def consumer(queue):
    while True:
        item = await queue.get()
        if item is None:            # break on the termination marker
            break
        print(f"processing: {item}")

queue = asyncio.Queue()
await asyncio.gather(producer(queue), consumer(queue))
# Output:
# processing: item-0
# processing: item-1
# processing: item-2

Build a setup where the producer puts 3 items and the consumer collects them into a list. Use None as the termination marker.

① Add import asyncio, create an empty asyncio.Queue, and an empty list results.

② Define async def producer(queue): — put f"item-{i}" 3 times, then put None as the termination marker.

③ Define async def consumer(queue, results): — while True: calls get; if None then break, otherwise append to results.

④ Run with asyncio.gather(producer(queue), consumer(queue, results)) and print as count: ◯ / first: ◯ / last: ◯.