Parallelism and Concurrency

Concurrency means work can overlap: several operations have started, and each can make progress before the others finish. Parallelism means execution happens at the same time on separate execution resources. A server can handle many waiting requests concurrently on one event-loop thread. Computing two image transformations at once requires resources that can actually execute them together. Neither arrangement guarantees a shorter completion time.

The distinction matters when choosing how to spend capacity. Waiting for a database response, consuming CPU, holding a connection, and waiting for a worker are different states. Increasing concurrency helps only when the system can turn the extra overlap into useful work.

Overlap waiting; parallelize computation

An event loop runs ready tasks on its thread. When a task suspends while waiting for I/O, another task can run. A CPU loop or blocking library call on that thread prevents those tasks from progressing. Writing async def around such a call does not change its behavior. Even await is not an unconditional handoff: an already available result may let execution continue immediately.

Download the complete runnable fixture. It requires native Python 3.11+ with its standard library, permission to start threads and child processes, and no packages or network services. Extract it and run python3 test_lab.py from its directory. Save examples as scripts rather than pasting the process-pool example into a notebook. The fixture includes five scripts and failure/boundary checks; the excerpts below show their central mechanisms.

First, prove overlap independently of wall-clock speed. In overlap.py, events represent an external source completing two reads. Both operations must start before either may return:

import asyncio

async def main():
    release = asyncio.Event()
    started = [asyncio.Event(), asyncio.Event()]

    async def read(index):
        started[index].set()
        await release.wait()
        return index

    async with asyncio.TaskGroup() as group:
        tasks = [group.create_task(read(i)) for i in range(2)]
        await asyncio.gather(*(event.wait() for event in started))
        assert all(not task.done() for task in tasks)
        release.set()
    print([task.result() for task in tasks])  # [0, 1]

asyncio.run(main())

This models waiting; it does not issue real network requests. The result order comes from the task list, independently of completion order. Replacing task creation with a sequential await read(0) would prevent the coordinator from reaching release.set().

Threads are execution paths scheduled by the operating system and share their process's memory. They are useful for integrating blocking libraries without blocking an event loop, provided the library supports concurrent use. Processes give workers separate memory and an explicit input/output boundary. They can execute CPU work in parallel when CPU capacity is available, but starting workers and transferring values have costs.

In GIL-enabled CPython, threads do not execute Python bytecode in parallel; some extension operations release the GIL. Free-threaded builds change that constraint, so check the interpreter and libraries rather than applying a universal Python rule. The GIL never protects an entire application-level update for you. Similarly, JavaScript promises do not move a CPU loop off the browser's UI thread; workers provide a separate execution environment.

cpu_pool.py sends byte payloads to processes. The small workload makes the transfer boundary visible, not a convincing benchmark:

from concurrent.futures import ProcessPoolExecutor
from multiprocessing import get_context

def checksum(payload):
    return sum(byte * byte for byte in payload)

if __name__ == "__main__":
    payloads = [b"alpha" * 1000, b"beta" * 1000]
    with ProcessPoolExecutor(
        max_workers=2, mp_context=get_context("spawn")
    ) as pool:
        actual = list(pool.map(checksum, payloads))
    assert actual == [checksum(value) for value in payloads]
    print(actual)  # [53842000, 42670000]

Workers import the script, which is why pool creation belongs under the entry-point guard. Arguments and results must be serializable. The full fixture also verifies that changing a worker's dictionary does not update the parent's dictionary and that retrieving a failed job raises its exception. It asserts equal values, never speedup or simultaneous scheduling.

Shared state needs an atomic operation

A race exists when correctness depends on an uncontrolled interleaving. Suppose two workers each read a counter, calculate its next value, then write it back. Safe individual reads and writes do not make that whole operation atomic. Both workers can read zero and write one, losing an increment.

shared_state.py deliberately forces that interleaving with a barrier. The repaired version holds a lock across the complete read-modify-write:

from threading import Barrier, Lock, Thread

def run(protected):
    state, lock, together = {"count": 0}, Lock(), Barrier(2)

    def update():
        if protected:
            together.wait()
            with lock:
                state["count"] = state["count"] + 1
        else:
            old = state["count"]
            together.wait()  # both workers have read zero
            state["count"] = old + 1

    workers = [Thread(target=update) for _ in range(2)]
    for worker in workers:
        worker.start()
    for worker in workers:
        worker.join()
    return state["count"]

print(run(False), run(True))  # 1 2

The barrier belongs outside the corrected critical section. Inside it, one thread would hold the lock while waiting for the other, which cannot acquire that lock to reach the barrier. Every access participating in the invariant must follow the same locking rule. Consistent lock ordering helps avoid cycles when several locks are necessary.

Async tasks can also race without executing in parallel: a task can read state, suspend, and resume after another task changes it. Prefer one owner for mutable state or protect the whole invariant with the appropriate synchronization primitive. Locks do not roll back partially completed changes. Holding one across slow I/O can also turn a small invariant into a system-wide waiting point.

More workers can mean more waiting

CPU utilization alone does not identify a bottleneck. Low CPU can accompany a saturated database connection pool. Inspect active work, pool wait, queue depth and age, dependency duration, and useful completions per second. Respect CPU quotas, memory, connection limits, and downstream capacity. A container's visible host CPUs do not establish its usable CPU budget. Replicating a service also multiplies its potential connection demand.

There are diminishing returns even without overload: if half a job must run serially, making only the other half infinitely fast cannot improve total time beyond twofold. Extra workers can add scheduling overhead, lock contention, and memory pressure.

Bound active work and waiting work separately. A semaphore can limit operations in flight while leaving an unlimited number of waiting tasks in memory. A worker pool can likewise accept more submissions than it can finish. Backpressure means the capacity signal reaches the producer, which slows down or rejects work.

In bounded_queue.py, two workers and two queue slots admit four items while processing is blocked. The fifth admission must wait:

import asyncio

async def main():
    queue = asyncio.Queue(maxsize=2)
    release, fifth_attempt = asyncio.Event(), asyncio.Event()
    accepted, completed = [], []

    async def produce():
        for item in range(6):
            if item == 4:
                fifth_attempt.set()
            await queue.put(item)
            accepted.append(item)

    async def work():
        while True:
            item = await queue.get()
            try:
                if item is None:
                    return
                await release.wait()
                completed.append(item)
            finally:
                queue.task_done()

    async with asyncio.TaskGroup() as group:
        workers = [group.create_task(work()) for _ in range(2)]
        producer = group.create_task(produce())
        await fifth_attempt.wait()
        assert accepted == [0, 1, 2, 3] and queue.qsize() == 2
        release.set()
        await producer
        await queue.join()
        for _ in workers:
            await queue.put(None)
    assert sorted(completed) == list(range(6))
    print("completed", len(completed))  # completed 6

asyncio.run(main())

The producer has one additional item pending admission; it does not create a task for every input. Real inputs should stream and payload sizes need limits too: a queue bounds item count, not bytes. FIFO dequeue order does not promise completion order across workers.

queue.join() follows unfinished-item accounting. It proves successful processing here because the workers append results before marking each item done. In failure handling, task_done() alone is no success record. The fixture separately injects a worker exception, verifies that the task group cancels the blocked producer and sibling, and shows accepted work remaining in the queue. It does not implement durable recovery; delivery and retry guarantees belong in Queues and Retries.

Completion includes cleanup

A parent needs to own its children, observe their exceptions, and decide when admission stops. A task group gives related async tasks a shared lifetime: a child failure cancels its siblings and the group waits for them before propagating failures. Detached tasks need an explicit owner too.

Cancellation is a request that the task encounters cooperatively. In cancellation.py, cleanup has not happened immediately after cancel(); awaiting the task observes its completion:

import asyncio

async def main():
    entered, log = asyncio.Event(), []

    async def worker():
        try:
            entered.set()
            await asyncio.Event().wait()
        finally:
            log.append("released")

    task = asyncio.create_task(worker())
    await entered.wait()
    task.cancel()
    assert log == []
    try:
        await task
    except asyncio.CancelledError:
        pass
    assert task.cancelled() and log == ["released"]
    print(log)  # ['released']

asyncio.run(main())

Release resources in finally or context managers; normally let cancellation propagate after cleanup. CPU code that never yields can prevent async cancellation from arriving.

Timeout behavior depends on the API. An executor future's result(timeout=...) ends the wait without stopping its job. Cancelling an already running executor future cannot kill the worker thread. The fixture holds a real thread at an event, checks both boundaries, then explicitly releases and joins it. Async timeout helpers may request cancellation, which still requires cooperative progress and cleanup.

A shutdown sequence therefore stops new admission, drains useful work within a budget, requests cancellation where supported, awaits owned children, then closes resources. Forced process termination can interrupt cleanup. None of these actions reverses a remote write already committed: execution lifetime and business outcome remain separate questions.