Multiprocessing #

Threading in Python is limited by the GIL — only one thread can run Python bytecode at a time. For tasks that genuinely need CPU parallelism, such as numeric computation, image processing, or video encoding, multiprocessing is the answer. Each process gets its own interpreter and memory, so the GIL is no longer a barrier. The trade-off: sharing data between processes can’t be done as easily as sharing variables — it requires special mechanisms like Queue, Pipe, and Manager.

Process vs Thread #

Before diving into code, it’s important to understand the fundamental difference between processes and threads, because it affects how you design your program.

Feature	Thread	Process
Memory	Shared	Isolated
GIL	Bound by the GIL	GIL-free
Overhead	Light	Heavier (fork/spawn)
Communication	Plain variables + Lock	Queue / Pipe / Manager
Best for	I/O-bound tasks	CPU-bound tasks
Crash isolation	One thread crashing can affect others	One process crashing doesn’t affect others

To visualize the architectural difference in memory modeling and GIL between Multithreading and Multiprocessing, look at the diagram below:

flowchart TD
    subgraph MultiThreading ["Multithreading (One Process)"]
        P1["Main Process"] ---> T1["Thread 1"]
        P1 ---> T2["Thread 2"]
        T1 & T2 --> SharedMem[("Shared Memory (Heap)")]
    end
    subgraph MultiProcessing ["Multiprocessing (Multi Process)"]
        Parent["Parent Process"] ---> Child1["Child Process 1 (GIL 1)"]
        Parent ---> Child2["Child Process 2 (GIL 2)"]
        Child1 --> Mem1[("Isolated Memory 1")]
        Child2 --> Mem2[("Isolated Memory 2")]
        Child1 <-->|"IPC: Queue / Pipe"| Child2
    end

On Windows, all code that creates processes must be inside the if __name__ == '__main__': block. This prevents child processes from recursively spawning when the module gets re-imported. On Linux/macOS it isn’t mandatory, but it’s still good practice.

Creating Processes #

Just like threading, there are two ways to create a process: using a target function or subclassing multiprocessing.Process.

Way 1: Process with a Function #

import multiprocessing
import time

def compress_file(file_name):
    print(f"[{file_name}] Starting compression...")
    time.sleep(2)  # simulated CPU-bound task
    print(f"[{file_name}] Compression done.")

if __name__ == '__main__':
    p1 = multiprocessing.Process(target=compress_file, args=("video_1.mp4",))
    p2 = multiprocessing.Process(target=compress_file, args=("video_2.mp4",))

    p1.start()
    p2.start()

    p1.join()
    p2.join()

    print("All files compressed.")

Way 2: Process with a Subclass #

Good for processes that need internal state or extra methods.

import multiprocessing
import time

class CompressionProcess(multiprocessing.Process):
    def __init__(self, file_name):
        super().__init__()
        self.file_name = file_name

    def run(self):
        print(f"[{self.file_name}] PID: {self.pid} — starting compression...")
        time.sleep(2)
        print(f"[{self.file_name}] Done.")

if __name__ == '__main__':
    processes = [CompressionProcess(f"video_{i}.mp4") for i in range(3)]
    for p in processes:
        p.start()
    for p in processes:
        p.join()

Process Pools #

Creating processes one by one for many tasks is inefficient because fork/spawn overhead is significant. multiprocessing.Pool manages a set of worker processes ready to accept tasks — like ThreadPoolExecutor but for processes.

import multiprocessing
import time

def compute_square(n):
    time.sleep(0.1)  # simulated computation
    return n * n

if __name__ == '__main__':
    data = list(range(20))

    # A pool with 4 worker processes
    with multiprocessing.Pool(processes=4) as pool:
        results = pool.map(compute_square, data)

    print(results)
    # [0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100, 121, 144, 169, 196, 225, 256, 289, 324, 361]

For more control, use pool.apply_async() to fetch results asynchronously:

import multiprocessing

def process_chunk(chunk):
    return sum(chunk)

if __name__ == '__main__':
    # Split large data into chunks
    data = list(range(1000))
    chunk_size = 100
    chunks = [data[i:i+chunk_size] for i in range(0, len(data), chunk_size)]

    with multiprocessing.Pool(processes=4) as pool:
        futures = [pool.apply_async(process_chunk, (chunk,)) for chunk in chunks]
        results = [f.get() for f in futures]  # fetch each result

    print(f"Total: {sum(results)}")  # 499500

ProcessPoolExecutor from concurrent.futures is a more modern API alternative, consistent with ThreadPoolExecutor:

from concurrent.futures import ProcessPoolExecutor, as_completed

def factorize(n):
    factors = []
    d = 2
    while d * d <= n:
        while n % d == 0:
            factors.append(d)
            n //= d
        d += 1
    if n > 1:
        factors.append(n)
    return factors

if __name__ == '__main__':
    numbers = [112272535095293, 112582705942171, 112272535095293]

    with ProcessPoolExecutor(max_workers=3) as executor:
        futures = {executor.submit(factorize, n): n for n in numbers}
        for future in as_completed(futures):
            n = futures[future]
            print(f"Factors of {n}: {future.result()}")

Inter-Process Communication with Queue #

Because processes don’t share memory, data must be sent explicitly. multiprocessing.Queue is a thread-safe and process-safe data structure ideal for the producer-consumer pattern.

import multiprocessing
import time

def producer(queue, count):
    for i in range(count):
        item = f"task-{i}"
        queue.put(item)
        print(f"Producer: sending {item}")
        time.sleep(0.3)
    queue.put(None)  # sentinel: done signal

def consumer(queue, worker_id):
    while True:
        item = queue.get()
        if item is None:
            queue.put(None)  # forward the sentinel to other consumers
            break
        print(f"Consumer-{worker_id}: processing {item}")
        time.sleep(0.5)

if __name__ == '__main__':
    q = multiprocessing.Queue()

    p_prod = multiprocessing.Process(target=producer, args=(q, 6))
    p_cons1 = multiprocessing.Process(target=consumer, args=(q, 1))
    p_cons2 = multiprocessing.Process(target=consumer, args=(q, 2))

    p_prod.start()
    p_cons1.start()
    p_cons2.start()

    p_prod.join()
    p_cons1.join()
    p_cons2.join()

    print("All tasks processed.")

Don’t use queue.empty() as a stop condition — there’s a race condition between checking empty() and fetching an item. Always use a sentinel value (a special value like None) as the marker that the producer has finished sending data.

Two-Way Communication with Pipe #

multiprocessing.Pipe() creates a pair of connections that can send and receive data to and from each other. Good for point-to-point communication between two processes.

import multiprocessing

def compute_worker(conn):
    data = conn.recv()  # receive data from the parent
    result = [x ** 2 for x in data]
    conn.send(result)   # send the result back
    conn.close()

if __name__ == '__main__':
    parent_conn, child_conn = multiprocessing.Pipe()

    p = multiprocessing.Process(target=compute_worker, args=(child_conn,))
    p.start()

    parent_conn.send(list(range(10)))  # send data to the child
    result = parent_conn.recv()        # wait for and receive the result

    p.join()
    print(f"Squared results: {result}")

The difference between Queue and Pipe:

Queue:
  ✓ Can be accessed by many processes at once
  ✓ Thread-safe and process-safe
  ✓ Good for the producer-consumer pattern with N workers

Pipe:
  ✓ Faster (lower overhead)
  ✓ Good for two-way point-to-point communication
  ✗ Only for two endpoints — not suitable for many processes

Shared State with Manager #

Processes can’t share plain Python variables. If you need several processes to read and write to the same data structure, use multiprocessing.Manager, which manages the objects in a separate server process and makes them accessible via proxies.

import multiprocessing

def collect_results(worker_id, result_list, result_dict, lock):
    data = worker_id * 10  # simulated computation
    with lock:
        result_list.append(data)
        result_dict[f"worker-{worker_id}"] = data

if __name__ == '__main__':
    with multiprocessing.Manager() as manager:
        shared_list = manager.list()
        shared_dict = manager.dict()
        lock = manager.Lock()

        processes = [
            multiprocessing.Process(
                target=collect_results,
                args=(i, shared_list, shared_dict, lock)
            )
            for i in range(5)
        ]

        for p in processes:
            p.start()
        for p in processes:
            p.join()

        print("List results:", sorted(shared_list))
        print("Dict results:", dict(shared_dict))

For simpler needs — a counter or a boolean flag — multiprocessing.Value and multiprocessing.Array are more efficient than Manager because they use direct shared memory:

import multiprocessing

def increment_counter(counter, lock):
    for _ in range(10000):
        with lock:
            counter.value += 1

if __name__ == '__main__':
    counter = multiprocessing.Value('i', 0)  # 'i' = integer
    lock = multiprocessing.Lock()

    processes = [
        multiprocessing.Process(target=increment_counter, args=(counter, lock))
        for _ in range(4)
    ]

    for p in processes:
        p.start()
    for p in processes:
        p.join()

    print(f"Final counter: {counter.value}")  # always 40000

When to Use Multiprocessing vs Alternatives #

Use multiprocessing when:
  ✓ Tasks are CPU-bound: heavy computation, encoding, parsing large data
  ✓ You need true parallelism beyond the GIL
  ✓ Tasks can be split into independent chunks

Consider threading when:
  ✗ Tasks are I/O-bound: HTTP requests, file read/write, DB queries
  ✗ You need easy state sharing (threading is simpler)
  ✗ Process overhead is too large for short tasks

Consider asyncio when:
  ✗ Tasks are I/O-bound with very high concurrency (thousands of tasks)
  ✗ You want a single-threaded event loop without process/thread overhead

Summary #

Multiprocessing overcomes the GIL — every process has its own interpreter, so CPU-bound computation can run with true parallelism.
Processes don’t share memory — data must be sent explicitly via Queue, Pipe, Manager, Value, or Array.
Pool.map() for batch processing — the most concise way to distribute a function over many inputs in parallel.
Queue for producer-consumer — use a sentinel value (None) as the completion marker, not queue.empty().
Pipe for point-to-point communication — faster than Queue but only for two endpoints.
Manager for complex shared objects — lists and dicts accessible by many processes, at the cost of higher overhead.
Value and Array for simple shared memory — more efficient than Manager for primitive data types.
Always use if __name__ == '__main__': — mandatory on Windows, good practice on every platform.

← Previous: Multithreading Next: Context Managers →