TechOverflow Logo
TechOverflow

How to use concurrent.futures map with a tqdm progress bar

Problem:

You have a concurrent.futures executor, e.g.

import concurrent.futures

executor = concurrent.futures.ThreadPoolExecutor(64)

Using this executor, you want to map a function over an iterable in parallel (e.g. parallel download of HTTP pages).

In order to aid interactive execution, you want to use tqdm to provide a progress bar, showing the fraction of futures

Solution

You can use this function:

from tqdm import tqdm
import concurrent.futures

def tqdm_parallel_map(executor, fn, *iterables, **kwargs):
    """
    Equivalent to executor.map(fn, *iterables),
    but displays a tqdm-based progress bar.
    
    Does not support timeout or chunksize as executor.submit is used internally
    
    **kwargs is passed to tqdm.
    """
    futures_list = []
    for iterable in iterables:
        futures_list += [executor.submit(fn, i) for i in iterable]
    for f in tqdm(concurrent.futures.as_completed(futures_list), total=len(futures_list), **kwargs):
        yield f.result()

Note that internally, executor.submit() is used, not executor.map() because there is no way of calling concurrent.futures.as_completed() on the iterator returned by executor.map().

Usage example if you are not interested in the actual values:

import concurrent.futures
executor = concurrent.futures.ThreadPoolExecutor(64)

# Run my_func with arguments ranging from 1 to 10000, in parallel
for _ in tqdm_parallel_map(executor, lambda i: my_func(i), range(1, 10000)):
    pass

In case you care about the return value of my_func(), use this snippet instead:

import concurrent.futures
executor = concurrent.futures.ThreadPoolExecutor(64)

# Run my_func with arguments ranging from 1 to 10000, in parallel
for result in tqdm_parallel_map(executor, lambda i: my_func(i), range(1, 10000)):
    # result is the return value of my_func().
    # NOTE: result is not neccessarily in the same order as the input Iterable !
    # Whichever parallel execution of my_func() finishes first will be printed first ! 
    print(result)

Note: In constract to executor.map() this function does NOT yield the arguments in the same order as the input.