Python - Multi Processing


Imap

Use imap if you want to load in the arguments as you go (use a generator that will return LOTS of results). Will be slightly slower than map. Set chunksize to a number greater than 1 for lots of speedup.

Imap_Unordered

Use imap_unordered when you would use imap but don't care about the order of the returned results.

Map

Use map when you don't care about all of the arguments being loaded into memory at once.

Progress Bar

from multiprocessing import Pool
from tqdm import tqdm

num_processes = 4
args = [(1, 2), (3, 4), (5, 6)]  # A generator also works. See `chunksize`
results = []

def func(a, b):
    return a + b

with Pool(num_processes) as pool:
    # Use `map` or `imap` instead of `starmap` if `func` only has 1 argument
    for item in tqdm(pool.starmap(func, args), total=len(args)):
        results.append(item)

Another option is p_tqdm

Starmap

Use starmap when a function requires multiple arguments.