Use imap if you want to load in the arguments as you go (use a generator that will return LOTS of results). Will be slightly slower than map
. Set chunksize
to a number greater than 1
for lots of speedup.
Use imap_unordered when you would use imap
but don't care about the order of the returned results.
Use map when you don't care about all of the arguments being loaded into memory at once.
from multiprocessing import Pool
from tqdm import tqdm
num_processes = 4
args = [(1, 2), (3, 4), (5, 6)] # A generator also works. See `chunksize`
results = []
def func(a, b):
return a + b
with Pool(num_processes) as pool:
# Use `map` or `imap` instead of `starmap` if `func` only has 1 argument
for item in tqdm(pool.starmap(func, args), total=len(args)):
results.append(item)
Another option is p_tqdm
Use starmap when a function requires multiple arguments.