In this blog post I'll try to cover the difference between the
functionality found in the different functions that the
multiprocessing package offers in python. The
package offers the ability to work with multiple threads and if you
wish with multiplce cores. First I'll try to cover the multiple core
scenario and explain the difference between
apply, apply_async. In a later blog post I'll try to
cover the multithreaded scenario. First I'll start with
map operates on function that accepts only one arguement, it is
concurrent, it blocks your code, meaning nothing else can be executed
until its finished and finally it returns your results in an ordered
map consumes your iterable by converting the iterable to a
list assuming it isn't already, breaking it into chunks, and
sending those chunks to the worker processes in the
Pool. Breaking the
iterable into chunks performs better than passing each item in the
iterable between processes one item at a time - particularly if the
iterable is large. However, turning the iterable into a list in order
to chunk it can have a very high memory cost, since the entire list
will need to be kept in memory.
map_async on the other hand can operate on a function
accepting multiple arguements it is not concurrent, it allows excution
of code while it is still running, and finally it doesn't return your
results in an ordered manner.
imap doesn't turn the iterable you give it into a list, nor does break
it into chunks by default. It will iterate over the iterable one
element at a time, and send them each to a worker process. This means
you don't take the memory hit of converting the whole iterable to a
list, but it also means the performance is slower for large iterables,
because of the lack of chunking. This can be mitigated by passing a
chunksize argument larger than default of 1, however.
The other major difference between
map/map_async, is that with
imap/imap_unordered, you can start
receiving results from workers as soon as they're ready, rather than
having to wait for all of them to be finished. With
AsyncResult is returned right away, but you can't actually retrieve
results from that object until all of them have been processed, at
which points it returns the same list that
map does, basically
is actually implemented internally as
no way to get partial results, you either have the entire result, or
imap_unordered both return iterables right away. With
the results will be yielded from the iterable as soon as they're
ready, while still preserving the ordering of the input iterable. With
imap_unordered, results will be yielded as soon as they're ready,
regardless of the order of the input iterable.
To summarize the key differences between
- 1. The way they consume the iterable you pass to them.
- 2. The way they return the result back to you.
apply sends a single task off to a worker process, and then blocks
until it's complete.
apply_async sends a single task off to a work
process, and then immediately returns an AsyncResult object, which can
be used to wait for the task to finish and retrieve the
apply is implemented by simply calling
apply_async, if an exception occurs
inside of your calling function, you won't know about it unless you
Pool['process_x'].get() on the failing
object, which would require iterating over all of
map_async the exception will be raised if you call
~async_result.get() - no iteration required.
built-in chunking functionality, which will make your code perform
noticeably better if arguement list is very large.
Here is a final table summarizing all of those properties and diffferencies: