How to run asynchronous python callables in parallel?

• By Romain Dorgueil
tips python async parallel tasks threading

Short answer

With python 3.5 and up (or python 3.4 using @asyncio.coroutine decorator), you can start various async functions in parallel, getting the callable results as soon as they’re available, using the following code snippet:

import asyncio


async def my_sleeping_task(name, secs):
    print('Starting {}'.format(name))
    await asyncio.sleep(secs)
    print('Finished {}'.format(name))
    return name


async def main():
    tasks = [
        my_sleeping_task('one', 3),
        my_sleeping_task('two', 2),
        my_sleeping_task('three', 1),
    ]

    while len(tasks):
        done, tasks = await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED)
        for task in done:
            print(task, task.result())


if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    try:
        loop.run_until_complete(main())
    finally:
        loop.close()

Details

Be aware that running asynchronous functions in parallel is not the same as running code in different threads. While different threads will effectively run concurrently, asynchronous tasks run in an event loop will run one at a time, yield on blocking calls to let other tasks perform their duties and resume when on top of the loop again.

This means that only one function will be run at a time, even if you’ll have the actual impression of them being run in parallel.

Here is a simplified schema of the sequence happening:

Simplified sequence diagram of the async process.

Links & references