Tips & Tricks Tips & Tricks How to run asynchronous python callables in parralel?

Short answer

With python 3.5 and up (or python 3.4 using @asyncio.coroutine decorator), you can start various async functions in parralel, getting the callable results as soon as they're available, using the following code snippet:

import asyncio


async def my_sleeping_task(name, secs):
    print('Starting {}'.format(name))
    await asyncio.sleep(secs)
    print('Finished {}'.format(name))
    return name


async def main():
    tasks = [
        my_sleeping_task('one', 3),
        my_sleeping_task('two', 2),
        my_sleeping_task('three', 1),
    ]

    while len(tasks):
        done, tasks = await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED)
        for task in done:
            print(task, task.result())


if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    try:
        loop.run_until_complete(main())
    finally:
        loop.close()

Details

Be aware that running asynchronous functions in parralel is not the same as running code in different threads. While different threads will effectively run concurrently, asynchronous tasks run in an event loop will run one at a time, yield on blocking calls to let other tasks perform their duties and resume when on top of the loop again.

This means that only one function will be run at a time, even if you'll have the actual impression of them being run in parralel.

Here is a simplified schema of the sequence happening:

Simplified sequence diagram of the async process.

Share the love!

Liked this article? Please consider sharing it on your favorite network, it really helps me a lot!

You can also add your valuable insights by commenting below.