Call 4 methods at once in Python 3-CodePudding

I want to call 4 methods at once so they run parallel-ly in Python. These methods make HTTP calls and do some basic operation like verify response. I want to call them at once so the time taken will be less. Say each method takes ~20min to run, I want all 4methods to return response in 20min and not 20*4 80min

It is important to note that the 4methods I'm trying to run in parallel are async functions. When I tried using ThreadPoolExecutor to run the 4methods in parallel I didn't see much difference in time taken.

Looking for suggestions

CodePudding user response：

You can use from concurrent.futures import ThreadPoolExecutor

from concurrent.futures import ThreadPoolExecutor

def foo_1():
    print("foo_1")

def foo_2():
    print("foo_2")

def foo_3():
    print("foo_3")

def foo_4():
    print("foo_4")

with ThreadPoolExecutor() as executor:
  for foo in [foo_1,foo_2,foo_3,foo_4]:
    executor.submit(foo)

CodePudding user response：

One way to run multiple methods in parallel in Python is to use the concurrent.futures library. This library provides a ThreadPoolExecutor class that you can use to run multiple methods concurrently using threads.

Here's an example of how you can use the ThreadPoolExecutor to run four methods in parallel:

import concurrent.futures

def method1():
    # HTTP call and operation

def method2():
    # HTTP call and operation

def method3():
    # HTTP call and operation

def method4():
    # HTTP call and operation

with concurrent.futures.ThreadPoolExecutor() as executor:
    method1_future = executor.submit(method1)
    method2_future = executor.submit(method2)
    method3_future = executor.submit(method3)
    method4_future = executor.submit(method4)

    # Wait for all methods to complete
    concurrent.futures.wait([method1_future, method2_future,method3_future, method4_future])

This will run method1(), method2(), method3(), and method4() in parallel using threads, and the program will wait for all four methods to complete before exiting.

Another option is to use asyncio library and async/await paradigm, which is built on top of threading and uses a single thread for all async tasks.

import asyncio

async def method1():
# HTTP call and operation
# method2,method3,...

await asyncio.gather(method1(), method2(), method3(), method4())

This will run all 4 methods concurrently and return the result of all 4 methods once they are done.

CodePudding user response：

you can use of "multiprocessing" in paython. it's so simple

from multiprocessing import Pool
pool = Pool()
result1 = pool.apply_async(solve1, [A])    # evaluate "solve1(A)" 
result2 = pool.apply_async(solve2, [B])    # evaluate "solve2(B)" 
answer1 = result1.get(timeout=10)
answer2 = result2.get(timeout=10)

you can see full details