How to multithread uploads in python?-CodePudding

In python I have a list of files that needs to be uploaded.

An example code might look like this:

        for path, key in upload_list:
            upload_function.upload_file(path, key)

How can I multithread such tasks?

I already came accross multiple solution like process loops, see: How to Multithread functions in Python? but to me these seeme to be kinda overkill just to process a list of files for upload. Is there maybe a smarter way?

CodePudding user response：

It's not that complicated with ThreadPoolExecutor

def upload_files(upload_list):
    def __upload(val):
        upload_function.upload_file(*val)

    with ThreadPoolExecutor() as executor:
        executor.map(__upload, upload_list)

upload_files(upload_list)

If you can modify upload_function.upload_file to receive the key and path as one parameter the function can be simplified to

upload_list = [...]
with ThreadPoolExecutor() as executor:
    executor.map(upload_function.upload_file, upload_list)

executor.map(__upload, upload_list) will run the __upload function with one sublist from upload_list as parameter, and it will run all of them (almost) simultaneity.

To control the number of threads you can use the max_workers parameter

with ThreadPoolExecutor(max_workers=len(upload_list)) as executor: