Add a column during getting data with ThreadPoolExecutor in Python-CodePudding

I want to read different pages from the link below with different numbers using ThreadPoolExecutor and save the related numbers to a dataframe as a new column.

https://booking.snav.it/api/v1/rates/1030/2019-02-25/1042/2019-02-25?lang=1

The numbers change as below:

from concurrent.futures import ThreadPoolExecutor, as_completed
from pandas import json_normalize
import pandas as pd
import requests


def download_file(url):
    url_info = requests.get(url, stream=True)
    jdata = url_info.json()
    return jdata


nums = [1030,1031,1040,1050,1020,1021,1010,1023]
urls= [f"https://booking.snav.it/api/v1/rates/{i}/2019-02-25/1042/2019-02-25?lang=1" for i in nums]
with ThreadPoolExecutor(max_workers=14) as executor:
     for url in urls:
         sleep(0.1)
         processes.append(executor.submit(download_file, url))

for index, task in enumerate(as_completed(processes)):
    jdata = task.result()
    tmp = json_normalize(jdata)
    tmp["num"] = nums[index]
df = df.append(tmp)
print(df.head())

In the code above I have tried to read the data using multi-threading and and the related number to each json response as a new column of df dataframe. But this code does not work, because of using multi-threading the order of nums's numbers are not the same as scraped json responses. What should I do?

CodePudding user response：

Try this:

from concurrent.futures import ThreadPoolExecutor

...

with ThreadPoolExecutor(max_workers=14) as executor:
     rv = executor.map(download_file, urls)

for index, jdata in enumerate(rv):
    tmp = json_normalize(jdata)
    tmp["num"] = nums[index]
    df.append(tmp)

print(df.head())