I have a python code and I want to speed it up using threads but when I try to I get the same lines getting duplicated, is there is any way I could speed it up without getting duplicate lines
code
import requests
import json
f = open("urls.json")
data = json.load(f)
def urls():
for i in data['urls']:
r = requests.get("https://" i)
print(r.headers)
CodePudding user response:
You can use ThreadPoolExecutor
class from concurrent.futures. It is efficient way according to Thread
class.
You can change the max_workers
value according to your task
Here is the piece of code:
import requests
from concurrent.futures import ThreadPoolExecutor
import json
f = open("urls.json")
data = json.load(f)
f.close()
def urls():
urls = ["https://" url for url in data['urls']]
print(urls)
with ThreadPoolExecutor(max_workers=5) as pool:
iterator = pool.map(requests.get,urls)
for response in iterator:
print(response.headers)
print("\n")
CodePudding user response:
Make async
or threaded
calls.
So, you would do something like this:
import aiohttp
import asyncio
import time
start_time = time.time()
async def main():
async with aiohttp.ClientSession() as session:
for number in range(1, 151):
pokemon_url = f'https://pokeapi.co/api/v2/pokemon/{number}'
async with session.get(pokemon_url) as resp:
pokemon = await resp.json()
print(pokemon['name'])
asyncio.run(main())
Could also do multiprocessing as per the comment, but async is better for i/o type tasks.