How do I get Python to send as many concurrent HTTP requests as possible?-CodePudding

I'm trying to send HTTPS requests as quickly as possible. I know this would have to be concurrent requests due to my goal being 150 to 500 requests a second. I've searched everywhere, but get no Python 3.11 answer or one that doesn't give me errors. I'm trying to avoid AIOHTTP as the rigmarole of setting it up was a pain, which didn't even work.

The input should be an array or URLs and the output an array of the html string.

CodePudding user response：

Hope this helps, this question asked What is the fastest way to send 10000 http requests

I observed 15000 requests in 10s, using wireshark to trap on localhost and saved packets to CSV, only counted packets that had GET in them.

FILE: a.py

from treq import get
from twisted.internet import reactor

def done(response):
   if response.code == 200:
       get("http://localhost:3000").addCallback(done)

get("http://localhost:3000").addCallback(done)

reactor.callLater(10, reactor.stop)
reactor.run()

Run test like this:

pip3 install treq
python3 a.py  # code from above

Setup test website like this, mine was on port 3000

mkdir myapp
cd myapp
npm init
npm install express
node app.js

FILE: app.js

const express = require('express')
const app = express()
const port = 3000

app.get('/', (req, res) => {
  res.send('Hello World!')
})

app.listen(port, () => {
  console.log(`Example app listening on port ${port}`)
})

OUTPUT

grep GET wireshark.csv  | head
"5","0.000418","::1","::1","HTTP","139","GET / HTTP/1.1 "
"13","0.002334","::1","::1","HTTP","139","GET / HTTP/1.1 "
"17","0.003236","::1","::1","HTTP","139","GET / HTTP/1.1 "
"21","0.004018","::1","::1","HTTP","139","GET / HTTP/1.1 "
"25","0.004803","::1","::1","HTTP","139","GET / HTTP/1.1 "

grep GET wireshark.csv  | tail
"62145","9.994184","::1","::1","HTTP","139","GET / HTTP/1.1 "
"62149","9.995102","::1","::1","HTTP","139","GET / HTTP/1.1 "
"62153","9.995860","::1","::1","HTTP","139","GET / HTTP/1.1 "
"62157","9.996616","::1","::1","HTTP","139","GET / HTTP/1.1 "
"62161","9.997307","::1","::1","HTTP","139","GET / HTTP/1.1 "

CodePudding user response：

This sort of works:

import time
import requests
import concurrent.futures

t0= int(time.time())

urls = []
for y in range(2000):urls.append("https://example.com?n=" str(y))

def send(pageUrl):
    response = requests.get(url=pageUrl)
    return pageUrl   " ~ "   str(response.status_code)

with concurrent.futures.ThreadPoolExecutor() as executor:
    futures = []
    for url in urls:
        futures.append(executor.submit(send, pageUrl=url))
    for future in concurrent.futures.as_completed(futures):
        print(future.result(), end="\r")
        
t1 = int(time.time())
print("\n")
print(len(urls)/(t1 - t0))

Output:

https://example.com?n=1994 ~ 200
166.66