How to use same proxy until specific response status?-CodePudding

I have currently written a code that has multiple threads (as example I used 50 threads) and for each thread only one proxy is allowed to be in one of these threads (meaning that 1 proxy cannot be in two threads).

import contextlib
import random
import threading
import time

import requests

my_proxies = [
    'http://140.99.107.100:2100',
    'http://140.99.107.101:2100',
    'http://140.99.107.102:2100',
    'http://140.99.107.103:2100',
    'http://140.99.107.104:2100',
    'http://140.99.107.105:2100',
    'http://140.99.107.106:2100',
    'http://140.99.107.107:2100',
    'http://140.99.107.108:2100',
    'http://140.99.107.109:2100',
    'http://140.99.107.110:2100',
    'http://140.99.107.111:2100',
    'http://140.99.107.112:2100',
    'http://140.99.107.113:2100',
    'http://140.99.107.114:2100',
    'http://140.99.107.115:2100',
    'http://140.99.107.116:2100',
    'http://140.99.107.117:2100',
    'http://140.99.107.118:2100',
    'http://140.99.107.119:2100',
    'http://140.99.107.120:2100',
    'http://140.99.107.121:2100',
    'http://140.99.107.122:2100',
]


# --------------------------------------------------------------------------- #

class AvailableProxiesManager:
    _proxy_lock: threading.Lock = threading.Lock()

    def __init__(self):
        self._proxy_dict = dict.fromkeys(my_proxies, True)

    @property
    @contextlib.contextmanager
    def proxies(self):
        """
        Context manager that yields a random proxy from the list of available proxies.

        :return: dict[str, str] - A random proxy.
        """
        proxy = None

        with self._proxy_lock:
            while not proxy:
                if available := [att for att, value in self._proxy_dict.items() if value]:
                    proxy = random.choice(available)
                    self._proxy_dict[proxy] = False
                else:
                    print('Waiting ... no proxies available')
                    time.sleep(.2)

        yield proxy
        self._proxy_dict[proxy] = True  # Return the proxy to the list of available proxies


# --------------------------------------------------------------------------- #

available_proxies = AvailableProxiesManager()


def main():
    while True:
        with available_proxies.proxies as proxy:
            response = requests.get('https://httpbin.org/ip', proxies={'https': proxy})

            if response.status_code == 403:
                print('Lets put proxy on cooldown for 10 minutes and try with new one!')

        time.sleep(120)


if __name__ == '__main__':
    threads = []
    for i in range(50):
        t = threading.Thread(target=main)
        threads.append(t)
        t.start()
        time.sleep(1)

However my problem is that currently for every while True that is going on, it uses a new random proxy and instead what I am trying to achieve is that I want the same proxy to be used in the same thread until the response status is 403. That means that in the beginning if thread-1 gets the proxy: http://140.99.107.100:2100 then it should be used in thread-1 until it gets 403.

My question is, how can I be able to make the same proxy to be used until it hits response 403?

Expect:

Proxy to be the same until 403

Actual:

New proxy for every GET requests

CodePudding user response：

What if you stop using a context manager, (remove @contextlib.contextmanager) and do something like this:

def main():
    proxy = next(available_proxies.proxies)
    while True:
        response = requests.get('https://httpbin.org/ip', proxies={'https': proxy})
        if response.status_code == 403:
            proxy = next(available_proxies.proxies)

        time.sleep(120)

Hope that helps, good luck !