Threading per line in text file-CodePudding

I'm trying to speed up my program by using threading. The program iterates lines in a text file and then sends an api request using that specific line infinitely to create accounts upon availability. How do I make it to send an api request, per line in a text file, across 500 threads?

CodePudding user response：

Please find example how you can use do what you asked for. The pattern called "producer-consumers pattern"

from threading import Thread
from queue import Queue
from time import sleep
from random import randint

WORKERS_NUM = 4

FILE = "file.txt"  # create file in same dir as main.py file (this file script)


class Worker(Thread):
    def __init__(self, q: Queue):
        # daemon = True means that if main thread is finished, all other threads do not prevent
        # process from finishing it's work
        super().__init__(daemon=True)  # call parent __init__
        self.q: Queue = q  # use queue to get lines from main thread

    def send_req_api(self, line: str) -> None:
        """Imagine the function process line and send req to api"""
        sleep(randint(1, 4))
        print(line.strip())

    def run(self):
        """We should define run when we inherit form Thread"""
        while True:
            item = self.q.get()  # get line from queue
            try:
                self.send_req_api(line=item)
            except Exception as ex:
                print(ex)
            finally:
                # confirm that message was processed
                # queue has internal task counter
                # otherwise main thread will not stop
                self.q.task_done()


if __name__ == '__main__':
    print("Started app")
    que = Queue(maxsize=6)  # max 6 messages inside queue simultaneously
    workers = [Worker(q=que) for _ in range(WORKERS_NUM)]  # create workers
    [w.start() for w in workers]  # run workers
    with open(file=FILE, mode="r", encoding="utf8") as f:
        for i in f:
            que.put(i)  # send line to workers
    que.join()  # when all tasks are done que will be unblocked and app will be finished
    print("All Done !")

CodePudding user response：

You can import the threading library and use the start() method

import threading
import time
def send_api_request(line):
    # simulate delay
    time.sleep(1)
    print("threaad.. "   line)


lines = [
    "file_line_1",
    "file_line_2",
    "file_line_3",
    "file_line_4",
    "file_line_5",
]
for line in lines:
    x = threading.Thread(target=send_api_request, args=(line,))
    x.start()

print("end")