Send lot of requests to several equipments at the same time-CodePudding

I’m currently working on a project and I need to fetch data from several switches by sending SSH requests as follow:

Switch 1 -> 100 requests
Switch 2 -> 500 requests
Switch 3 -> 1000 requests
…
Switch 70 -> 250 requests

So several requests (5500 in total) spread over 70 switches.

Today, I am using a json file built like this:


{
  "ip_address1": 
  [
    {"command":"command1"},
    {"command":"command2"},
    ...
    {"command":"command100"}
    
  ],

"ip_address2": 
  [
    {"command":"command1"},
    {"command":"command2"},
    ...
    {"command":"command100"}
    
  ],

…

"ip_address70": 
  [
    {"command":"command1"},
    {"command":"command2"},
    ...
    {"command":"command100"}
    
  ],
}

Each command is a CLI command to a switch which I’m connecting on by SSH.

Today, I’m using Python with multi threading with 8 workers because I have only 4 CPUs.

The total of the script make 1 hour to proceed so it’s too long.

Is there a way to drastically speed up this process please? A friend told me about Golang channels and go routines but I’m not sure if it’s interesting to move from Python to Go if there’s no difference about the time.

Can you please give me some advices?

Thank you very much,

CodePudding user response：

Python offers a pretty straight forward multiprocessing library. Especially for a straight forward task like yours I would stick to the language I am the most comfortable with.

In python you would basically generate a list from your list of commands and ip addresses. Using an example straight from the documentation: https://docs.python.org/3/library/multiprocessing.html#module-multiprocessing.pool

With the pool.map function from the multiprocessing module, you can pass each element from your list to a function, where you can pass your commands to the servers. You might want to have another look at the different mapping functions provided for the pool module.

from multiprocessing import Pool, TimeoutError
import os

def execute_ssh(address_command_mapping):
    # add your logic to pass the commands to the corresponding IP address
    return

if __name__ == '__main__':
    # assuming your ip addresses are stored in a json file
    with open("ip_addresses.json", "r") as file:
        ip_addresses = json.load(file)

    # transforming the address dictionary to a list of dictionaries
    address_list = [{ip: commands} for ip, commands in ip_addresses.items()]

    # start 4 worker processes
    with Pool(processes=4) as pool:
        # pool.map passes each element to the 'execute_ssh' function
        pool.map(execute_ssh, address_list)

CodePudding user response：

Thank you Leon,

Is the pool.map function working the same way as the thread pool executor module?

Here is what I’m using:

from concurrent.futures import ThreadPoolExecutor

def task(n):
 // sending command

def main():
 print("Starting ThreadPoolExecutor")
 with ThreadPoolExecutor(max_workers=3) as executor:
   for element in mylist:
      executor.submit(task, (element))
 print("All tasks complete")

if __name__ == '__main__':
 main()

So is it working the same way? Thank you