I'm trying to optimize a script that connects to some routers based on a list that I provide with IPs.

however, it's taking a quite long time to finish the configuration once starts the configuration once when it finishes the previous router, my question is, would be possible to create a function that doesn't need to wait for it, maybe a multithreading function that can configure a few elements simultaneously.

def configure_routers():

    hosts = create_vrdc_dic()  # -> it's the dictonary that pass the variables

    for i in hosts .keys():
        hostname = hosts [i].get('hostname')
        mgmt_ip = hosts [i].get('mgmt_ip')

        console_config.config_vrdc(hostname, mgmt_ip)  # -> here I'm calling another module that configures the routers using pexpect

Thanks for any help here.

CodePudding user response：

You might want to use the multiprocessing module:

##################################################

# Glue functions to get the original function running

import time

def create_vrdc_dic():
    return {
        'host1': { 'hostname': 'host1', 'mgmt_ip': '10.0.0.1' },
        'host2': { 'hostname': 'host2', 'mgmt_ip': '10.0.0.2' },
        'host3': { 'hostname': 'host3', 'mgmt_ip': '10.0.0.3' },
        'host4': { 'hostname': 'host4', 'mgmt_ip': '10.0.0.4' },
        'host5': { 'hostname': 'host5', 'mgmt_ip': '10.0.0.5' } }

class ConsoleConfig:

    def config_vrdc(self, hostname, mgmt_ip):
        print("Configure [%s] [%s]" % (hostname, mgmt_ip))
        time.sleep(10)
        print("Finished configuration [%s] [%s]" % (hostname, mgmt_ip))

console_config = ConsoleConfig()
        
##################################################

from multiprocessing import Process

def configure_routers():

    hosts = create_vrdc_dic()  # -> it's the dictonary that pass the variables

    processes = []
    
    for i in hosts .keys():
        hostname = hosts [i].get('hostname')
        mgmt_ip = hosts [i].get('mgmt_ip')

        p = Process(target=console_config.config_vrdc, args=(hostname, mgmt_ip))
        p.start()
        processes.append(p)

    for p in processes:
        p.join()


##################################################

configure_routers()

Possible output:

Configure [host1] [10.0.0.1]
Configure [host2] [10.0.0.2]
Configure [host3] [10.0.0.3]
Configure [host4] [10.0.0.4]
Configure [host5] [10.0.0.5]
Finished configuration [host1] [10.0.0.1]
Finished configuration [host2] [10.0.0.2]
Finished configuration [host5] [10.0.0.5]
Finished configuration [host4] [10.0.0.4]
Finished configuration [host3] [10.0.0.3]

Runtime: ~10 secs

CodePudding user response：

To configure your routers simultaneously, you first need to make sure it is safe to do so with the module/library you are using. If only one instance should be running at a time, you will just have to wait for each router to be configured one by one. For example, if the process you are using reads/writes to the same file/database while running and you multi-thread that process, you can end up with a corrupted file/database.

If you determine that your operation is safe to multi-thread, try something like the following code:

import threading


# Create a class for multi-threading the configuration of a router
class RouterConfigure(threading.Thread):
    def __init__(self, ip: str, hostname: str):
        threading.Thread.__init__(self)
        self.ip = ip
        self.hostname = hostname

    def run(self):
        console_config.config_vrdc(self.hostname, self.ip)


threads = []
hosts = create_vrdc_dic()

# Create a thread for each IP/hostname in the dict
for i in hosts.keys():
    hostname = hosts[i].get('hostname')
    ip = hosts[i].get('mgmt_ip')
    
    thread = RouterConfigure(ip=ip, hostname=hostname)
    thread.start()
    threads.append(thread)

# Wait for threads to finish before exiting main program
for t in threads:
    t.join()

If there are more hosts to configure than your system can handle with multiple threads, figure out how to split the process up into chunks so that only x threads are running at the same time.

CodePudding user response：

Try this code , please make necessary changes according to your need .

import thread

def Configure (hostname, mgmt_ip):
    
    console_config.config_vrdc(hostname, mgmt_ip)


try :
    def configure_routers():

        hosts = create_vrdc_dic()  # -> it's the dictonary that pass the variables

        for i in hosts .keys():
            hostname = hosts [i].get('hostname')
            mgmt_ip = hosts [i].get('mgmt_ip')

            thread.start_new_thread( Configure, (hostname, mgmt_ip) )

    configure_routers()
except :
    print("The operation cannot be performed")