With Text File Create List then use List to Scrape from Webpage to create dictionary-CodePudding

I have a text file that has a list of hostnames:

router-1
firewall-1
firewall-2
distro-1

What I would like to do is create a list from that file, then from that list web scrape for certain a certain text string to determine the device's device_platform. Then with that create a dictionary that would look something like this:

[
    {'hostname': 'router-1', 'device_platform': 'cisco_ios'},
    {'hostname': 'firewall-1', 'device_platform': 'juniper'},
    {'hostname': 'firewall-2', 'device_platform': 'cisco_asa'},
    {'hostname': 'distro-1', 'device_platform': 'broadcom_icos'}
]

So far this is what I have, the web scraping portion works. I just need to know how to add a part to create the list then from that list scrape the info and return it to create a dictionary like the one above.

import requests
import urllib3


with open('devices.txt', 'r') as f:
    host = f.read().splitlines()
    print(host)
    for device in host:
        url = f'http://nunya.com/device/{device}.nunya.com/config'
        urllib3.disable_warnings()
        request = requests.get(url).text
        if 'event manager applet' in request and 'pager' not in request:
            device_platform = 'cisco_ios'
        elif 'junos' in request:
            device_platform = 'juniper'
        elif 'pager' in request:
            device_platform = 'cisco_asa'
        else:
            device_platform = 'broadcom_icos'
        print(f'hostname: {device}, device_platform: {device_platform}')

CodePudding user response：

I see that what you want in the end is of this format,

[
    {'hostname': 'router-1', 'device_platform': 'cisco_ios'},
    {'hostname': 'firewall-1', 'device_platform': 'juniper'},
    {'hostname': 'firewall-2', 'device_platform': 'cisco_asa'},
    {'hostname': 'distro-1', 'device_platform': 'broadcom_icos'}
]

This is a list of dictionaries and not much change is required to your code to get this.

Just initiate an empty list before the for loop starts and keep appending the required dictionaries to it as the loop progresses. Here is the code:

import requests
import urllib3


with open('devices.txt', 'r') as f:
    host = f.read().splitlines()
    print(host)
    data = []
    for device in host:
        url = f'http://nunya.com/device/{device}.nunya.com/config'
        urllib3.disable_warnings()
        request = requests.get(url).text
        if 'event manager applet' in request and 'pager' not in request:
            device_platform = 'cisco_ios'
        elif 'junos' in request:
            device_platform = 'juniper'
        elif 'pager' in request:
            device_platform = 'cisco_asa'
        else:
            device_platform = 'broadcom_icos'
        data.append({'hostname': device, 'device_platform': device_platform})
        print(f'hostname: {device}, device_platform: {device_platform}')
    print(data)

As you can see, I am initiating an empty list by saying data=[] and inside the loop you can see that I am appending dictionaries with the required data to the list in the line data.append({'hostname': device, 'device_platform': device_platform}).

Hope this works for you.

CodePudding user response：

file = 'devices.txt'

hostnames = []

with open(file, 'r') as f:
    lines = f.readlines()
    for line in lines:
        hostnames.append(line.strip())

# output: ['router-1', 'firewall-1', 'firewall-2', 'distro-1']
print(hostnames)