from bs4 import BeautifulSoup
import requests
import pandas as pd
import time
columns=['ID','Toll Plaza Name','State','Type Of Vehicle','Single Journey','Return Journey','Monthly Pass','Commercial Vehicle Registered within the district of plaza']
df=pd.DataFrame(columns=columns)
row_count=0
URL = "https://tis.nhai.gov.in/TollInformation.aspx?TollPlazaID="
for i in range(0,491):
URL =str(i 4)
print(i 4)
r = requests.get(URL)
soup = BeautifulSoup(r.content, 'html.parser')
div = soup.find('div', attrs = {'id':'DivContaint','class':'content'})
#print(div)
name= div.find('lable').text
#print('Name: ',name)
state=div.find('div', attrs = {'class':'PA15'}).p.text.split('in')[1].split('Stretch')[0].strip()
#print('State: ', state)
table=div.find('table',attrs={'class':'tollinfotbl'})
rows=table.find_all('tr')
for j in range(1,14,2):
df.loc[row_count,'ID']=i 4
df.loc[row_count,'Toll Plaza Name']=name
df.loc[row_count,'State']=state
typeOfVehicle=rows[j].th.text
df.loc[row_count,'Type Of Vehicle']=typeOfVehicle.strip()
td=rows[j].find_all('td')
df.loc[row_count,'Single Journey']=td[0].text
df.loc[row_count,'Return Journey']=td[1].text
df.loc[row_count,'Monthly Pass']=td[2].text
df.loc[row_count,'Commercial Vehicle Registered within the district of plaza']=td[3].text
row_count =1
del r, soup, div, name, state, table, td
df.to_excel("output.xlsx")
#time.sleep(2)
Output: 4 5 6 7 8
AttributeError Traceback (most recent call last) in 15 div = soup.find('div', attrs = {'id':'DivContaint','class':'content'}) 16 #print(div) ---> 17 name= div.find('lable').text 18 #print('Name: ',name) 19 state=div.find('div', attrs = {'class':'PA15'}).p.text.split('in')[1].split('Stretch')[0].strip()
AttributeError: 'NoneType' object has no attribute 'text'
So, on the 8th id iteration, i am getting empty div content. But if i try just for that particular id i.e. 8. getting proper output.
CodePudding user response：
Well the way you have the logic, it's not iterating through each "plaza", but concatenating it. By that "8th" plaza, it's trying to go to url 'https://tis.nhai.gov.in/TollInformation.aspx?TollPlazaID=45678' which doesn't exist.
Also, if you are just adding 4 to each i, why not just adjust the range from (0,491) to (4,495)?
I'd also consider using pandas to parse that table. Just a personnel preference.
Lastly, yo do realize after each iteration, you are overwriting your output.xlsx file? I suppose it's fine as you are just appending to it, but not sure that's what you want.
Here's the code with a sample of first pages 4-15 (I'm not going to sit through all 490 iterations).
from bs4 import BeautifulSoup
import requests
import pandas as pd
import time
root_url = "https://tis.nhai.gov.in/TollInformation.aspx?TollPlazaID="
final_df = pd.DataFrame()
for i in range(4,495):
url = f'{root_url}{i}'
print(i)
r = requests.get(url)
df = pd.read_html(r.text)[0].dropna(how='all', axis=0)
soup = BeautifulSoup(r.content, 'html.parser')
div = soup.find('div', attrs = {'id':'DivContaint','class':'content'})
#print(div)
name= div.find('lable').text
#print('Name: ',name)
state=div.find('div', attrs = {'class':'PA15'}).p.text.split('in')[1].split('Stretch')[0].strip()
#print('State: ', state)
df['ID'] = i
df['Toll Plaza Name'] = name
df['State'] = state
final_df = final_df.append(df).reset_index(drop=True)
#time.sleep(2)
final_df.to_excel("output.xlsx", index=False)
Output:
print(final_df.to_string())
Type of vehicle Single Journey Return Journey Monthly Pass Commercial Vehicle Registered within the district of plaza ID Toll Plaza Name State
0 Car/Jeep/Van 95.0 145.0 2860.0 NaN 4 Thakurtola (End of Durg Bypass) ( BOT (Toll) ) Chhattisgarh
1 LCV 165.0 250.0 5005.0 NaN 4 Thakurtola (End of Durg Bypass) ( BOT (Toll) ) Chhattisgarh
2 Bus/Truck 335.0 500.0 10010.0 NaN 4 Thakurtola (End of Durg Bypass) ( BOT (Toll) ) Chhattisgarh
3 Upto 3 Axle Vehicle 535.0 805.0 16090.0 NaN 4 Thakurtola (End of Durg Bypass) ( BOT (Toll) ) Chhattisgarh
4 4 to 6 Axle 535.0 805.0 16090.0 NaN 4 Thakurtola (End of Durg Bypass) ( BOT (Toll) ) Chhattisgarh
5 HCM/EME 535.0 805.0 16090.0 NaN 4 Thakurtola (End of Durg Bypass) ( BOT (Toll) ) Chhattisgarh
6 7 or more Axle 535.0 805.0 16090.0 NaN 4 Thakurtola (End of Durg Bypass) ( BOT (Toll) ) Chhattisgarh
7 Car/Jeep/Van 55.0 80.0 1795.0 25.0 5 Bankapur ( Public Funded ) Karnataka
8 LCV 85.0 130.0 2905.0 45.0 5 Bankapur ( Public Funded ) Karnataka
9 Bus/Truck 180.0 275.0 6080.0 90.0 5 Bankapur ( Public Funded ) Karnataka
10 Upto 3 Axle Vehicle 200.0 300.0 6635.0 100.0 5 Bankapur ( Public Funded ) Karnataka
11 4 to 6 Axle 285.0 430.0 9540.0 145.0 5 Bankapur ( Public Funded ) Karnataka
12 HCM/EME 285.0 430.0 9540.0 145.0 5 Bankapur ( Public Funded ) Karnataka
13 7 or more Axle 350.0 525.0 11610.0 175.0 5 Bankapur ( Public Funded ) Karnataka
14 Car/Jeep/Van 100.0 145.0 3270.0 NaN 6 Hirebagewadi ( BOT (Toll) ) Karnataka
15 LCV 160.0 240.0 5280.0 NaN 6 Hirebagewadi ( BOT (Toll) ) Karnataka
16 Bus/Truck 330.0 500.0 11065.0 NaN 6 Hirebagewadi ( BOT (Toll) ) Karnataka
17 Upto 3 Axle Vehicle 520.0 780.0 17350.0 NaN 6 Hirebagewadi ( BOT (Toll) ) Karnataka
18 4 to 6 Axle 520.0 780.0 17350.0 NaN 6 Hirebagewadi ( BOT (Toll) ) Karnataka
19 HCM/EME 520.0 780.0 17350.0 NaN 6 Hirebagewadi ( BOT (Toll) ) Karnataka
20 7 or more Axle 635.0 950.0 21120.0 NaN 6 Hirebagewadi ( BOT (Toll) ) Karnataka
21 Car/Jeep/Van 60.0 NaN NaN NaN 7 Durg Bypass (Dhamdanaka) ( BOT (Toll) ) Chhattisgarh
22 LCV 125.0 NaN NaN NaN 7 Durg Bypass (Dhamdanaka) ( BOT (Toll) ) Chhattisgarh
23 Bus/Truck 185.0 NaN NaN NaN 7 Durg Bypass (Dhamdanaka) ( BOT (Toll) ) Chhattisgarh
24 Upto 3 Axle Vehicle 280.0 NaN NaN NaN 7 Durg Bypass (Dhamdanaka) ( BOT (Toll) ) Chhattisgarh
25 4 to 6 Axle 280.0 NaN NaN NaN 7 Durg Bypass (Dhamdanaka) ( BOT (Toll) ) Chhattisgarh
26 HCM/EME 375.0 NaN NaN NaN 7 Durg Bypass (Dhamdanaka) ( BOT (Toll) ) Chhattisgarh
27 7 or more Axle 280.0 NaN NaN NaN 7 Durg Bypass (Dhamdanaka) ( BOT (Toll) ) Chhattisgarh
28 Car/Jeep/Van 30.0 45.0 980.0 15.0 8 Hattargi ( Public Funded ) Karnataka
29 LCV 45.0 70.0 1580.0 25.0 8 Hattargi ( Public Funded ) Karnataka
30 Bus/Truck 100.0 150.0 3310.0 50.0 8 Hattargi ( Public Funded ) Karnataka
31 Upto 3 Axle Vehicle 110.0 160.0 3610.0 55.0 8 Hattargi ( Public Funded ) Karnataka
32 4 to 6 Axle 155.0 235.0 5190.0 80.0 8 Hattargi ( Public Funded ) Karnataka
33 HCM/EME 155.0 235.0 5190.0 80.0 8 Hattargi ( Public Funded ) Karnataka
34 7 or more Axle 190.0 285.0 6315.0 95.0 8 Hattargi ( Public Funded ) Karnataka
35 Car/Jeep/Van 75.0 115.0 2500.0 40.0 9 Kognoli ( Public Funded ) Karnataka
36 LCV 120.0 180.0 4040.0 60.0 9 Kognoli ( Public Funded ) Karnataka
37 Bus/Truck 255.0 380.0 8465.0 125.0 9 Kognoli ( Public Funded ) Karnataka
38 Upto 3 Axle Vehicle 275.0 415.0 9235.0 140.0 9 Kognoli ( Public Funded ) Karnataka
39 4 to 6 Axle 400.0 595.0 13275.0 200.0 9 Kognoli ( Public Funded ) Karnataka
40 HCM/EME 400.0 595.0 13275.0 200.0 9 Kognoli ( Public Funded ) Karnataka
41 7 or more Axle 485.0 725.0 16165.0 240.0 9 Kognoli ( Public Funded ) Karnataka
42 Car/Jeep/Van 90.0 135.0 3015.0 45.0 10 Tundla ( BOT (Toll) ) Uttar Pradesh
43 LCV 145.0 220.0 4870.0 75.0 10 Tundla ( BOT (Toll) ) Uttar Pradesh
44 Bus/Truck 305.0 460.0 10205.0 155.0 10 Tundla ( BOT (Toll) ) Uttar Pradesh
45 Upto 3 Axle Vehicle 335.0 500.0 11130.0 165.0 10 Tundla ( BOT (Toll) ) Uttar Pradesh
46 4 to 6 Axle 480.0 720.0 16000.0 240.0 10 Tundla ( BOT (Toll) ) Uttar Pradesh
47 HCM/EME 480.0 720.0 16000.0 240.0 10 Tundla ( BOT (Toll) ) Uttar Pradesh
48 7 or more Axle 585.0 870.0 19480.0 290.0 10 Tundla ( BOT (Toll) ) Uttar Pradesh
49 Car/Jeep/Van 25.0 40.0 895.0 15.0 11 Brahamarakotlu ( Public Funded ) Karnataka
50 LCV 45.0 65.0 1445.0 20.0 11 Brahamarakotlu ( Public Funded ) Karnataka
51 Bus/Truck 90.0 135.0 3025.0 45.0 11 Brahamarakotlu ( Public Funded ) Karnataka
52 Upto 3 Axle Vehicle 100.0 150.0 3300.0 50.0 11 Brahamarakotlu ( Public Funded ) Karnataka
53 4 to 6 Axle 140.0 215.0 4745.0 70.0 11 Brahamarakotlu ( Public Funded ) Karnataka
54 HCM/EME 140.0 215.0 4745.0 70.0 11 Brahamarakotlu ( Public Funded ) Karnataka
55 7 or more Axle 175.0 260.0 5775.0 85.0 11 Brahamarakotlu ( Public Funded ) Karnataka
56 Car/Jeep/Van 25.0 35.0 785.0 10.0 12 Cable Stayed Naini Bridge ( Public Funded ) i Bridge ( Public Funded ) Km 1.600 - NH-30
57 LCV 40.0 55.0 1265.0 20.0 12 Cable Stayed Naini Bridge ( Public Funded ) i Bridge ( Public Funded ) Km 1.600 - NH-30
58 Bus/Truck 80.0 120.0 2650.0 40.0 12 Cable Stayed Naini Bridge ( Public Funded ) i Bridge ( Public Funded ) Km 1.600 - NH-30
59 Upto 3 Axle Vehicle 85.0 130.0 2890.0 45.0 12 Cable Stayed Naini Bridge ( Public Funded ) i Bridge ( Public Funded ) Km 1.600 - NH-30
60 4 to 6 Axle 125.0 185.0 4155.0 60.0 12 Cable Stayed Naini Bridge ( Public Funded ) i Bridge ( Public Funded ) Km 1.600 - NH-30
61 HCM/EME 125.0 185.0 4155.0 60.0 12 Cable Stayed Naini Bridge ( Public Funded ) i Bridge ( Public Funded ) Km 1.600 - NH-30
62 7 or more Axle 150.0 230.0 5055.0 75.0 12 Cable Stayed Naini Bridge ( Public Funded ) i Bridge ( Public Funded ) Km 1.600 - NH-30
63 Car/Jeep/Van 80.0 120.0 2655.0 40.0 13 Bassi ( OMT ) Rajasthan
64 LCV 130.0 195.0 4285.0 65.0 13 Bassi ( OMT ) Rajasthan
65 Bus/Truck 270.0 405.0 8980.0 135.0 13 Bassi ( OMT ) Rajasthan
66 Upto 3 Axle Vehicle 295.0 440.0 9795.0 145.0 13 Bassi ( OMT ) Rajasthan
67 4 to 6 Axle 420.0 635.0 14080.0 210.0 13 Bassi ( OMT ) Rajasthan
68 HCM/EME 420.0 635.0 14080.0 210.0 13 Bassi ( OMT ) Rajasthan
69 7 or more Axle 515.0 770.0 17145.0 255.0 13 Bassi ( OMT ) Rajasthan
70 Car/Jeep/Van 70.0 100.0 2260.0 35.0 14 Aroli ( OMT ) Rajasthan
71 LCV 110.0 165.0 3645.0 55.0 14 Aroli ( OMT ) Rajasthan
72 Bus/Truck 230.0 345.0 7640.0 115.0 14 Aroli ( OMT ) Rajasthan
73 Upto 3 Axle Vehicle 250.0 375.0 8335.0 125.0 14 Aroli ( OMT ) Rajasthan
74 4 to 6 Axle 360.0 540.0 11985.0 180.0 14 Aroli ( OMT ) Rajasthan
75 HCM/EME 360.0 540.0 11985.0 180.0 14 Aroli ( OMT ) Rajasthan
76 7 or more Axle 440.0 655.0 14590.0 220.0 14 Aroli ( OMT ) Rajasthan