Scraping multiple links with bs4 and request-CodePudding

I'm new to this please go easy on me. So i want to scrape name and number of multiple pages but only scraping first url. Also the code is scraping only one item either Name Or Phone Number. Name is in "h2" class "c411ListedName" and number is in 'span' class "c411Phone"

import csv
import requests
from bs4 import BeautifulSoup

urls = ['https://www.canada411.ca/search/si/1/kumar/Canada/?pgLen=100','https://www.canada411.ca/search/si/2/kumar/Canada/?pgLen=100']


for url in urls:
    response = requests.get(url)
    html = requests.get(url).text
    soup = BeautifulSoup(response.content, "html.parser")
    products = soup.findAll('span','h2', class_=['c411Phone', 'c411ListedName'])

for div in products: 
    print(div.text)

CodePudding user response：

What happens

Your requesting both urls, but process only the last one, cause your second loop is will not be executed after your first loop is finished.

Solution

Improve your indentation and put the secaonde loop in the first

import csv
import requests
from bs4 import BeautifulSoup

urls = ['https://www.canada411.ca/search/si/1/kumar/Canada/?pgLen=100','https://www.canada411.ca/search/si/2/kumar/Canada/?pgLen=100']


for url in urls:
    response = requests.get(url)
    html = requests.get(url).text
    soup = BeautifulSoup(response.content, "html.parser")
    products = soup.findAll(['span','h2'], class_=['c411Phone', 'c411ListedName'])

    for div in products: 
        
        print(url,div.text)

Additional solution

Select and process the enclosing <div>

import csv
import requests
from bs4 import BeautifulSoup

urls = ['https://www.canada411.ca/search/si/1/kumar/Canada/?pgLen=100','https://www.canada411.ca/search/si/2/kumar/Canada/?pgLen=100']

data=[]

for url in urls:
    response = requests.get(url)
    html = requests.get(url).text
    soup = BeautifulSoup(response.content, "html.parser")
    
    for row in soup.select('div.c411Listing.jsResultsList > div.listing__row:nth-of-type(1)'):
        data.append({
            'name':row.h2.a.text,
            'number':row.span.text
        })

data

CodePudding user response：

You should do two things:

Put the products out of your for as an array to be able to store all urls' content
Use findall separately for each h2 and span

Here is the code:

import requests
from bs4 import BeautifulSoup

urls = ['https://www.canada411.ca/search/si/1/kumar/Canada/?pgLen=100',
        'https://www.canada411.ca/search/si/2/kumar/Canada/?pgLen=100']

products_h = []
products_span = []

for url in urls:
    response = requests.get(url)
    soup = BeautifulSoup(response.content, "html.parser")
    products_h.append(soup.findAll('span', class_=['c411Phone']))
    products_span.append(soup.findAll('h2', class_=['c411ListedName']))

for i in range(len(urls)):
    for span, h in zip(products_span[i], products_h[i]):
        print(f'{span.text},{h.text}')

Here is the result that I got:

B Kumar,(416) 444-5781
V Roy Kumar,(613) 226-2668
C Kumar,(514) 388-4472
Ananth Kumar,(905) 389-2429
ASHOK KUMAR,(905) 575-7795
R Kumar,(905) 820-2447
A Kumar,(613) 736-1882
S R Kumar,(705) 426-9866
R Kumar,(519) 886-9588
S & H Kumar,(905) 303-6949
V KUMAR,(613) 834-1453
Ashwani Kumar,(867) 633-5854
S Kumar,(514) 620-1780
V Kumar,(514) 421-8659
B C M Kumar,(905) 335-4480
Vasant Kumar,(613) 723-3485
MICHAEL KUMAR,(519) 824-8555
M Kumar,(613) 747-3790
G Kumar,(905) 265-2017
T Kumar,(905) 824-8625
A Kumar,(905) 508-1561
U Kumar,(905) 433-2165
J Kumar,(905) 426-5249
Priya Kumar,(905) 827-0090
C Kumar,(905) 216-4078
M Kumar,(905) 456-6423
R Kumar,(647) 827-5029
N Kumar,(647) 350-2730
Sanjai Kumar,(416) 913-3648
R Kumar,(905) 916-2177
S Kumar,(289) 232-5477
J Kumar,(416) 238-9504
P Kumar,(905) 956-1699
R Kumar,(905) 793-9551
S Kumar,(905) 848-8282
Naresh Kumar,(905) 257-6640
K Kumar-Telang,(905) 450-1745
Poonam Kumar,(905) 937-6120
R Kumar,(416) 741-1314
A Kumar,(905) 454-4490
A Kumar,(905) 799-8742
Subodh Kumar,(416) 239-4590
R Kumar,(905) 888-0007
S Kumar,(416) 421-2311
R Kumar,(905) 654-8063
N Kumar,(905) 295-4343
R Kumar,(905) 303-4532
S Kumar,(519) 653-0474
Aneal Kumar,(905) 792-1051
Unesh Kumar,(905) 451-3614
Virender Kumar,(416) 483-9116
R Kumar,(905) 796-3413
M Kumar,(519) 691-1176
S SAMPATH-kUMAR,(416) 745-7911
S Kumar,(416) 724-5511
R KUMAR,(416) 256-7991
C Kumar,(905) 509-0211
A Kumar,(905) 455-2071
Ram Kumar,(905) 829-4686
D Kumar,(905) 608-1849
R Kumar,(416) 283-9395
M Kumar,(905) 456-0497
P Kumar,(416) 284-5028
M Kumar,(705) 724-3507
Pradeep Kumar,(613) 548-4619
R Kumar,(416) 481-6271
V Kumar,(905) 451-0797
Rakesh Kumar,(416) 626-8349
J Kumar,(416) 604-4806
A Kumar,(416) 759-7526
A Kumar,(905) 460-9268
Parapurath S Kumar,(905) 794-3673
N Kumar,(905) 642-9708
Praveen Kumar,(647) 827-1562
V Kumar,(905) 488-7531
A Kumar,(905) 901-0050
R Kumar,(905) 216-7945
Praveen Kumar,(416) 238-1247
J Kumar,(905) 265-7537
Praveen Kumar,(416) 855-3744
S Kumar,(905) 792-7671
Ragesh Kumar,(647) 727-5826
Goutham Kumar,(905) 670-9268
V Kumar,(519) 472-2394
Dharmana Kumar,(905) 908-2216
K Kumar,(905) 654-7177
J Kumar,(905) 554-2210
Praveen Kumar,(416) 855-3091
D Kumar,(416) 901-7977
Parmod Kumar,(905) 456-2286
S Kumar,(416) 674-5392
Suby Kumar,(905) 831-8886
A Kumar,(416) 282-0182
A Kumar,(905) 915-7124
Surinder Kumar,(905) 264-7743
Rajesh Kumar,(416) 855-1081
Selva Kumar,(905) 769-4456
Santhosh Kumar,(905) 769-4503
Praveen Kumar,(416) 855-1880
K Kumar,(416) 467-6727
J Kumar,(905) 453-9596
C Kumar,(905) 683-5983
k kumar,(416) 746-2705
A Kumar,(416) 667-9363
Praveen Kumar,(416) 855-0367
J Kumar,(905) 666-4988
Ashok N Kumar,(905) 877-2302
V Chandra Kumar,(416) 724-0156
D Kumar,(416) 332-8825
S Kumar,(905) 476-1995
Maria Kumar,(905) 499-4918
S KUMAR,(905) 814-0518
P Kumar,(905) 522-1386
SURESH KUMAR,(514) 693-0564
S Kumar,(514) 932-1504
Santosh Kumar,(416) 855-3747
p kumar,(905) 604-0670
V R Kumar,(416) 766-3335
Satish Kumar,(416) 406-2340
J Kumar,(905) 854-2337
Sam Kumar,(905) 854-2342
R Kumar,(416) 593-4306
M Kumar,(905) 338-7543
Nita Kumar,(905) 653-0474
Rajender Kumar,(705) 522-2213
R Kumar,(416) 438-3855
V KUMAR,(705) 522-6114
Raman Kumar,(905) 886-0558
B Kumar,(416) 445-4501
S Kumar,(416) 246-9104
S Kumar,(905) 792-7124
S Kumar,(905) 495-3315
D Kumar,(416) 742-8025
R Kumar,(905) 824-8170
Pj Kumar,(905) 785-3110
Rai Bal Kumar,(418) 522-8497
S Kumar,(514) 620-0626
S Kumar,(705) 352-0418
V KUMAR,(647) 345-3130
R Kumar,(416) 208-7965
R Kumar,(226) 663-3309
R Kumar,(905) 553-6099
K Kumar,(905) 566-5855
P Kumar Rudra,(905) 565-6553
Rashmi Kumar,(905) 363-7578
Anil Kumar,(905) 497-6514
Shiv Kumar,(416) 265-5763
Aprul Kumar,(416) 240-9379
A Kumar,(289) 752-7462
P Kumar,(416) 744-3493
T Melburn-Kumar,(905) 524-5529
V Kumar,(416) 724-7624
Poonam Kumar,(905) 430-1763
C Kumar,(905) 470-1296
V Kumar,(905) 237-8979
M KUMAR,(905) 458-1234
R Kumar,(416) 335-0731
A Kumar,(416) 213-9003
M Kumar,(905) 564-5779
S Kumar,(905) 890-3061
P & R Kumar,(905) 726-4444
L Kumar,(416) 438-3283
R Kumar,(613) 841-4638
R KUMAR,(416) 490-9860
K Kumar,(226) 647-1075
J Kumar,(905) 654-1532
S Kumar,(905) 472-4343
D Kumar,(905) 956-7503
Jagjit Kumar,(905) 915-8826
Vimala Krishna Kumar,(416) 551-3021
R Kumar,(647) 340-0236
A Kumar,(905) 791-8114
Santosh Kumar,(416) 431-2661
Vijay Kumar,(905) 487-9475
Dr Vinod Dr Uma Kumar,(613) 737-4113
S Kumar,(905) 281-8837
L KUMAR,(905) 860-2970
Mifan Kumar,(905) 201-0539
S Kumar,(416) 746-5671
Padin Kumar,(905) 881-3870
Shiue Kumar,(905) 499-0950
M Kumar,(905) 216-8303
P Kumar,(289) 752-6480
A Kumar,(613) 837-3511
R Kumar,(905) 846-7810
R Kumar,(416) 286-5743
E Kumar,(613) 440-2647
S Kumar,(905) 840-1426
V Kumar,(819) 681-6376
M Kumar,(416) 287-1812
D Kumar,(705) 352-2627
S Kumar,(905) 455-2242
Raj Kumar,(438) 288-3934
A Kumar,(289) 752-2619
S Kumar,(416) 284-0542
S Kumar,(705) 352-5001
V Kumar,(905) 820-6636
P KUMAR,(514) 342-4183
A Kumar,(519) 256-2012
P Kumar,(416) 412-7972

Hope it was helpful.