Home > Net >  How to index through <li of a <ul given by a class name
How to index through <li of a <ul given by a class name

Time：07-16

I have an html code similar to the following

<div>
<div>
    <div>
        <div>
            <div>
                <div>
                <div>
                    <section >
                        <article>
                            <div>
                                <div>
                                <ul    class='hover p-0  job-search-key-kgm6qi exy0tjh1'>
                                    <li     <a....</a>    </li>
                                    <li     <a....</a>    </li>
                                    <li     <a....</a>    </li>
                                    <li     <a....</a>    </li>
                                    <li     <a....</a>    </li>
                                    <li     <a....</a>    </li>
                                <ul/>
                                <div/>
                            <div/>
                        <article/>
                    <section/>
                <div/>
                <div/>
            <div/>
        <div/>
    <div/>
<div/>

Also, here's an image of the real code enter image description here

I want to get all the links that are contained in the <li.... of the <ul where the class is named 'hover p-0 job-search-key-kgm6qi exy0tjh1' I tried to use the following code, but it gives me an empty list

import requests
from bs4 import BeautifulSoup


page = requests.get('https://www.glassdoor.com/Job/united-states-business-intelligence-analyst-jobs-SRCH_IL.0,13_IN1_KO14,43_IP3.htm?includeNoSalaryJobs=true&pgc=AB4AAoEAPAAAAAAAAAAAAAAAAd1d+b4AbwECARFcBgSsznIZFPk4fW3sDR3jdfmJewv2+LA8TQi8YFhbqVFBx1lHzaVOYjw909IQ+IC38g6D/3KlCEAfpAkJoqF3AECpEZLOSE/DN2rQ33BRLDrpbxe6MwndhK4pkphs5Qq5uzHuklUQKM+rUQAA')


soup = BeautifulSoup(page.text, 'html.parser')


results = soup.find_all('ul', attrs={'class':'hover p-0  job-search-key-kgm6qi exy0tjh1'})


print(results)

CodePudding user response：

You need to give requests a proper header, and also to revisit Beautiful Soup syntax (afaik you cannot select multiple classes at once, separated by spaces, in selector). The following code works:

import requests
from bs4 import BeautifulSoup

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.60 Safari/537.17'}

r = requests.get('https://www.glassdoor.com/Job/united-states-business-intelligence-analyst-jobs-SRCH_IL.0,13_IN1_KO14,43_IP3.htm?includeNoSalaryJobs=true&pgc=AB4AAoEAPAAAAAAAAAAAAAAAAd1d+b4AbwECARFcBgSsznIZFPk4fW3sDR3jdfmJewv2+LA8TQi8YFhbqVFBx1lHzaVOYjw909IQ+IC38g6D/3KlCEAfpAkJoqF3AECpEZLOSE/DN2rQ33BRLDrpbxe6MwndhK4pkphs5Qq5uzHuklUQKM+rUQAA', headers=headers)
soup = BeautifulSoup(r.text, 'html.parser')
results = soup.select('li.eigr9kq0')
for result in results:
    print(result.text)

CodePudding user response：

Here, CSS selectors .p-0.job-search-key-kgm6qi li will select all the items/joblistings inside the page where exists the a tags.

import requests
from bs4 import BeautifulSoup
page = requests.get('https://www.glassdoor.com/Job/united-states-business-intelligence-analyst-jobs-SRCH_IL.0,13_IN1_KO14,43_IP3.htm?includeNoSalaryJobs=true&pgc=AB4AAoEAPAAAAAAAAAAAAAAAAd1d+b4AbwECARFcBgSsznIZFPk4fW3sDR3jdfmJewv2+LA8TQi8YFhbqVFBx1lHzaVOYjw909IQ+IC38g6D/3KlCEAfpAkJoqF3AECpEZLOSE/DN2rQ33BRLDrpbxe6MwndhK4pkphs5Qq5uzHuklUQKM+rUQAA')

soup = BeautifulSoup(page.text, 'html.parser')
for card in soup.select('.p-0.job-search-key-kgm6qi li'):
        link='https://www.glassdoor.com'   card.a.get('href')
        print(link)

Output:

https://www.glassdoor.com/partner/jobListing.htm?pos=301&ao=1136043&s=58&guid=00000182066fdb7e96f74b88d925bfc5&src=GD_JOB_AD&t=SR&vt=w&cs=1_700511c1&cb=1657965370987&jobListingId=1008003280530&jrtk=3-0-1g836vmu02hiq001-1g836vmugg4e4800-783554786194c580-
https://www.glassdoor.com/partner/jobListing.htm?pos=302&ao=1136043&s=58&guid=00000182066fdb7e96f74b88d925bfc5&src=GD_JOB_AD&t=SR&vt=w&cs=1_f8bbc903&cb=1657965370987&jobListingId=1007790311958&jrtk=3-0-1g836vmu02hiq001-1g836vmugg4e4800-3c723d57343ab33a-
https://www.glassdoor.com/partner/jobListing.htm?pos=303&ao=1136043&s=58&guid=00000182066fdb7e96f74b88d925bfc5&src=GD_JOB_AD&t=SR&vt=w&ea=1&cs=1_475f2b1e&cb=1657965370987&jobListingId=1008005830630&jrtk=3-0-1g836vmu02hiq001-1g836vmugg4e4800-01e135fc17eca753-
https://www.glassdoor.com/partner/jobListing.htm?pos=304&ao=1136043&s=58&guid=00000182066fdb7e96f74b88d925bfc5&src=GD_JOB_AD&t=SR&vt=w&ea=1&cs=1_07909ad6&cb=1657965370987&jobListingId=1008002556707&jrtk=3-0-1g836vmu02hiq001-1g836vmugg4e4800-a178780a2b0f5319-
https://www.glassdoor.com/partner/jobListing.htm?pos=305&ao=1136043&s=58&guid=00000182066fdb7e96f74b88d925bfc5&src=GD_JOB_AD&t=SR&vt=w&ea=1&cs=1_0c126dde&cb=1657965370987&jobListingId=1008006059736&jrtk=3-0-1g836vmu02hiq001-1g836vmugg4e4800-6eadcfe562af029c-
https://www.glassdoor.com/partner/jobListing.htm?pos=306&ao=1136043&s=58&guid=00000182066fdb7e96f74b88d925bfc5&src=GD_JOB_AD&t=SR&vt=w&cs=1_307ffb22&cb=1657965370987&jobListingId=1007829741391&jrtk=3-0-1g836vmu02hiq001-1g836vmugg4e4800-32439b3b2ca53df5-
https://www.glassdoor.com/partner/jobListing.htm?pos=307&ao=1136043&s=58&guid=00000182066fdb7e96f74b88d925bfc5&src=GD_JOB_AD&t=SR&vt=w&ea=1&cs=1_69366e40&cb=1657965370988&jobListingId=1008005717049&jrtk=3-0-1g836vmu02hiq001-1g836vmugg4e4800-5288b8793dbf85aa-
https://www.glassdoor.com/partner/jobListing.htm?pos=308&ao=1136043&s=58&guid=00000182066fdb7e96f74b88d925bfc5&src=GD_JOB_AD&t=SR&vt=w&cs=1_175ba654&cb=1657965370988&jobListingId=1007987531693&jrtk=3-0-1g836vmu02hiq001-1g836vmugg4e4800-c88bc86f3db7c43f-
https://www.glassdoor.com/partner/jobListing.htm?pos=309&ao=1136043&s=58&guid=00000182066fdb7e96f74b88d925bfc5&src=GD_JOB_AD&t=SR&vt=w&cs=1_e47d6ca3&cb=1657965370988&jobListingId=1007993393530&jrtk=3-0-1g836vmu02hiq001-1g836vmugg4e4800-9e957426b9de234f-
https://www.glassdoor.com/partner/jobListing.htm?pos=310&ao=1136043&s=58&guid=00000182066fdb7e96f74b88d925bfc5&src=GD_JOB_AD&t=SR&vt=w&cs=1_ca6260ff&cb=1657965370988&jobListingId=1007993059565&jrtk=3-0-1g836vmu02hiq001-1g836vmugg4e4800-5f932659844029d3-
https://www.glassdoor.com/partner/jobListing.htm?pos=311&ao=1136043&s=58&guid=00000182066fdb7e96f74b88d925bfc5&src=GD_JOB_AD&t=SR&vt=w&ea=1&cs=1_983b6d93&cb=1657965370988&jobListingId=1008002283519&jrtk=3-0-1g836vmu02hiq001-1g836vmugg4e4800-6d35c4d19c644392-
https://www.glassdoor.com/partner/jobListing.htm?pos=312&ao=1136043&s=58&guid=00000182066fdb7e96f74b88d925bfc5&src=GD_JOB_AD&t=SR&vt=w&cs=1_27ba48cd&cb=1657965370988&jobListingId=1007977745227&jrtk=3-0-1g836vmu02hiq001-1g836vmugg4e4800-fb4561a71d5af956-
https://www.glassdoor.com/partner/jobListing.htm?pos=313&ao=1136043&s=58&guid=00000182066fdb7e96f74b88d925bfc5&src=GD_JOB_AD&t=SR&vt=w&ea=1&cs=1_b14fe5ed&cb=1657965370988&jobListingId=1007990143647&jrtk=3-0-1g836vmu02hiq001-1g836vmugg4e4800-ccaf54823ef26c92-
https://www.glassdoor.com/partner/jobListing.htm?pos=314&ao=1136043&s=58&guid=00000182066fdb7e96f74b88d925bfc5&src=GD_JOB_AD&t=SR&vt=w&cs=1_e66da6b8&cb=1657965370989&jobListingId=1007974504706&jrtk=3-0-1g836vmu02hiq001-1g836vmugg4e4800-e83639f381f40986-
https://www.glassdoor.com/partner/jobListing.htm?pos=315&ao=1136043&s=58&guid=00000182066fdb7e96f74b88d925bfc5&src=GD_JOB_AD&t=SR&vt=w&cs=1_98aaf035&cb=1657965370989&jobListingId=1007689059431&jrtk=3-0-1g836vmu02hiq001-1g836vmugg4e4800-47deea5f9a341ed8-
https://www.glassdoor.com/partner/jobListing.htm?pos=316&ao=1136043&s=58&guid=00000182066fdb7e96f74b88d925bfc5&src=GD_JOB_AD&t=SR&vt=w&cs=1_49c767dd&cb=1657965370989&jobListingId=1007987449387&jrtk=3-0-1g836vmu02hiq001-1g836vmugg4e4800-3b87071e561c0d97-
https://www.glassdoor.com/partner/jobListing.htm?pos=317&ao=1136043&s=58&guid=00000182066fdb7e96f74b88d925bfc5&src=GD_JOB_AD&t=SR&vt=w&cs=1_457a4421&cb=1657965370989&jobListingId=1007998164683&jrtk=3-0-1g836vmu02hiq001-1g836vmugg4e4800-76e49b518b106c03-
https://www.glassdoor.com/partner/jobListing.htm?pos=318&ao=1136043&s=58&guid=00000182066fdb7e96f74b88d925bfc5&src=GD_JOB_AD&t=SR&vt=w&cs=1_5a506b4f&cb=1657965370989&jobListingId=1007826304948&jrtk=3-0-1g836vmu02hiq001-1g836vmugg4e4800-ccf1d0dbbfc5ed01-
https://www.glassdoor.com/partner/jobListing.htm?pos=319&ao=1136043&s=58&guid=00000182066fdb7e96f74b88d925bfc5&src=GD_JOB_AD&t=SR&vt=w&ea=1&cs=1_44459b13&cb=1657965370989&jobListingId=1008008155219&jrtk=3-0-1g836vmu02hiq001-1g836vmugg4e4800-cc6a8c52afe369a8-
https://www.glassdoor.com/partner/jobListing.htm?pos=320&ao=1136043&s=58&guid=00000182066fdb7e96f74b88d925bfc5&src=GD_JOB_AD&t=SR&vt=w&cs=1_d99048f4&cb=1657965370990&jobListingId=1008003617355&jrtk=3-0-1g836vmu02hiq001-1g836vmugg4e4800-ad28fa8a1146b429-
https://www.glassdoor.com/partner/jobListing.htm?pos=321&ao=1136043&s=58&guid=00000182066fdb7e96f74b88d925bfc5&src=GD_JOB_AD&t=SR&vt=w&ea=1&cs=1_0033647c&cb=1657965370990&jobListingId=1007925943602&jrtk=3-0-1g836vmu02hiq001-1g836vmugg4e4800-8da49fc866efccc1-
https://www.glassdoor.com/partner/jobListing.htm?pos=322&ao=1136043&s=58&guid=00000182066fdb7e96f74b88d925bfc5&src=GD_JOB_AD&t=SR&vt=w&cs=1_927ea256&cb=1657965370990&jobListingId=1008002520460&jrtk=3-0-1g836vmu02hiq001-1g836vmugg4e4800-1d9d1b7652ffd9f6-
https://www.glassdoor.com/partner/jobListing.htm?pos=323&ao=1136043&s=58&guid=00000182066fdb7e96f74b88d925bfc5&src=GD_JOB_AD&t=SR&vt=w&ea=1&cs=1_b417cfd0&cb=1657965370990&jobListingId=1008008926897&jrtk=3-0-1g836vmu02hiq001-1g836vmugg4e4800-8065e3128ef0c5c7-
https://www.glassdoor.com/partner/jobListing.htm?pos=324&ao=1136043&s=58&guid=00000182066fdb7e96f74b88d925bfc5&src=GD_JOB_AD&t=SR&vt=w&cs=1_a64fcc14&cb=1657965370990&jobListingId=1008008261004&jrtk=3-0-1g836vmu02hiq001-1g836vmugg4e4800-eb3d644a2154f906-
https://www.glassdoor.com/partner/jobListing.htm?pos=325&ao=1136043&s=58&guid=00000182066fdb7e96f74b88d925bfc5&src=GD_JOB_AD&t=SR&vt=w&cs=1_978905b7&cb=1657965370991&jobListingId=1007918040205&jrtk=3-0-1g836vmu02hiq001-1g836vmugg4e4800-b797a33dbf9666ce-
https://www.glassdoor.com/partner/jobListing.htm?pos=326&ao=1136043&s=58&guid=00000182066fdb7e96f74b88d925bfc5&src=GD_JOB_AD&t=SR&vt=w&cs=1_ad56424f&cb=1657965370991&jobListingId=1007505824978&jrtk=3-0-1g836vmu02hiq001-1g836vmugg4e4800-4e13b474885fdb09-
https://www.glassdoor.com/partner/jobListing.htm?pos=327&ao=1136043&s=58&guid=00000182066fdb7e96f74b88d925bfc5&src=GD_JOB_AD&t=SR&vt=w&ea=1&cs=1_454e9a5e&cb=1657965370991&jobListingId=1008006071045&jrtk=3-0-1g836vmu02hiq001-1g836vmugg4e4800-5604e535b1240678-
https://www.glassdoor.com/partner/jobListing.htm?pos=328&ao=1136043&s=58&guid=00000182066fdb7e96f74b88d925bfc5&src=GD_JOB_AD&t=SR&vt=w&cs=1_7ad8fc72&cb=1657965370991&jobListingId=1007997701801&jrtk=3-0-1g836vmu02hiq001-1g836vmugg4e4800-0d0b34a03fc5d105-
https://www.glassdoor.com/partner/jobListing.htm?pos=329&ao=1136043&s=58&guid=00000182066fdb7e96f74b88d925bfc5&src=GD_JOB_AD&t=SR&vt=w&ea=1&cs=1_2e0abe52&cb=1657965370991&jobListingId=1007971249327&jrtk=3-0-1g836vmu02hiq001-1g836vmugg4e4800-11ba1bbd01df8a67-
https://www.glassdoor.com/partner/jobListing.htm?pos=330&ao=1136043&s=58&guid=00000182066fdb7e96f74b88d925bfc5&src=GD_JOB_AD&t=SR&vt=w&cs=1_9cc68642&cb=1657965370991&jobListingId=1007956964861&jrtk=3-0-1g836vmu02hiq001-1g836vmugg4e4800-3debd8799038ca17-

Page link：https//www.codepudding.com/net/477677.html

Prev:How do i handle this dropdown using selenium with no select class?
Next:scraping python requests soraredata
  • Related

About Us:  Contact Us      Terms of Service       Privacy Policy

Copyright © 2010-2023，Powered By CodePudding