I am trying to scrape some links from https://www.mckinsey.com/capabilities/operations/our-insights using selenium with python.
from selenium.webdriver.common.by import By
from selenium import webdriver
from bs4 import BeautifulSoup
import time
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--disable-notifications")
# browser = webdriver.Chrome('C:\\chromedriver.exe', options=chrome_options)
browser = webdriver.Firefox()
url = "https://www.mckinsey.com/capabilities/operations/our-insights"
browser.get(url)
time.sleep(5)
try:
accept = browser.find_element(By.XPATH, '//*[@id="onetrust-accept-btn-handler"]')
accept.click()
time.sleep(2)
browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")
except:
pass
n = 1
while n < 3:
try:
browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")
button = browser.find_element(By.XPATH, '//*[@id="skipToMain"]/div[2]/section[11]/div[2]/a')
button.click()
time.sleep(2)
browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")
print('page', n)
n = n 1
except:
print('page ended at', n)
break
source = browser.execute_script("return document.body.innerHTML")
time.sleep(5)
soup = BeautifulSoup(source, 'lxml')
Running above code gave the following error.
Exception ignored in: <function Service.__del__ at 0x000002AE1979DAF0>
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\lib\site-packages\selenium\webdriver\common\service.py", line 177, in __del__
AttributeError: 'NoneType' object has no attribute 'suppress'
I tried both Chrome and Firefox. Both of them are giving the same error.
Python version - 3.9
CodePudding user response：
You can try the next example selenium with bs4
url='https://www.mckinsey.com/capabilities/operations/our-insights'
driver.get(url)
driver.maximize_window()
time.sleep(3)
accept = driver.find_element(By.XPATH, '//*[@id="onetrust-accept-btn-handler"]')
accept.click()
time.sleep(2)
data = []
for x in range(3):
try:
soup = BeautifulSoup(driver.page_source, 'lxml')
links = soup.select('[]>div')
print(len(links))
for x in links:
link = x.a
link = 'https://www.mckinsey.com' link.get('href') if link else None
data.append(link)
loadMoreButton = driver.find_element(By.XPATH, "//a[contains(text(),'View more')]")
if loadMoreButton:
driver.execute_script("arguments[0].click();" ,loadMoreButton)
#loadMoreButton.click()
time.sleep(3)
except Exception as e:
print(e)
break
print(set(data))
# df = pd.DataFrame(set(data))
# print(df)
Output:
{'https://www.mckinsey.com/capabilities/operations/our-insights/global-infrastructure-initiative/voices/v
oices-introduction-october-2022', 'https://www.mckinsey.com/capabilities/people-and-organizational-perfor
mance/our-insights/leading-operating-model-modernization-what-do-transformation-leaders-say', 'https://ww
w.mckinsey.com/capabilities/operations/our-insights/sustainable-spaces-countering-climate-risk-in-capital
-projects', 'https://www.mckinsey.com/capabilities/operations/our-insights/harnessing-volatility-technolo
gy-transformation-in-oil-and-gas', 'https://www.mckinsey.com/capabilities/operations/our-insights/buildin
g-supply-chain-resilience', 'https://www.mckinsey.com/capabilities/operations/our-insights/industrial-res
ource-productivity-and-the-road-to-sustainability', 'https://www.mckinsey.com/capabilities/operations/our
-insights/outsprinting-the-energy-crisis', 'https://www.mckinsey.com/capabilities/operations/our-insights
/emerging-from-disruption-the-future-of-pharma-operations-strategy', 'https://www.mckinsey.com/industries
/public-and-social-sector/our-insights/using-advanced-analytics-to-improve-performance-in-customs-agencie
s', 'https://www.mckinsey.com/industries/life-sciences/our-insights/against-the-odds-how-life-sciences-co
mpanies-excel-in-large-transformations', 'https://www.mckinsey.com/capabilities/operations/our-insights/t
he-hidden-value-of-voice-conversations-part-1-trends-and-technologies', 'https://www.mckinsey.com/feature
d-insights/mckinsey-on-books/the-titanium-economy', 'https://www.mckinsey.com/capabilities/operations/our
-insights/smart-scheduling-how-to-solve-workforce-planning-challenges-with-ai', 'https://www.mckinsey.com
/capabilities/operations/our-insights/generative-scheduling-saving-time-and-money-in-capital-projects', '
https://www.mckinsey.com/capabilities/operations/our-insights/global-infrastructure-initiative/voices/on-
the-path-to-net-zero-steel-in-building-and-construction', 'https://www.mckinsey.com/capabilities/operatio
ns/our-insights/global-infrastructure-initiative/voices/disrupting-transport-an-interview-with-robert-fal
ck-of-einride', 'https://www.mckinsey.com/capabilities/operations/our-insights/the-industrial-revolution-
in-services', 'https://www.mckinsey.com/capabilities/operations/our-insights/how-mining-companies-reach-t
he-operational-excellence-gold-standard', 'https://www.mckinsey.com/capabilities/operations/our-insights/
a-more-resilient-supply-chain-from-optimized-operations-planning', 'https://www.mckinsey.com/capabilities
/operations/our-insights/full-potential-procurement-lessons-amid-inflation-and-volatility', 'https://www.
mckinsey.com/capabilities/operations/our-insights/taking-the-pulse-of-shifting-supply-chains', 'https://w
ww.mckinsey.com/capabilities/operations/our-insights/global-infrastructure-initiative/voices/news-from-th
e-global-infrastructure-initiative-august-2022', 'https://www.mckinsey.com/capabilities/operations/our-in
sights/delivering-the-us-manufacturing-renaissance', 'https://www.mckinsey.com/capabilities/operations/ou
r-insights/building-sustainability-into-operations', 'https://www.mckinsey.com/capabilities/operations/ou
r-insights/global-infrastructure-initiative/voices/news-from-the-global-infrastructure-initiative-october
-2022', 'https://www.mckinsey.com/capabilities/operations/our-insights/global-infrastructure-initiative/v
oices/dhl-on-sustainable-customer-centric-delivery-in-the-last-mile', 'https://www.mckinsey.com/industrie
s/advanced-electronics/our-insights/sustainability-in-packaging-five-key-levers-for-significant-impact',
'https://www.mckinsey.com/capabilities/operations/our-insights/global-infrastructure-initiative/voices/do
ing-good-demands-doing-better-delivering-net-zero-capital-projects', 'https://www.mckinsey.com/industries
/life-sciences/our-insights/reimagining-the-future-of-biopharma-manufacturing', 'https://www.mckinsey.com
/capabilities/operations/our-insights/coca-cola-the-people-first-story-of-a-digital-transformation', 'htt
ps://www.mckinsey.com/capabilities/operations/our-insights/is-your-manufacturing-network-an-anchor-or-a-s
ail', 'https://www.mckinsey.com/industries/semiconductors/our-insights/rapid-throughput-improvement-at-ma
ture-semiconductor-fabs', 'https://www.mckinsey.com/capabilities/operations/our-insights/global-infrastru
cture-initiative/voices/reducing-embodied-carbon-in-new-construction', 'https://www.mckinsey.com/industri
es/healthcare-systems-and-services/our-insights/optimizing-health-system-supply-chain-performance', 'http
s://www.mckinsey.com/capabilities/operations/our-insights/global-infrastructure-initiative/voices/managin
g-capital-risk-in-the-race-to-net-zero', 'https://www.mckinsey.com/capabilities/operations/our-insights/a
ccelerating-green-growth-in-the-built-environment', 'https://www.mckinsey.com/capabilities/transformation
/our-insights/you-cant-move-too-fast-a-conversation-with-andy-penn', 'https://www.mckinsey.com/capabiliti
es/operations/our-insights/inflation-fighter-and-value-creator-procurements-best-kept-secret', 'https://w
ww.mckinsey.com/capabilities/operations/our-insights/global-infrastructure-initiative/voices/preparing-fo
r-tomorrow-an-interview-with-tariq-taherbhai', 'https://www.mckinsey.com/capabilities/operations/our-insi
ghts/value-speed-and-scale-a-new-era-for-operations-in-asia', 'https://www.mckinsey.com/capabilities/oper
ations/our-insights/stepping-up-what-coos-will-need-to-succeed-in-2023-and-beyond', None, 'https://www.mc
kinsey.com/capabilities/operations/our-insights/digital-twins-what-could-they-do-for-your-business', 'htt
ps://www.mckinsey.com/capabilities/operations/our-insights/global-infrastructure-initiative/voices/voices
-introduction-august-2022', 'https://www.mckinsey.com/capabilities/operations/our-insights/how-good-are-y
our-internal-operations-really', 'https://www.mckinsey.com/capabilities/operations/our-insights/114-down-
10-million-to-go-the-global-lighthouse-networks-mission', 'https://www.mckinsey.com/capabilities/operatio
ns/our-insights/people-and-places-how-and-where-to-work-next', 'https://www.mckinsey.com/capabilities/ope
rations/our-insights/accelerating-capital-projects-to-secure-advantages-in-the-net-zero-transition', 'htt
ps://www.mckinsey.com/capabilities/operations/our-insights/global-infrastructure-initiative/voices/unlock
ing-hydrogens-power-for-long-haul-freight-transport', 'https://www.mckinsey.com/industries/engineering-co
nstruction-and-building-materials/our-insights/how-much-is-a-brick-that-depends', 'https://www.mckinsey.c
om/capabilities/operations/our-insights/global-infrastructure-initiative/voices/mapping-the-way-decarboni
zing-roads', 'https://www.mckinsey.com/capabilities/operations/our-insights/global-infrastructure-initiat
ive/voices/investing-in-pathways-to-decarbonize-infrastructure', 'https://www.mckinsey.com/capabilities/o
perations/our-insights/the-hidden-value-of-voice-conversations-part-2-reaping-the-rewards', 'https://www.
mckinsey.com/capabilities/operations/our-insights/power-spike-how-battery-makers-can-respond-to-surging-d
emand-from-evs', 'https://www.mckinsey.com/capabilities/operations/our-insights/the-care-of-one-hyperpers
onalization-of-customer-care', 'https://www.mckinsey.com/capabilities/operations/our-insights/the-scaling
-imperative-for-industry-4-point-0', 'https://www.mckinsey.com/capabilities/operations/our-insights/utili
ty-procurement-ready-to-meet-new-market-challenges', 'https://www.mckinsey.com/capabilities/operations/ou
r-insights/global-infrastructure-initiative/voices/the-art-of-the-possible-an-interview-with-leaders-from
-scottish-water', 'https://www.mckinsey.com/capabilities/operations/our-insights/converge-it-and-ot-to-tu
rbocharge-business-operations-scaling-power'}