I am trying to scrape some data from this website.

For data scraping I'm using Scrapy framework.

I inspected the webpage and found out that the data I want to extract has the following XPath:

//*[@id="weather-widget"]/div[2]/div[1]/div[1]/div[1]/h2

When I scraped the webpage and started looking through its content I found out that the page does not contain an element with the XPath above.

Is it possible to somehow wait for the page to load and extract the values that I need?

CodePudding user response：

You are not finding the data you are looking for because it's loaded from another request. you can either use something like selenium or puppeteer to load the full page, Or you can try to open send the request directly to the api to get the data.

For the site you provided the request that generates the data is something similar to

https://openweathermap.org/data/2.5/weather?id=625665&appid=439d4b804bc8187953eb36d2a8c26a02

you can confirm it by opening DevTools > Network tab and refresh the page to see the request