I am working on a project to scrape data from a webpage, which is dynamic in nature. The div elements may change on hover or button clicks which are on-page actions, and do not take me to a different page. How can I scrape the HTML elements which get generated on hover certain items or performing in-page button clicks in the web page? I tried using selenium, but as such it requires to be run as a separate file, in a webdriver in python. Is there a way to achieve this without a webdriver executing the code in a browser, and can be tested in the shell?

I have been using beautifulsoup till now for the project. Great if I could find something there? Or else, any new library that can help me here would be great to know as well.

CodePudding user response：

No, you cannot click buttons and hovers with BeautifulSoup/Requests. What you can is:

• a button click might cause a GET/POST network call, to an API endpoint: you can send get/post requests qith Requests to that endpoint (url).
• likewise, a hover might start a network request: inspect browser's Dev tools - Network tab, and do that network call with Requests.
• use Selenium or Playwright, both tools being created to interact with dynamic JS pages

CodePudding user response：

Actually, found the solution. Using headless chrome for the web driver worked for me