I am trying to access data on this website: https://vemcount.app/embed/widget/uOCRuLPangWo5fT?locale=en

My code so far is as follows:

from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import TimeoutException


def configure_driver():
    chrome_options = Options()
    chrome_options.add_argument("--headless")
    chrome_options.add_argument("--use-fake-ui-for-media-stream")
    driver = webdriver.Chrome(executable_path="C:\\Users\\uqiabde1\\Downloads\\chromedriver.exe", options = chrome_options)
    return driver


def getNumber(driver):
    # Step 1: Go to website
    driver.get(f"https://vemcount.app/embed/widget/uOCRuLPangWo5fT?locale=en")
    # wait for the element to load
    try:
        WebDriverWait(driver, 10).until(lambda s: s.find_element_by_id("flex items-center").is_displayed())
    except TimeoutException:
        print("TimeoutException: Element not found")
        return None

    # Step 2: Create a parse tree of page sources after searching
    soup = BeautifulSoup(driver.page_source, "lxml")
    # Step 3: Iterate over the search result and fetch the number
    for number in soup.select("div.items-center"):
        number_needed = "p span"
        print({
            "title": number.select_one(number_needed).text,
        })

# create the driver object.
driver = configure_driver()
getNumber(driver)
# close the driver.
driver.close()

I get the following error in chromedriver

[0414/150051.086:INFO:CONSOLE(2)] "The AudioContext was not allowed to start. It must be resumed (or created) after a user gesture on the page. https://goo[dot]gl/7K7WLu", source: https://vemcount.app/build/embed.js?id=2ff0173dd78c5c1f99c6 (2)

I am not sure which chrome_option to use to bypass this error. I tried a few such as

--no-user-gesture-required

and

--disable-gesture-requirement-for-presentation

Your help would be highly appreciated. Thanks.

CodePudding user response：

Just in case you had the same query or were struggling with dynamic web scraping I found a way to scrape the data without using selenium and bs4.

I used playwright which is far more straightforward and has a very much appreciated inner_html() function which reads straight into the dynamic flex HTML code. Here is the code for reference.

#part of the help to write the script I got from https://stackoverflow.com/questions/64303326/using-playwright-for-python-how-do-i-select-or-find-an-element

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(slow_mo=1000)

    page = browser.new_page()
    page.goto('https://vemcount.app/embed/widget/uOCRuLPangWo5fT')
    central = page.query_selector("p.w-full span");
    print({'central': central.inner_html()})
        
    browser.close()
 

If there is a better way I am more than happy to hear your suggestions.

Yours,

A Beginner in Python. :)