Python, Selenium - parse. Can't get info from dynamic filling-CodePudding

I want to learn how to get information from dynamically generated fields. When I tried simple sites everything worked. Then I decided to try more difficult and now I can't figure it out. It took me about a two weeks to cross out the solution options that I found on the Internet over and over again. Now I'm not sure that I can get information that appears on sites in this way. Of course, most likely I'm doing something wrong, but I can't take some new idea how it do. Now, I decided to ask here. Perhaps there are those who understand this and can prompt. If yes - please give me some example.

The site I use to learn - kbp.aero/en

The information I'm trying to get (arrival schedule) - .tbody .tr .td

For example I tried:

URL = 'https://kbp.aero/en/'
    HEADERS = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36'
    }
    time.sleep(1)
    response = requests.get(URL, headers = HEADERS)
    soup = BeautifulSoup(response.content, 'html.parser')
    items = soup.find('div', class_ = 'table_wrp out yesterday')
    items = items.findAll('tr', class_ = 'tr')
    comps = []
    if(len(items) > 0):
        for item in items:
            comps.append({
                'title':item.find('td', class_ = 'td').get_text(strip = True),
            })
    for comp in comps:
        print(comp['title'])
    # for item in items:
    #     comps.append({
    #         'text': item.get_text(strip=True)
    #     })
    #
    # for comp in comps:
    #     print(comp['text'])

from selenium import webdriver
import time
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC


def main():
    driver = webdriver.Chrome()
    driver.get("https://kbp.aero/en/")

    wait = WebDriverWait(driver, 10)
    element = wait.until(EC.text_to_be_present_in_element((By.CLASS_NAME, 'tbody'), ''))

    tds = element.find_elements(By.CLASS_NAME, "td")
    for td in tds:
        print(td.text)

    # try:
    #     element = WebDriverWait(driver, 10).until(
    #         EC.presence_of_element_located((By.CLASS_NAME, "tbody"))
    #     )
    #     tds = element.find_elements(By.CLASS_NAME, "td")
    #     for td in tds:
    #         print(td.text)
    #
    # finally:
    #     driver.quit()

Thanks for any advice.

CodePudding user response：

This will fetch the entire table data:

   from time import sleep
   from selenium import webdriver
   from selenium.webdriver.common.by import By

   PATH = r"chromedriverexe path"
   driver = webdriver.Chrome(PATH)

   driver.get("https://kbp.aero/en/")
   driver.maximize_window()
   sleep(3)
   print(driver.find_element(By.CSS_SELECTOR, "div.table_wrp.out.today > table").text)

Output:

Рейс Час Призначення Перевізник Термінал Гейт Статус
TK 1256 15:05 Istanbul Turkish Airlines D D5 Boarding Completed
PS 9556 15:05 Istanbul Ukraine International Airlines D D5 Boarding Completed
7W 163 15:10 Lviv Wind Rose D D19 Boarding
FR 3167 15:10 Warsaw Ryanair D D9 Boarding
PS 9013 15:15 Ivano-Frankivsk Ukraine International Airlines D D18 Boarding
7W 113 15:15 Ivano-Frankivsk Wind Rose D D18 Boarding