I'm trying to extract a table from a webpage and have tried a number of alternatives, but the table always seems to remain empty.
Two of what I thought were the most promising sets of code are attached below. Any means of extracting the data from the webpage would be considered as helpful. I have also included a screenshot of the table I want to extract.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
browser = webdriver.Chrome()
browser.set_window_size(1120, 550)
# Create an URL object
url = 'https://www.flightradar24.com/data/aircraft/ja11jc'
browser.get(url)
element = WebDriverWait(browser, 3).until(
EC.presence_of_element_located((By.ID, "tbl-datatable"))
)
data = element.get_attribute('tbl-datatable')
print(data)
browser.quit()
or alternatively,
# Import libraries
import requests
from bs4 import BeautifulSoup
import pandas as pd
# Create an URL object
url = 'https://www.flightradar24.com/data/aircraft/ja11jc'
# Create object page
page = requests.get(url)
# parser-lxml = Change html to Python friendly format
# Obtain page's information
soup = BeautifulSoup(page.text, 'lxml')
soup
# Obtain information from tag <table>
table1 = soup.find("table", id='tbl-datatable')
table1
# Obtain every title of columns with tag <th>
headers = []
for i in table1.find_all('th'):
title = i.text
headers.append(title)
# Create a dataframe
mydata = pd.DataFrame(columns = headers)
# Create a for loop to fill mydata
for j in table1.find_all('tr')[1:]:
row_data = j.find_all('td')
row = [i.text for i in row_data]
length = len(mydata)
mydata.loc[length] = row
CodePudding user response：
Best practice is and first shot scraping table data should go with pandas.read_html(), it works in most cases, needs adjustments in some cases and only fails in specific ones.
Issue here is, that a user-agent is needed with requests to avoid the 403, so we have to help pandas with that:
requests.get('http://www.flightradar24.com/data/aircraft/ja11jc',
headers={'User-Agent': 'some user agent string'}).text
)[0]
Now the table could be scraped, but have to be transformed a bit, cause that is what the browser will do while rendering - .dropna(axis=1) drops columns with NaN values and [:-1] slices the last row, that contains non relevant information:
requests.get('http://www.flightradar24.com/data/aircraft/ja11jc',
headers={'User-Agent': 'some user agent string'}).text
)[0].dropna(axis=1)[:-1]
You could also use selenium give it some time.sleep(3) while browser renders table in final form and process the driver.page_source but in my opinion this is a bit to much, in this case.
Example
import pandas as pd
import requests
df = pd.read_html(
requests.get('http://www.flightradar24.com/data/aircraft/ja11jc',
headers={'User-Agent': 'some user agent string'}).text
)[0].dropna(axis=1)[:-1]
df.columns = ['DATE','FROM', 'TO', 'FLIGHT', 'FLIGHT TIME', 'STD', 'ATD', 'STA','STATUS']
df
Output
| DATE | FROM | TO | FLIGHT | FLIGHT TIME | STD | ATD | STA | STATUS | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | 10 Dec 2022 | Tokunoshima (TKN) | Kagoshima (KOJ) | JL3798 | — | 10:00 | — | 11:10 | Scheduled |
| 1 | 10 Dec 2022 | Amami (ASJ) | Tokunoshima (TKN) | JL3843 | — | 08:55 | — | 09:30 | Scheduled |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 58 | 03 Dec 2022 | Amami (ASJ) | Kagoshima (KOJ) | JL3724 | 0:56 | 01:45 | 02:02 | 02:50 | Landed 02:58 |
| 59 | 03 Dec 2022 | Kagoshima (KOJ) | Amami (ASJ) | JL3725 | 1:06 | 00:00 | 00:09 | 01:15 | Landed 01:14 |
CodePudding user response：
Try this
import requests
import pandas as pd
response = requests.get('https://www.flightradar24.com/data/aircraft/ja11jc', headers={'User-agent': 'Mozilla/5.0'})
df = pd.read_html(response.text)[0][:-1]
df = df.dropna(axis=1, how='all')
print(df.to_string(index=False))
OUTPUT:
JL3798 10 Dec 2022 - Scheduled STD 10:00 ATD — STA 11:10 FROM Tokunoshima (TKN) TO Kagoshima (KOJ) 10 Dec 2022 Tokunoshima (TKN) Kagoshima (KOJ) JL3798 — 10:00 — 11:10 Scheduled Play
JL3843 10 Dec 2022 - Scheduled STD 08:55 ATD — STA 09:30 FROM Amami (ASJ) TO Tokunoshima (TKN) 10 Dec 2022 Amami (ASJ) Tokunoshima (TKN) JL3843 — 08:55 — 09:30 Scheduled Play
JL3844 10 Dec 2022 - Scheduled STD 07:45 ATD — STA 08:15 FROM Tokunoshima (TKN) TO Amami (ASJ) 10 Dec 2022 Tokunoshima (TKN) Amami (ASJ) JL3844 — 07:45 — 08:15 Scheduled Play
JL3710 10 Dec 2022 - Scheduled STD 06:45 ATD — STA 07:15 FROM Okierabu (OKE) TO Tokunoshima (TKN) 10 Dec 2022 Okierabu (OKE) Tokunoshima (TKN) JL3710 — 06:45 — 07:15 Scheduled Play
JL3715 10 Dec 2022 - Estimated departure 05:25 STD 05:25 ATD — STA 06:15 FROM Okinawa (OKA) TO Okierabu (OKE) 10 Dec 2022 Okinawa (OKA) Okierabu (OKE) JL3715 — 05:25 — 06:15 Estimated departure 05:25 Play
JL3716 10 Dec 2022 - Scheduled STD 03:55 ATD — STA 04:45 FROM Okierabu (OKE) TO Okinawa (OKA) 10 Dec 2022 Okierabu (OKE) Okinawa (OKA) JL3716 — 03:55 — 04:45 Scheduled Play
JL3711 10 Dec 2022 - Scheduled STD 02:55 ATD — STA 03:25 FROM Tokunoshima (TKN) TO Okierabu (OKE) 10 Dec 2022 Tokunoshima (TKN) Okierabu (OKE) JL3711 — 02:55 — 03:25 Scheduled Play
JL3841 10 Dec 2022 - Scheduled STD 01:50 ATD — STA 02:25 FROM Amami (ASJ) TO Tokunoshima (TKN) 10 Dec 2022 Amami (ASJ) Tokunoshima (TKN) JL3841 — 01:50 — 02:25 Scheduled Play
JL3842 10 Dec 2022 - Scheduled STD 00:35 ATD — STA 01:05 FROM Tokunoshima (TKN) TO Amami (ASJ) 10 Dec 2022 Tokunoshima (TKN) Amami (ASJ) JL3842 — 00:35 — 01:05 Scheduled Play
JL3791 09 Dec 2022 - Estimated departure 22:45 STD 22:45 ATD — STA 00:05 FROM Kagoshima (KOJ) TO Tokunoshima (TKN) 09 Dec 2022 Kagoshima (KOJ) Tokunoshima (TKN) JL3791 — 22:45 — 00:05 Estimated departure 22:45 Play
JL3734 09 Dec 2022 0:57 Landed 09:38 STD 08:40 ATD 08:41 STA 09:45 FROM Amami (ASJ) TO Kagoshima (KOJ) 09 Dec 2022 Amami (ASJ) Kagoshima (KOJ) JL3734 0:57 08:40 08:41 09:45 Landed 09:38 KML CSV Play
JL3836 09 Dec 2022 0:15 Landed 07:52 STD 07:40 ATD 07:37 STA 08:00 FROM Kikai (KKX) TO Amami (ASJ) 09 Dec 2022 Kikai (KKX) Amami (ASJ) JL3836 0:15 07:40 07:37 08:00 Landed 07:52 KML CSV Play
JL3837 09 Dec 2022 0:11 Landed 07:00 STD 06:50 ATD 06:49 STA 07:10 FROM Amami (ASJ) TO Kikai (KKX) 09 Dec 2022 Amami (ASJ) Kikai (KKX) JL3837 0:11 06:50 06:49 07:10 Landed 07:00 KML CSV Play
JL3867 09 Dec 2022 0:46 Landed 05:45 STD 04:50 ATD 04:59 STA 05:50 FROM Okinawa (OKA) TO Amami (ASJ) 09 Dec 2022 Okinawa (OKA) Amami (ASJ) JL3867 0:46 04:50 04:59 05:50 Landed 05:45 KML CSV Play
JL3866 09 Dec 2022 0:30 Landed 04:15 STD 03:25 ATD 03:45 STA 04:10 FROM Yoronjima (RNJ) TO Okinawa (OKA) 09 Dec 2022 Yoronjima (RNJ) Okinawa (OKA) JL3866 0:30 03:25 03:45 04:10 Landed 04:15 KML CSV Play
JL3861 09 Dec 2022 0:35 Landed 02:59 STD 02:10 ATD 02:24 STA 02:55 FROM Amami (ASJ) TO Yoronjima (RNJ) 09 Dec 2022 Amami (ASJ) Yoronjima (RNJ) JL3861 0:35 02:10 02:24 02:55 Landed 02:59 KML CSV Play
JL3803 09 Dec 2022 - Unknown STD 01:12 ATD 01:12 STA 01:28 FROM Kikai (KKX) TO Amami (ASJ) 09 Dec 2022 Kikai (KKX) Amami (ASJ) JL3803 — 01:12 01:12 01:28 Unknown KML CSV Play
JL3830 09 Dec 2022 - Estimated departure 01:12 STD 01:10 ATD — STA 01:30 FROM Kikai (KKX) TO Amami (ASJ) 09 Dec 2022 Kikai (KKX) Amami (ASJ) JL3830 — 01:10 — 01:30 Estimated departure 01:12 Play
JL3831 09 Dec 2022 0:12 Landed 00:30 STD 00:20 ATD 00:18 STA 00:40 FROM Amami (ASJ) TO Kikai (KKX) 09 Dec 2022 Amami (ASJ) Kikai (KKX) JL3831 0:12 00:20 00:18 00:40 Landed 00:30 KML CSV Play
JL3721 08 Dec 2022 0:59 Landed 23:34 STD 22:25 ATD 22:35 STA 23:40 FROM Kagoshima (KOJ) TO Amami (ASJ) 08 Dec 2022 Kagoshima (KOJ) Amami (ASJ) JL3721 0:59 22:25 22:35 23:40 Landed 23:34 KML CSV Play
JL3734 08 Dec 2022 0:55 Landed 09:35 STD 08:40 ATD 08:41 STA 09:45 FROM Amami (ASJ) TO Kagoshima (KOJ) 08 Dec 2022 Amami (ASJ) Kagoshima (KOJ) JL3734 0:55 08:40 08:41 09:45 Landed 09:35 KML CSV Play
JL3836 08 Dec 2022 0:09 Landed 07:51 STD 07:40 ATD 07:42 STA 08:00 FROM Kikai (KKX) TO Amami (ASJ) 08 Dec 2022 Kikai (KKX) Amami (ASJ) JL3836 0:09 07:40 07:42 08:00 Landed 07:51 KML CSV Play
JL3837 08 Dec 2022 0:09 Landed 07:03 STD 06:50 ATD 06:54 STA 07:10 FROM Amami (ASJ) TO Kikai (KKX) 08 Dec 2022 Amami (ASJ) Kikai (KKX) JL3837 0:09 06:50 06:54 07:10 Landed 07:03 KML CSV Play
JL3867 08 Dec 2022 0:48 Landed 05:41 STD 04:50 ATD 04:53 STA 05:50 FROM Okinawa (OKA) TO Amami (ASJ) 08 Dec 2022 Okinawa (OKA) Amami (ASJ) JL3867 0:48 04:50 04:53 05:50 Landed 05:41 KML CSV Play
JL3866 08 Dec 2022 0:25 Landed 04:01 STD 03:25 ATD 03:36 STA 04:10 FROM Yoronjima (RNJ) TO Okinawa (OKA) 08 Dec 2022 Yoronjima (RNJ) Okinawa (OKA) JL3866 0:25 03:25 03:36 04:10 Landed 04:01 KML CSV Play
JL3861 08 Dec 2022 0:32 Landed 02:55 STD 02:10 ATD 02:24 STA 02:55 FROM Amami (ASJ) TO Yoronjima (RNJ) 08 Dec 2022 Amami (ASJ) Yoronjima (RNJ) JL3861 0:32 02:10 02:24 02:55 Landed 02:55 KML CSV Play
JL3830 08 Dec 2022 0:09 Landed 01:20 STD 01:10 ATD 01:11 STA 01:30 FROM Kikai (KKX) TO Amami (ASJ) 08 Dec 2022 Kikai (KKX) Amami (ASJ) JL3830 0:09 01:10 01:11 01:30 Landed 01:20 KML CSV Play
JL3831 08 Dec 2022 0:08 Landed 00:28 STD 00:20 ATD 00:20 STA 00:40 FROM Amami (ASJ) TO Kikai (KKX) 08 Dec 2022 Amami (ASJ) Kikai (KKX) JL3831 0:08 00:20 00:20 00:40 Landed 00:28 KML CSV Play
JL3721 07 Dec 2022 0:57 Landed 23:29 STD 22:25 ATD 22:31 STA 23:40 FROM Kagoshima (KOJ) TO Amami (ASJ) 07 Dec 2022 Kagoshima (KOJ) Amami (ASJ) JL3721 0:57 22:25 22:31 23:40 Landed 23:29 KML CSV Play
JL3772 07 Dec 2022 0:30 Landed 09:32 STD 09:00 ATD 09:02 STA 09:30 FROM Tanegashima (TNE) TO Kagoshima (KOJ) 07 Dec 2022 Tanegashima (TNE) Kagoshima (KOJ) JL3772 0:30 09:00 09:02 09:30 Landed 09:32 KML CSV Play
JL3777 07 Dec 2022 0:27 Landed 08:25 STD 07:50 ATD 07:58 STA 08:30 FROM Kagoshima (KOJ) TO Tanegashima (TNE) 07 Dec 2022 Kagoshima (KOJ) Tanegashima (TNE) JL3777 0:27 07:50 07:58 08:30 Landed 08:25 KML CSV Play
JL3784 07 Dec 2022 0:55 Landed 07:13 STD 06:05 ATD 06:18 STA 07:10 FROM Kikai (KKX) TO Kagoshima (KOJ) 07 Dec 2022 Kikai (KKX) Kagoshima (KOJ) JL3784 0:55 06:05 06:18 07:10 Landed 07:13 KML CSV Play
JL3785 07 Dec 2022 1:00 Landed 05:32 STD 04:25 ATD 04:31 STA 05:35 FROM Kagoshima (KOJ) TO Kikai (KKX) 07 Dec 2022 Kagoshima (KOJ) Kikai (KKX) JL3785 1:00 04:25 04:31 05:35 Landed 05:32 KML CSV Play
JL3762 07 Dec 2022 0:26 Landed 03:37 STD 03:10 ATD 03:10 STA 03:45 FROM Tanegashima (TNE) TO Kagoshima (KOJ) 07 Dec 2022 Tanegashima (TNE) Kagoshima (KOJ) JL3762 0:26 03:10 03:10 03:45 Landed 03:37 KML CSV Play
JL3763 07 Dec 2022 0:24 Landed 02:32 STD 02:00 ATD 02:08 STA 02:40 FROM Kagoshima (KOJ) TO Tanegashima (TNE) 07 Dec 2022 Kagoshima (KOJ) Tanegashima (TNE) JL3763 0:24 02:00 02:08 02:40 Landed 02:32 KML CSV Play
JL3780 07 Dec 2022 0:51 Landed 01:13 STD 00:15 ATD 00:22 STA 01:20 FROM Kikai (KKX) TO Kagoshima (KOJ) 07 Dec 2022 Kikai (KKX) Kagoshima (KOJ) JL3780 0:51 00:15 00:22 01:20 Landed 01:13 KML CSV Play
JL3783 06 Dec 2022 1:00 Landed 23:45 STD 22:35 ATD 22:45 STA 23:45 FROM Kagoshima (KOJ) TO Kikai (KKX) 06 Dec 2022 Kagoshima (KOJ) Kikai (KKX) JL3783 1:00 22:35 22:45 23:45 Landed 23:45 KML CSV Play
JL3772 06 Dec 2022 0:25 Landed 09:15 STD 09:00 ATD 08:50 STA 09:30 FROM Tanegashima (TNE) TO Kagoshima (KOJ) 06 Dec 2022 Tanegashima (TNE) Kagoshima (KOJ) JL3772 0:25 09:00 08:50 09:30 Landed 09:15 KML CSV Play
JL3777 06 Dec 2022 0:25 Landed 08:15 STD 07:50 ATD 07:50 STA 08:30 FROM Kagoshima (KOJ) TO Tanegashima (TNE) 06 Dec 2022 Kagoshima (KOJ) Tanegashima (TNE) JL3777 0:25 07:50 07:50 08:30 Landed 08:15 KML CSV Play
JL3784 06 Dec 2022 0:55 Landed 07:09 STD 06:05 ATD 06:14 STA 07:10 FROM Kikai (KKX) TO Kagoshima (KOJ) 06 Dec 2022 Kikai (KKX) Kagoshima (KOJ) JL3784 0:55 06:05 06:14 07:10 Landed 07:09 KML CSV Play
JL3785 06 Dec 2022 0:59 Landed 05:34 STD 04:25 ATD 04:35 STA 05:35 FROM Kagoshima (KOJ) TO Kikai (KKX) 06 Dec 2022 Kagoshima (KOJ) Kikai (KKX) JL3785 0:59 04:25 04:35 05:35 Landed 05:34 KML CSV Play
JL3762 06 Dec 2022 0:22 Landed 03:48 STD 03:10 ATD 03:26 STA 03:45 FROM Tanegashima (TNE) TO Kagoshima (KOJ) 06 Dec 2022 Tanegashima (TNE) Kagoshima (KOJ) JL3762 0:22 03:10 03:26 03:45 Landed 03:48 KML CSV Play
JL3763 06 Dec 2022 0:24 Landed 02:47 STD 02:00 ATD 02:23 STA 02:40 FROM Kagoshima (KOJ) TO Tanegashima (TNE) 06 Dec 2022 Kagoshima (KOJ) Tanegashima (TNE) JL3763 0:24 02:00 02:23 02:40 Landed 02:47 KML CSV Play
JL3780 06 Dec 2022 0:54 Landed 01:38 STD 00:15 ATD 00:44 STA 01:20 FROM Kikai (KKX) TO Kagoshima (KOJ) 06 Dec 2022 Kikai (KKX) Kagoshima (KOJ) JL3780 0:54 00:15 00:44 01:20 Landed 01:38 KML CSV Play
JL3783 05 Dec 2022 1:04 Landed 23:45 STD 22:35 ATD 22:41 STA 23:45 FROM Kagoshima (KOJ) TO Kikai (KKX) 05 Dec 2022 Kagoshima (KOJ) Kikai (KKX) JL3783 1:04 22:35 22:41 23:45 Landed 23:45 KML CSV Play
JL3686 05 Dec 2022 0:53 Landed 12:09 STD 11:10 ATD 11:16 STA 12:10 FROM Matsuyama (MYJ) TO Kagoshima (KOJ) 05 Dec 2022 Matsuyama (MYJ) Kagoshima (KOJ) JL3686 0:53 11:10 11:16 12:10 Landed 12:09 KML CSV Play
JL3687 05 Dec 2022 0:42 Landed 10:44 STD 09:45 ATD 10:02 STA 10:40 FROM Kagoshima (KOJ) TO Matsuyama (MYJ) 05 Dec 2022 Kagoshima (KOJ) Matsuyama (MYJ) JL3687 0:42 09:45 10:02 10:40 Landed 10:44 KML CSV Play
JL3808 05 Dec 2022 1:07 Landed 09:22 STD 07:50 ATD 08:15 STA 09:05 FROM Okierabu (OKE) TO Kagoshima (KOJ) 05 Dec 2022 Okierabu (OKE) Kagoshima (KOJ) JL3808 1:07 07:50 08:15 09:05 Landed 09:22 KML CSV Play
JL3809 05 Dec 2022 1:28 Landed 07:31 STD 05:55 ATD 06:03 STA 07:20 FROM Kagoshima (KOJ) TO Okierabu (OKE) 05 Dec 2022 Kagoshima (KOJ) Okierabu (OKE) JL3809 1:28 05:55 06:03 07:20 Landed 07:31 KML CSV Play
JL3785 05 Dec 2022 - Canceled STD 04:25 ATD — STA 05:35 FROM Kagoshima (KOJ) TO Kikai (KKX) 05 Dec 2022 Kagoshima (KOJ) Kikai (KKX) JL3785 — 04:25 — 05:35 Canceled Play
JL3762 05 Dec 2022 0:25 Landed 03:49 STD 03:10 ATD 03:24 STA 03:45 FROM Tanegashima (TNE) TO Kagoshima (KOJ) 05 Dec 2022 Tanegashima (TNE) Kagoshima (KOJ) JL3762 0:25 03:10 03:24 03:45 Landed 03:49 KML CSV Play
JL3763 05 Dec 2022 0:28 Landed 02:48 STD 02:00 ATD 02:20 STA 02:40 FROM Kagoshima (KOJ) TO Tanegashima (TNE) 05 Dec 2022 Kagoshima (KOJ) Tanegashima (TNE) JL3763 0:28 02:00 02:20 02:40 Landed 02:48 KML CSV Play
JL3780 05 Dec 2022 0:51 Landed 01:34 STD 00:15 ATD 00:42 STA 01:20 FROM Kikai (KKX) TO Kagoshima (KOJ) 05 Dec 2022 Kikai (KKX) Kagoshima (KOJ) JL3780 0:51 00:15 00:42 01:20 Landed 01:34 KML CSV Play
JL3783 04 Dec 2022 1:03 Landed 23:45 STD 22:35 ATD 22:42 STA 23:45 FROM Kagoshima (KOJ) TO Kikai (KKX) 04 Dec 2022 Kagoshima (KOJ) Kikai (KKX) JL3783 1:03 22:35 22:42 23:45 Landed 23:45 KML CSV Play
JL3464 04 Dec 2022 0:57 Landed 08:10 STD 07:10 ATD 07:13 STA 08:15 FROM Amami (ASJ) TO Kagoshima (KOJ) 04 Dec 2022 Amami (ASJ) Kagoshima (KOJ) JL3464 0:57 07:10 07:13 08:15 Landed 08:10 KML CSV Play
JL3465 04 Dec 2022 1:05 Landed 06:31 STD 05:25 ATD 05:26 STA 06:40 FROM Kagoshima (KOJ) TO Amami (ASJ) 04 Dec 2022 Kagoshima (KOJ) Amami (ASJ) JL3465 1:05 05:25 05:26 06:40 Landed 06:31 KML CSV Play
JL3724 04 Dec 2022 0:53 Landed 02:43 STD 01:45 ATD 01:50 STA 02:50 FROM Amami (ASJ) TO Kagoshima (KOJ) 04 Dec 2022 Amami (ASJ) Kagoshima (KOJ) JL3724 0:53 01:45 01:50 02:50 Landed 02:43 KML CSV Play
JL3725 04 Dec 2022 0:56 Landed 01:02 STD 00:00 ATD 00:05 STA 01:15 FROM Kagoshima (KOJ) TO Amami (ASJ) 04 Dec 2022 Kagoshima (KOJ) Amami (ASJ) JL3725 0:56 00:00 00:05 01:15 Landed 01:02 KML CSV Play
JL3724 03 Dec 2022 0:56 Landed 02:58 STD 01:45 ATD 02:02 STA 02:50 FROM Amami (ASJ) TO Kagoshima (KOJ) 03 Dec 2022 Amami (ASJ) Kagoshima (KOJ) JL3724 0:56 01:45 02:02 02:50 Landed 02:58 KML CSV Play
JL3725 03 Dec 2022 1:06 Landed 01:14 STD 00:00 ATD 00:09 STA 01:15 FROM Kagoshima (KOJ) TO Amami (ASJ) 03 Dec 2022 Kagoshima (KOJ) Amami (ASJ) JL3725 1:06 00:00 00:09 01:15 Landed 01:14 KML CSV Play