Getting values from one specific table from a website-CodePudding

I am trying to get the average price from

When I use the following code, I just get content from the first table at the top of the URL, the able underneath "Tournament Eligibility".

import pandas as pd
from bs4 import BeautifulSoup
import requests

url = 'https://yugiohprices.com/card_price?name=Triple Tactics Talent'
r = requests.get(url)
soup = BeautifulSoup(r.text, 'lxml')
table = soup.findAll('table',{'id':'item_stats'})
print(table)

Is it that my code is not waiting long enough for the entire website to load? I tried waiting for 5 seconds after I request the url, but it still shows the first table at the top. Any suggestions for how to get this value?

CodePudding user response：

What happens?

Site is dynamically loading the content you like t get - Cause requests could not simulate browser behavior, it do not work to wait?

How to fix?

Inormation comes from another url, you can choose this:

import pandas as pd
from bs4 import BeautifulSoup
import requests
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36'}
url = 'https://yugiohprices.com/get_card_prices/Triple Tactics Talent'
r = requests.get(url, headers= headers)
soup = BeautifulSoup(r.text, 'lxml')
soup.select('table#item_stats tr:nth-of-type(3) td')[1].text

Alternativ and maybe better solution if you like to go deeper is selenium so you can simulate a browser.