So i wanted to get all "b" tags with "price" class form this website, but soup.findAll return empty list with 0 elements. But when I try the same thing, but with local html file, everything just works fine.
I've searched all over the internet, but nothing seems to help.
The code itself:
from bs4 import BeautifulSoup
from urllib.request import Request, urlopen
url = 'https://warframe.market/items/nami_skyla_prime_blueprint'
req = Request(url, headers={'User-Agent': 'Mozilla/5.0'})
webpage = urlopen(req).read()
soup = BeautifulSoup(webpage, 'html.parser')
tags = soup.findAll('b')
print(tags)
CodePudding user response:
The data you see on the page is loaded from external URL via JavaScript (so beautifulsoup
doesn't see it). To load the data into pandas dataframe you can use this example:
import requests
import pandas as pd
api_url = (
"https://api.warframe.market/v1/items/nami_skyla_prime_blueprint/orders"
)
df = pd.DataFrame(requests.get(api_url).json()["payload"]["orders"])
df = pd.concat([df, df.pop("user").apply(pd.Series)], axis=1)
print(df.to_markdown(index=False))
Prints:
platform | region | order_type | creation_date | platinum | quantity | last_update | visible | id | reputation | region | avatar | last_seen | ingame_name | id | status |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
pc | en | sell | 2018-03-24T13:19:44.000 00:00 | 5 | 1 | 2018-03-24T13:19:44.000 00:00 | True | 5ab6507049ef000fcf1920a9 | 0 | en | 2022-07-30T18:00:20.943 00:00 | lastcrist | 62e56f91330ca40377b19895 | offline | |
pc | en | sell | 2019-01-10T18:21:06.000 00:00 | 14 | 1 | 2022-07-26T18:45:05.000 00:00 | True | 5c378d12af045700a4291477 | 3 | en | 2022-07-27T11:04:01.133 00:00 | kekxqt | 5a0628ddc611360011c7a66e | offline | |
pc | en | sell | 2019-12-09T07:11:43.000 00:00 | 35 | 1 | 2021-12-02T04:32:53.000 00:00 | True | 5dedf3afc72155005cf4028c | 968 | en | user/avatar/55f4e189b66f831701dbff77.png?0fe7b910face5429601e7085d289f869 | 2022-07-30T16:18:02.366 00:00 | Sahysa | 55f4e189b66f831701dbff77 | offline |
pc | en | sell | 2020-02-04T03:29:02.000 00:00 | 13 | 1 | 2022-07-29T23:22:45.000 00:00 | True | 5e38e4fe7b0275005e6d29a9 | 4 | en | 2022-07-30T06:40:38.150 00:00 | YellowSnowBoi | 5b3c268c0c3535012e9b6ee3 | offline | |
pc | en | sell | 2020-03-29T20:14:39.000 00:00 | 15 | 1 | 2022-07-24T18:14:30.000 00:00 | True | 5e8101aff3441505a6b292c6 | 19 | en | user/avatar/5b2931d4eb069f04986adb6d.png?822ece14d8951796338ef3739d5c6b2c | 2022-07-29T20:55:01.617 00:00 | xblq | 5b2931d4eb069f04986adb6d | offline |
pc | ko | sell | 2020-11-07T09:15:58.000 00:00 | 9 | 1 | 2020-11-07T09:15:58.000 00:00 | True | 5fa665ced5b8ca0262d04cf0 | 0 | ko | 2022-07-30T17:33:05.154 00:00 | blazespirit | 5a4380778c474d007384823b | ingame | |
pc | en | sell | 2021-02-07T12:42:27.000 00:00 | 15 | 1 | 2021-12-26T14:35:01.000 00:00 | True | 601fe033ec5e7e01d47859a1 | 3 | en | user/avatar/600fcf3dc59d3703156a15e3.png?5f9404dcf7c36b3200d2eaab5ed9dba9 | 2022-07-30T10:26:53.816 00:00 | pavelfizek | 600fcf3dc59d3703156a15e3 | offline |
pc | en | buy | 2021-02-12T14:34:49.000 00:00 | 10 | 6 | 2022-07-20T21:11:53.000 00:00 | True | 60269209039a2702632da531 | 412 | en | user/avatar/598d8bf2d3ffb648a838abba.png?dfe50b7091f0e8c44b63976a700eebd9 | 2022-07-30T10:02:13.214 00:00 | Xnidior | 598d8bf2d3ffb648a838abba | offline |
pc | en | sell | 2021-02-13T03:24:21.000 00:00 | 10 | 1 | 2022-07-23T06:48:55.000 00:00 | True | 602746651c9894026f386fb0 | 2 | en | 2022-07-28T02:31:25.058 00:00 | Reaper575 | 5cd904444532a609fef941db | offline | |
pc | en | sell | 2021-03-13T18:03:01.000 00:00 | 15 | 1 | 2022-07-22T14:35:07.000 00:00 | True | 604cfe55171b1903005aa375 | 23 | en | user/avatar/5c1cc13e6ad4a90015110a59.png?3eca6f848f62560068b7c0f17f141d6d | 2022-07-27T11:36:30.529 00:00 | -CG-Luthix | 5c1cc13e6ad4a90015110a59 | offline |
pc | ru | buy | 2021-04-03T05:56:22.000 00:00 | 5 | 1 | 2022-04-07T10:38:12.000 00:00 | True | 606803861a02f00165d2965c | 2 | ru | user/avatar/624ebb41596f2f00881ce522.png?c1556e2a317fe8dfbd7e86ba051e371a | 2022-07-27T14:19:19.788 00:00 | 1FioLet1 | 624ebb41596f2f00881ce522 | offline |
pc | en | sell | 2021-06-11T13:42:09.000 00:00 | 20 | 1 | 2021-06-11T13:42:09.000 00:00 | True | 60c368311970de00c952915f | 0 | en | 2022-07-30T02:14:54.187 00:00 | NEXUREX21 | 6039915229f04b0128def8fa | offline |
...