Home > database >  BeautifulSoup findAll return empty list
BeautifulSoup findAll return empty list

Time:08-01

So i wanted to get all "b" tags with "price" class form this website, but soup.findAll return empty list with 0 elements. But when I try the same thing, but with local html file, everything just works fine.

I've searched all over the internet, but nothing seems to help.

The code itself:

from bs4 import BeautifulSoup
from urllib.request import Request, urlopen

url = 'https://warframe.market/items/nami_skyla_prime_blueprint'

req = Request(url, headers={'User-Agent': 'Mozilla/5.0'})
webpage = urlopen(req).read()
soup = BeautifulSoup(webpage, 'html.parser')
tags = soup.findAll('b')
print(tags)

CodePudding user response:

The data you see on the page is loaded from external URL via JavaScript (so beautifulsoup doesn't see it). To load the data into pandas dataframe you can use this example:

import requests
import pandas as pd

api_url = (
    "https://api.warframe.market/v1/items/nami_skyla_prime_blueprint/orders"
)

df = pd.DataFrame(requests.get(api_url).json()["payload"]["orders"])
df = pd.concat([df, df.pop("user").apply(pd.Series)], axis=1)
print(df.to_markdown(index=False))

Prints:

platform region order_type creation_date platinum quantity last_update visible id reputation region avatar last_seen ingame_name id status
pc en sell 2018-03-24T13:19:44.000 00:00 5 1 2018-03-24T13:19:44.000 00:00 True 5ab6507049ef000fcf1920a9 0 en 2022-07-30T18:00:20.943 00:00 lastcrist 62e56f91330ca40377b19895 offline
pc en sell 2019-01-10T18:21:06.000 00:00 14 1 2022-07-26T18:45:05.000 00:00 True 5c378d12af045700a4291477 3 en 2022-07-27T11:04:01.133 00:00 kekxqt 5a0628ddc611360011c7a66e offline
pc en sell 2019-12-09T07:11:43.000 00:00 35 1 2021-12-02T04:32:53.000 00:00 True 5dedf3afc72155005cf4028c 968 en user/avatar/55f4e189b66f831701dbff77.png?0fe7b910face5429601e7085d289f869 2022-07-30T16:18:02.366 00:00 Sahysa 55f4e189b66f831701dbff77 offline
pc en sell 2020-02-04T03:29:02.000 00:00 13 1 2022-07-29T23:22:45.000 00:00 True 5e38e4fe7b0275005e6d29a9 4 en 2022-07-30T06:40:38.150 00:00 YellowSnowBoi 5b3c268c0c3535012e9b6ee3 offline
pc en sell 2020-03-29T20:14:39.000 00:00 15 1 2022-07-24T18:14:30.000 00:00 True 5e8101aff3441505a6b292c6 19 en user/avatar/5b2931d4eb069f04986adb6d.png?822ece14d8951796338ef3739d5c6b2c 2022-07-29T20:55:01.617 00:00 xblq 5b2931d4eb069f04986adb6d offline
pc ko sell 2020-11-07T09:15:58.000 00:00 9 1 2020-11-07T09:15:58.000 00:00 True 5fa665ced5b8ca0262d04cf0 0 ko 2022-07-30T17:33:05.154 00:00 blazespirit 5a4380778c474d007384823b ingame
pc en sell 2021-02-07T12:42:27.000 00:00 15 1 2021-12-26T14:35:01.000 00:00 True 601fe033ec5e7e01d47859a1 3 en user/avatar/600fcf3dc59d3703156a15e3.png?5f9404dcf7c36b3200d2eaab5ed9dba9 2022-07-30T10:26:53.816 00:00 pavelfizek 600fcf3dc59d3703156a15e3 offline
pc en buy 2021-02-12T14:34:49.000 00:00 10 6 2022-07-20T21:11:53.000 00:00 True 60269209039a2702632da531 412 en user/avatar/598d8bf2d3ffb648a838abba.png?dfe50b7091f0e8c44b63976a700eebd9 2022-07-30T10:02:13.214 00:00 Xnidior 598d8bf2d3ffb648a838abba offline
pc en sell 2021-02-13T03:24:21.000 00:00 10 1 2022-07-23T06:48:55.000 00:00 True 602746651c9894026f386fb0 2 en 2022-07-28T02:31:25.058 00:00 Reaper575 5cd904444532a609fef941db offline
pc en sell 2021-03-13T18:03:01.000 00:00 15 1 2022-07-22T14:35:07.000 00:00 True 604cfe55171b1903005aa375 23 en user/avatar/5c1cc13e6ad4a90015110a59.png?3eca6f848f62560068b7c0f17f141d6d 2022-07-27T11:36:30.529 00:00 -CG-Luthix 5c1cc13e6ad4a90015110a59 offline
pc ru buy 2021-04-03T05:56:22.000 00:00 5 1 2022-04-07T10:38:12.000 00:00 True 606803861a02f00165d2965c 2 ru user/avatar/624ebb41596f2f00881ce522.png?c1556e2a317fe8dfbd7e86ba051e371a 2022-07-27T14:19:19.788 00:00 1FioLet1 624ebb41596f2f00881ce522 offline
pc en sell 2021-06-11T13:42:09.000 00:00 20 1 2021-06-11T13:42:09.000 00:00 True 60c368311970de00c952915f 0 en 2022-07-30T02:14:54.187 00:00 NEXUREX21 6039915229f04b0128def8fa offline

...

  • Related