I get HTML code tag style with BeautifulSoup instead of the rest of HTML code (divs etc)-CodePudding

I am trying to get prices data from Binance.
So while I am trying to make it, I go through HTML divs and classes to make it step by step.
Here is the example:

import requests
from bs4 import BeautifulSoup
from fake_useragent import UserAgent

user_agent = UserAgent(verify_ssl=False)
headers = {'user-agent': user_agent.random}

url = 'https://p2p.binance.com/ru/trade/all-payments/USDT?fiat=RUB'
res = requests.get(url, headers=headers).text
soup = BeautifulSoup(res, 'lxml')

lst = soup.find('div', class_="css-186r813")
underlst = lst.find('div', class_="css-cjwhpx")

print(underlst)

But suddenly it return not CSS code, but HTML code with tag style.
Here is the output:

<div ><style data-emotion-css="62wjvl">.css-62wjvl{box-sizing:border-box;margin:0;min-width:0;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;width:100%;padding-top:80px;padding-bottom:80px;}</style><div ><style data-emotion-css="14f6ssc">.css-14f6ssc{box-sizing:border-box;margin:0;min-width:0;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;width:30px;height:30px;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;-webkit-box-pack:justify;-webkit-justify-content:space-between;-ms-flex-pack:justify;justify-content:space-between;width:48px;height:48px;margin:auto;}</style><div ><style data-emotion-css="2bc2yz animation-hqgv56">.css-2bc2yz{box-sizing:border-box;margin:0;min-width:0;background-color:#F0B90B;height:100%;width:3px;-webkit-animation:animation-hqgv56 1.2s infinite ease-in-out;animation:animation-hqgv56 1.2s infinite ease-in-out;-webkit-animation-delay:-0.3s;animation-delay:-0.3s;}@-webkit-keyframes animation-hqgv56{0%,40%,100%{-webkit-transform:scaleY(0.4);-ms-transform:scaleY(0.4);transform:scaleY(0.4);}20%{-webkit-transform:scaleY(1.0);-ms-transform:scaleY(1.0);transform:scaleY(1.0);}}@keyframes animation-hqgv56{0%,40%,100%{-webkit-transform:scaleY(0.4);-ms-transform:scaleY(0.4);transform:scaleY(0.4);}20%{-webkit-transform:scaleY(1.0);-ms-transform:scaleY(1.0);transform:scaleY(1.0);}}</style><div ></div><style data-emotion-css="i6pav4 animation-hqgv56">.css-i6pav4{box-sizing:border-box;margin:0;min-width:0;background-color:#F0B90B;height:100%;width:3px;-webkit-animation:animation-hqgv56 1.2s infinite ease-in-out;animation:animation-hqgv56 1.2s infinite ease-in-out;-webkit-animation-delay:-0.2s;animation-delay:-0.2s;}@-webkit-keyframes animation-hqgv56{0%,40%,100%{-webkit-transform:scaleY(0.4);-ms-transform:scaleY(0.4);transform:scaleY(0.4);}20%{-webkit-transform:scaleY(1.0);-ms-transform:scaleY(1.0);transform:scaleY(1.0);}}@keyframes animation-hqgv56{0%,40%,100%{-webkit-transform:scaleY(0.4);-ms-transform:scaleY(0.4);transform:scaleY(0.4);}20%{-webkit-transform:scaleY(1.0);-ms-transform:scaleY(1.0);transform:scaleY(1.0);}}</style><div ></div><style data-emotion-css="o1uf9l animation-hqgv56">.css-o1uf9l{box-sizing:border-box;margin:0;min-width:0;background-color:#F0B90B;height:100%;width:3px;-webkit-animation:animation-hqgv56 1.2s infinite ease-in-out;animation:animation-hqgv56 1.2s infinite ease-in-out;-webkit-animation-delay:-0.1s;animation-delay:-0.1s;}@-webkit-keyframes animation-hqgv56{0%,40%,100%{-webkit-transform:scaleY(0.4);-ms-transform:scaleY(0.4);transform:scaleY(0.4);}20%{-webkit-transform:scaleY(1.0);-ms-transform:scaleY(1.0);transform:scaleY(1.0);}}@keyframes animation-hqgv56{0%,40%,100%{-webkit-transform:scaleY(0.4);-ms-transform:scaleY(0.4);transform:scaleY(0.4);}20%{-webkit-transform:scaleY(1.0);-ms-transform:scaleY(1.0);transform:scaleY(1.0);}}</style><div ></div><style data-emotion-css="14jhrrk animation-hqgv56">.css-14jhrrk{box-sizing:border-box;margin:0;min-width:0;background-color:#F0B90B;height:100%;width:3px;-webkit-animation:animation-hqgv56 1.2s infinite ease-in-out;animation:animation-hqgv56 1.2s infinite ease-in-out;-webkit-animation-delay:-0s;animation-delay:-0s;}@-webkit-keyframes animation-hqgv56{0%,40%,100%{-webkit-transform:scaleY(0.4);-ms-transform:scaleY(0.4);transform:scaleY(0.4);}20%{-webkit-transform:scaleY(1.0);-ms-transform:scaleY(1.0);transform:scaleY(1.0);}}@keyframes animation-hqgv56{0%,40%,100%{-webkit-transform:scaleY(0.4);-ms-transform:scaleY(0.4);transform:scaleY(0.4);}20%{-webkit-transform:scaleY(1.0);-ms-transform:scaleY(1.0);transform:scaleY(1.0);}}</style><div ></div></div></div></div>

But I expect the rest of HTML code (I mean other divs as I see it when I am inspecting the HTML code).
Could you tell me please, what am I doing wrong?

CodePudding user response：

The following is one way to obtain that data - based on inspecting Dev tools and looking at APIs being accessed & their response:

import requests
import pandas as pd

s = requests.Session()

headers = {
    'content-type': 'application/json',
    'accept-language': 'en-US,en;q=0.9',
    'User-Agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"
    }

payload = '{"proMerchantAds":false,"page":1,"rows":10,"payTypes":[],"countries":[],"publisherType":null,"asset":"USDT","fiat":"RUB","tradeType":"BUY"}'

url = 'https://p2p.binance.com/bapi/c2c/v2/friendly/c2c/adv/search'

s.headers.update(headers)
r = s.post(url, data=payload)
df = pd.json_normalize(r.json()['data'])
print(df)

This returns in terminal:

adv.advNo   adv.classify    adv.tradeType   adv.asset   adv.fiatUnit    adv.advStatus   adv.priceType   adv.priceFloatingRatio  adv.rateFloatingRatio   adv.currencyRate    adv.price   adv.initAmount  adv.surplusAmount   adv.amountAfterEditing  adv.maxSingleTransAmount    adv.minSingleTransAmount    adv.buyerKycLimit   adv.buyerRegDaysLimit   adv.buyerBtcPositionLimit   adv.remarks adv.autoReplyMsg    adv.payTimeLimit    adv.tradeMethods    adv.userTradeCountFilterTime    adv.userBuyTradeCountMin    adv.userBuyTradeCountMax    adv.userSellTradeCountMin   adv.userSellTradeCountMax   adv.userAllTradeCountMin    adv.userAllTradeCountMax    adv.userTradeCompleteRateFilterTime adv.userTradeCompleteCountMin   adv.userTradeCompleteRateMin    adv.userTradeVolumeFilterTime   adv.userTradeType   adv.userTradeVolumeMin  adv.userTradeVolumeMax  adv.userTradeVolumeAsset    adv.createTime  adv.advUpdateTime   adv.fiatVo  adv.assetVo adv.advVisibleRet   adv.assetLogo   adv.assetScale  adv.fiatScale   adv.priceScale  adv.fiatSymbol  adv.isTradable  adv.dynamicMaxSingleTransAmount adv.minSingleTransQuantity  adv.maxSingleTransQuantity  adv.dynamicMaxSingleTransQuantity   adv.tradableQuantity    adv.commissionRate  adv.tradeMethodCommissionRates  adv.launchCountry   adv.abnormalStatusList  adv.closeReason advertiser.userNo   advertiser.realName advertiser.nickName advertiser.margin   advertiser.marginUnit   advertiser.orderCount   advertiser.monthOrderCount  advertiser.monthFinishRate  advertiser.advConfirmTime   advertiser.email    advertiser.registrationTime advertiser.mobile   advertiser.userType advertiser.tagIconUrls  advertiser.userGrade    advertiser.userIdentity advertiser.proMerchant  advertiser.isBlocked
0   11403393152657350656    mass    SELL    USDT    RUB None    None    None    None    None    59.69   None    299.57  None    5001.00 5000.00 None    None    None    None        None    [{'payId': None, 'payMethodId': '', 'payType': None, 'payAccount': None, 'payBank': None, 'paySubBank': None, 'identifier': 'Payeer', 'iconUrlColor': None, 'tradeMethodName': 'Payeer', 'tradeMethodShortName': 'Payeer', 'tradeMethodBgColor': '#03A9F4'}]    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    2   2   2   ₽   True    5001.00 83.76   83.78   83.78   299.27  0.00100000  []  None    None    None    s987ef83f7c373b2d939960f36abde794   None    Kripto-Monah    None    None    None    427 0.982   None    None    None    None    user    []  2       None    None
1   11384733631785136128    mass    SELL    USDT    RUB None    None    None    None    None    59.97   None    70.13   None    5000.00 3000.00 None    None    None    None        None    [{'payId': None, 'payMethodId': '', 'payType': None, 'payAccount': None, 'payBank': None, 'paySubBank': None, 'identifier': 'Payeer', 'iconUrlColor': None, 'tradeMethodName': 'Payeer', 'tradeMethodShortName': 'Payeer', 'tradeMethodBgColor': '#03A9F4'}, {'payId': None, 'payMethodId': '', 'payType': None, 'payAccount': None, 'payBank': None, 'paySubBank': None, 'identifier': 'Advcash', 'iconUrlColor': None, 'tradeMethodName': 'Advcash', 'tradeMethodShortName': 'Advcash', 'tradeMethodBgColor': '#00B27A'}] None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    2   2   2   ₽   True    4201.49 50.02   83.37   70.05   70.05   0.00100000  []  None    None    None    s6ed17a60bae438359be89c30fb44e546   None    small-bank  None    None    None    362 1.000   None    None    None    None    user    []  2       None    None
2   11404898262677299200    mass    SELL    USDT    RUB None    None    None    None    None    60.00   None    1626.84 None    30000.00    3000.00 None    None    None    None        None    [{'payId': None, 'payMethodId': '', 'payType': None, 'payAccount': None, 'payBank': None, 'paySubBank': None, 'identifier': 'Payeer', 'iconUrlColor': None, 'tradeMethodName': 'Payeer', 'tradeMethodShortName': 'Payeer', 'tradeMethodBgColor': '#03A9F4'}]    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    2   2   2   ₽   True    30000.00    50.00   500.00  500.00  1625.21 0.00100000  []  None    None    None    sa3a5876f166e3283b000717ed3e5193a   None    solo    None    None    None    103 1.000   None    None    None    None    user    []  2       None    None
3   11405436724645101568    mass    SELL    USDT    RUB None    None    None    None    None    60.07   None    88.69   None    3000.00 500.00  None    None    None    None        None    [{'payId': None, 'payMethodId': '', 'payType': None, 'payAccount': None, 'payBank': None, 'paySubBank': None, 'identifier': 'RUBfiatbalance', 'iconUrlColor': None, 'tradeMethodName': 'BinancePay (RUB)', 'tradeMethodShortName': 'RUB fiat balance', 'tradeMethodBgColor': '#D89F00'}]    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    2   2   2   ₽   True    3000.00 8.32    49.94   49.94   88.60   0.00100000  []  None    None    None    s670441da55ca382ba2562649864e1d7a   None    Goodprice1  None    None    None    8   1.000   None    None    None    None    user    []  2       None    None
4   11405109823435702272    mass    SELL    USDT    RUB None    None    None    None    None    60.18   None    372.56  None    870000.00   3000.00 None    None    None    None        None    [{'payId': None, 'payMethodId': '', 'payType': None, 'payAccount': None, 'payBank': None, 'paySubBank': None, 'identifier': 'RUBfiatbalance', 'iconUrlColor': None, 'tradeMethodName': 'BinancePay (RUB)', 'tradeMethodShortName': 'RUB fiat balance', 'tradeMethodBgColor': '#D89F00'}, {'payId': None, 'payMethodId': '', 'payType': None, 'payAccount': None, 'payBank': None, 'paySubBank': None, 'identifier': 'Payeer', 'iconUrlColor': None, 'tradeMethodName': 'Payeer', 'tradeMethodShortName': 'Payeer', 'tradeMethodBgColor': '#03A9F4'}, {'payId': None, 'payMethodId': '', 'payType': None, 'payAccount': None, 'payBank': None, 'paySubBank': None, 'identifier': 'Advcash', 'iconUrlColor': None, 'tradeMethodName': 'Advcash', 'tradeMethodShortName': 'Advcash', 'tradeMethodBgColor': '#00B27A'}] None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    2   2   2   ₽   True    22398.26    49.85   14456.63    372.18  372.18  0.00100000  []  None    None    None    s9d93b43f0c4d3978bebe853f492fe1a4   None    AirSeller   None    None    None    342 0.998   None    None    None    None    user    []  2       None    None
5   11405456453578379264    mass    SELL    USDT    RUB None    None    None    None    None    60.20   None    162.08  None    4000.00 500.00  None    None    None    None        None    [{'payId': None, 'payMethodId': '', 'payType': None, 'payAccount': None, 'payBank': None, 'paySubBank': None, 'identifier': 'RUBfiatbalance', 'iconUrlColor': None, 'tradeMethodName': 'BinancePay (RUB)', 'tradeMethodShortName': 'RUB fiat balance', 'tradeMethodBgColor': '#D89F00'}]    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    2   2   2   ₽   True    4000.00 8.30    66.44   66.44   161.91  0.00100000  []  None    None    None    sa9086873626d3c15ad73b6272c83e6a1   None    Kurbanov    None    None    None    165 1.000   None    None    None    None    user    []  2       None    None
6   11403640005146021888    mass    SELL    USDT    RUB None    None    None    None    None    60.23   None    225.99  None    15000.00    500.00  None    None    None    None        None    [{'payId': None, 'payMethodId': '', 'payType': None, 'payAccount': None, 'payBank': None, 'paySubBank': None, 'identifier': 'Payeer', 'iconUrlColor': None, 'tradeMethodName': 'Payeer', 'tradeMethodShortName': 'Payeer', 'tradeMethodBgColor': '#03A9F4'}]    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    2   2   2   ₽   True    13597.77    8.30    249.04  225.76  225.76  0.00100000  []  None    None    None    s6a454028703b3d28ab1cbc882fe30609   None    OVN_SKS None    None    None    79  0.964   None    None    None    None    user    []  2       None    None
7   11404121800352935936    mass    SELL    USDT    RUB None    None    None    None    None    60.25   None    375.92  None    870000.00   1000.00 None    None    None    None        None    [{'payId': None, 'payMethodId': '', 'payType': None, 'payAccount': None, 'payBank': None, 'paySubBank': None, 'identifier': 'Payeer', 'iconUrlColor': None, 'tradeMethodName': 'Payeer', 'tradeMethodShortName': 'Payeer', 'tradeMethodBgColor': '#03A9F4'}, {'payId': None, 'payMethodId': '', 'payType': None, 'payAccount': None, 'payBank': None, 'paySubBank': None, 'identifier': 'RUBfiatbalance', 'iconUrlColor': None, 'tradeMethodName': 'BinancePay (RUB)', 'tradeMethodShortName': 'RUB fiat balance', 'tradeMethodBgColor': '#D89F00'}, {'payId': None, 'payMethodId': '', 'payType': None, 'payAccount': None, 'payBank': None, 'paySubBank': None, 'identifier': 'Advcash', 'iconUrlColor': None, 'tradeMethodName': 'Advcash', 'tradeMethodShortName': 'Advcash', 'tradeMethodBgColor': '#00B27A'}] None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    2   2   2   ₽   True    22626.55    16.59   14439.83    375.54  375.54  0.00100000  []  None    None    None    sc0fc469822e13c05adf72829988742d9   None    ILKABU  None    None    None    137 0.993   None    None    None    None    user    []  2       None    None
8   11403137594511839232    mass    SELL    USDT    RUB None    None    None    None    None    60.27   None    781.46  None    40000.00    10000.00    None    None    None    None        None    [{'payId': None, 'payMethodId': '', 'payType': None, 'payAccount': None, 'payBank': None, 'paySubBank': None, 'identifier': 'RUBfiatbalance', 'iconUrlColor': None, 'tradeMethodName': 'BinancePay (RUB)', 'tradeMethodShortName': 'RUB fiat balance', 'tradeMethodBgColor': '#D89F00'}]    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    2   2   2   ₽   True    40000.00    165.92  663.68  663.68  780.67  0.00100000  []  None    None    None    sa4e26204141639948cd02fdb14389606   None    valeriya_obmen  None    None    None    241 1.000   None    None    None    None    user    []  2       None    None
9   11393521586032951296    mass    SELL    USDT    RUB None    None    None    None    None    60.32   None    842.67  None    150000.00   2000.00 None    None    None    None        None    [{'payId': None, 'payMethodId': '', 'payType': None, 'payAccount': None, 'payBank': None, 'paySubBank': None, 'identifier': 'RUBfiatbalance', 'iconUrlColor': None, 'tradeMethodName': 'BinancePay (RUB)', 'tradeMethodShortName': 'RUB fiat balance', 'tradeMethodBgColor': '#D89F00'}]    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    None    2   2   2   ₽   True    50779.07    33.15   2486.73 841.82  841.82  0.00100000  []  None    None    None    sd01eb1e9b3523bf1977dd3411f1e5c1e   None    Julia_P2P_  None    None    None    100 1.000   None    None    None    None    user    []  2       None    None

This data is extensive: you want to drill down into that json object/dataframe, and slice& dice it to get only what you need.

Relevant documentation for Requests: https://requests.readthedocs.io/en/latest/

Also, relevant documentation for pandas: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.json_normalize.html