Python request.get() not working as intended-CodePudding

I'm trying to scrape some data from https://p2p.binance.com/en/trade/all-payments/USDT?fiat=AUD

!pip install bs4
from lxml import lxml 
import requests
from bs4 import BeautifulSoup
url = 'https://p2p.binance.com/en/trade/all-payments/USDT?fiat=AUD'
requests.get(url)
response = requests.get(url)
soup = BeautifulSoup(response.content, 'html')

after that

soup.find('div',attrs={'class' : 'css-1m1f8hn'}) shows nothing. This is just one of many items I'd like to scrape.

How can I fix that?

CodePudding user response：

You should use find_all with class_ argument: https://www.crummy.com/software/BeautifulSoup/bs4/doc/#searching-by-css-class

CodePudding user response：

Try this:

import requests
from pprint import pprint

link = 'https://p2p.binance.com/bapi/c2c/v2/friendly/c2c/adv/search'

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36',
}
payload = {"proMerchantAds":False,"page":1,"rows":10,"payTypes":[],"countries":[],"publisherType":None,"asset":"USDT","fiat":"AUD","tradeType":"BUY"}

with requests.Session() as s:
    s.headers.update(headers)
    res = s.post(link,json=payload)
    for item in res.json()['data']:
        print(item['adv']['price'])

Prints:

1.59
1.60
1.61
1.62
1.63
1.63
1.64
1.64
1.66
1.66

CodePudding user response：

It will be little difficult to scrap exactly this element because it loads by js after the page loads. It means that there will no target element in html page that you receive with get request.

You can check it out like that:

url = 'https://p2p.binance.com/en/trade/all-payments/USDT?fiat=AUD'
response = requests.get(url)
open('test.html', 'wb').write(responce.content)

and then open html with browser to see raw page from server.

You can look for requests to internal binance api with browser devtools in network tab. Then there is a way to repeat this request from python code and retreive target data

upd: that's what robots.txt did in his answer