Can I extract JSON data from this XHR request?-CodePudding

I'm a beginner trying to webscrape on the section of this site https://www.csgoroll.com/en/withdraw/csgo/p2p I'm trying to extract the prices and name of each item in the request called "graphql?operationName=TradeList&variables=" soley using the requests library in Python but i'm unsure of how to do it. I've done some research on this and it lead me to use an app called Postman and if I copy the cURL to it, it doesn't return the JSON data. Further research it looks like it uses graphql but when I connect to https://api.csgoroll.com/ the playground which I think it used to query data says server cannot be reached. So i'm wondering if it's possible to to extract the data solely using the requests library in Python and if so how?

CodePudding user response：

It needs header User-Agent from real web browser or at least short 'Mozilla/5.0'.

requests as default uses something like python/3.x so server asks for resolving Captcha.

import requests

url = 'https://api.csgoroll.com/graphql'

headers = {
    'User-Agent': 'Mozilla/5.0',
#    'Accept': 'application/json, text/plain, */*',
}

params = {
    'operationName': 'TradeList',
    'variables': '{"first":50,"orderBy":"TOTAL_VALUE_DESC","status":"LISTED","steamAppName":"CSGO"}',
    'extensions': '{"persistedQuery":{"version":1,"sha256Hash":"87239fc5fa143cf0437964a20937aa558145cc8385eae48ca8734cb16abfd266"}}',
}

r = requests.get(url, headers=headers, params=params)
#print(r.text)

data = r.json()
trades = data['data']['trades']['edges']

for trade in trades:
    #print(item)
    for item in trade['node']['tradeItems']:
        print(item['marketName'])

Result:

★ Sport Gloves | Pandora's Box (Factory New)
★ StatTrak™ M9 Bayonet | Crimson Web (Factory New)
★ StatTrak™ Karambit | Crimson Web (Factory New)
★ Butterfly Knife | Sapphire (Factory New)
★ Butterfly Knife | Sapphire (Factory New)
Sticker | HellRaisers (Holo) | Katowice 2014
AWP | Dragon Lore (Factory New)
★ StatTrak™ Karambit | Emerald (Factory New)
★ Butterfly Knife | Ruby (Factory New)
★ M9 Bayonet | Emerald (Minimal Wear)
StatTrak™ M4A4 | Howl (Field-Tested)
★ Paracord Knife | Crimson Web (Factory New)
★ Paracord Knife | Crimson Web (Factory New)
★ Moto Gloves | Smoke Out (Factory New)
M4A1-S | Welcome to the Jungle (Factory New)
AWP | Dragon Lore (Battle-Scarred)
AWP | Desert Hydra (Factory New)
★ Sport Gloves | Slingshot (Minimal Wear)
★ Sport Gloves | Slingshot (Minimal Wear)
★ Classic Knife | Crimson Web (Factory New)
Sticker | compLexity Gaming (Holo) | Katowice 2014
AK-47 | Wild Lotus (Battle-Scarred)
AK-47 | Wild Lotus (Battle-Scarred)
StatTrak™ P90 | Emerald Dragon (Factory New)
StatTrak™ P90 | Emerald Dragon (Factory New)
★ StatTrak™ Skeleton Knife | Case Hardened (Minimal Wear)
★ Hand Wraps | Constrictor (Factory New)
Sticker | ESL Wolf (Foil) | Katowice 2014
Sticker | ESL Skull (Foil) | Katowice 2014
★ StatTrak™ Gut Knife | Ultraviolet (Battle-Scarred)
MP9 | Dark Age (Minimal Wear)
Souvenir P90 | Fallout Warning (Factory New)
Souvenir P250 | Gunsmoke (Factory New)
Sticker | Aerial (Gold) | Katowice 2019
StatTrak™ M4A4 | In Living Color (Battle-Scarred)
StatTrak™ M4A4 | In Living Color (Battle-Scarred)
StatTrak™ M4A4 | In Living Color (Battle-Scarred)
Sticker | TYLOO (Gold) | 2020 RMR
Sticker | yuurih (Gold) | Berlin 2019
Sticker | TYLOO (Gold) | 2020 RMR
Sticker | Natus Vincere (Gold) | 2020 RMR
Sticker | TYLOO (Gold) | 2020 RMR
Sticker | TYLOO (Gold) | 2020 RMR
M4A4 | Neo-Noir (Field-Tested)
M4A4 | Neo-Noir (Field-Tested)
StatTrak™ AWP | Atheris (Field-Tested)
CZ75-Auto | Tuxedo (Minimal Wear)
Sealed Graffiti | G2 Esports | Atlanta 2017
StatTrak™ Desert Eagle | Bronze Deco (Minimal Wear)
Sealed Graffiti | Astralis | Atlanta 2017