Is I want to grab the data, it is dynamic content, url returns the text inside do not have these data


Then I consult XHR, found a ManuArtNr intermediate page has returned json data,



The following basic ManuArtNr this page, request header, parameters such as



I wrote to grab the code according to these pages information, but did not return json data, but an error page, the content and I take parameters built directly in the browser to access the same url, the following


Below is I grab the code

MyId refererurl='https://web1.carparts-cat.com/Sub/GVS/' + + '/4/1/1/0///- 1-1-1/9992/? SupplierNumber=1368 & amp; SupplierArticleNumber=54927 & amp; GenArtNr=3324 & amp; VknId=0 & amp; HKatNr=0 & amp; CatalogType=0 '
Appliurl='https://web1.carparts-cat.com/SUB/GVS/ManuArtNr/'
Applidata={
https://bbs.csdn.net/topics/'sessionID' : MyId,
'artNr: PartID,
'typeArt: 1,
'genArtNr: SupID
}
Print (applidata)

AppliHeader={
'accept' : 'application/json, text/javascript, */*; Q=0.01 ',
'the accept - encoding' : 'gzip, deflate, br',
'the accept - language' : 'useful - CN, useful; Q=0.9 ',
'the content-type' : 'application/x - WWW - form - urlencoded',
'origin' : 'https://web1.carparts-cat.com',
='+' cookies' : '__cfduid resp. Cookies [' __cfduid] +'; HideBranding=16=0; __tawkuuid=e::web1.carparts-cat.com: : QoRXwsNWAMIqMY0456ZtfYYIWbLKUGCDtOuXxgKzepl4/qf7B8A6rdIkdC8f7PUS: : 2; SlidePaneOpened=1; TawkConnectionTime=0 ',
'referer: refererurl,
'the SEC - fetch - dest' : 'empty'
'the SEC - fetch - mode:' cors,
'the SEC - fetch - site' : 'the same - origin,
'the user-agent' : 'Mozilla/5.0 (Windows NT 6.1; Win64. X64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36 ',
'x - requested - with' : 'the XMLHttpRequest,
}
Resp5=session. Post (appliurl data=https://bbs.csdn.net/topics/applidata, headers=appliHeader)

There are several variables among the header and the data, it is from my previous search page to log in to get, in the process of the program to run, I output view, content is correct,

CodePudding user response:

Don't sink, this problem bothering me for a few days, check information everywhere, really didn't have the answer, seek help from bosses, are willing to pay a certain amount of remuneration,

CodePudding user response:

Can try to use the selenium grab, have window, then no window

CodePudding user response:

Home page url to the

CodePudding user response:

Haven't finished can see the effect, using selenium iframe extraction hasn't do

 

The import requests, time, threading 
The from the selenium import webdriver 
The from selenium.webdriver.com mon. Action_chains import ActionChains 
The from selenium.webdriver.com mon. Keys import keys 

CHROME_PATH="C:/Python/chromedriverV81. Exe" 

If __name__=="__main__ ': 

Uname="" 
Password="" 

Browser=webdriver. Chrome (CHROME_PATH) 
The get (" https://web1.carparts-cat.com/loginh.aspx? SID=016001 ") 
Time. Sleep (1) 
The find_element_by_id (" username "). Send_keys (uname) 
The find_element_by_id (" password "). Send_keys (password) 
The find_element_by_id (" login "). Click () 
Time. Sleep (1) 

The find_element_by_id (' ok '.) click () 
Time. Sleep (1) 

# id="home_txt_fzg_freitext" 
The find_element_by_id (' home_txt_fzg_freitext). Send_keys (" 036103464 ah ") 
The find_element_by_id (' home_fzg_freitext_imgBtn). Click () 

Action=ActionChains (browser) 
# the find_element_by_class_name (' menuBasketQuantity). Send_keys (Keys. CONTROL + "t") 
The find_element_by_class_name (' menuBasketQuantity). Click () 

CarManufacturers=the find_elements_by_class_name (" FilterByCarManufacturers ") 

Print (" before the iframe, "the page_source) 

Iframe0=the switch_to. Frame (the find_element_by_tag_name (" iframe ")) 
Print (" iframe0 iframe0) 

Print (the page_source) 
While (True) : 
Time. Sleep (20)