Home > Net >  How to POST JSON with data payload and header with Python Requests?
How to POST JSON with data payload and header with Python Requests?

Time:03-11

I am trying to do some scraping from websites using GET and POST methods, but now I am facing a new challenge.

I am trying to get data from a credit simulator, I found this portuguese site (https://www.bancomontepio.pt/particulares/credito/pessoal/credito-pessoal-online).

As far as I know, I need to use POST method, but I have to specify the data (the Amount value, the Term...). I usually do it by creating a dictionary structure but that is not working.

I'm kinda lost to be fair, maybe the problem is on the header...

Here is my code:

import requests

import warnings
warnings.filterwarnings("ignore")

term=24
amount=5000

url = 'https://simuladores.bancomontepio.pt/ITSCredit.External/Calculator/ITSCredit.Calculator.UI.External/gateway/Calculator/api/Calculator/Calculate?hash=-1359629931'

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.122 Safari/537.36',
'Accept-Language': 'pt-PT,pt;q=0.9,en-US;q=0.8,en;q=0.7'}

payload = {'Amount': amount,'Term': term,'ProductCode':"26B1129900X"}

response = requests.post(url, headers=headers, data=payload, verify=False).json()

If i take off the .json(), I get the error Response [410].

The goal is to get the TAN or TAEG that change when term ("Prazo") or amount ("Montante") values change.

Any ideias?

[EDIT]

headers = {'Accept': 'application/json, text/plain, */*' ,
'Accept-Encoding': 'gzip, deflate, br',
'Accept-Language': 'pt-PT,pt;q=0.9,en-US;q=0.8,en;q=0.7',
'Connection': 'keep-alive',
'Content-Length': '957',
'content-type': 'text/plain',
'Cookie': '_gcl_au=1.1.911606195.1646064658; OptanonAlertBoxClosed=2022-02-28T16:45:11.586Z; _ga=GA1.2.1147601977.1646064657; _ga_8WVEJF7X11=GS1.1.1646305654.3.1.1646309750.0; _ga_63QCVBV1V3=GS1.1.1646305679.1.1.1646309750.0; ASP.NET_SessionId=wlfbf2dx4oatlio0vl1ftinq; _gid=GA1.2.449121330.1646650093; calc-cookie=; OptanonConsent=isGpcEnabled=0&datestamp=Mon Mar 07 2022 11:38:48 GMT+0000 (Hora padrão da Europa Ocidental)&version=6.30.0&isIABGlobal=false&consentId=6caccc97-6af1-4b55-9049-5694835d9f7a&interactionCount=2&landingPath=NotLandingPage&groups=C0001:1,C0002:1,C0003:1,C0004:1&hosts=H10:1,H20:1,H7:1,H8:1,H23:1,H11:1,H24:1,H13:1,H25:1&genVendors=&geolocation=ES;&AwaitingReconsent=false; _gali=slider-container; _gat_UA-186811106-6=1',
'Host': 'simuladores.bancomontepio.pt',
'Origin': 'https://simuladores.bancomontepio.pt',
'Referer': 'https://simuladores.bancomontepio.pt/',
'sec-ch-ua': '" Not A;Brand";v="99", "Chromium";v="90", "Google Chrome";v="90"',
'sec-ch-ua-mobile': '?0',
'Sec-Fetch-Dest': 'empty',
'Sec-Fetch-Mode': 'cors',
'Sec-Fetch-Site': 'same-origin',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36'}headers = {'Accept': 'application/json, text/plain, */*' ,
'Accept-Encoding': 'gzip, deflate, br',
'Accept-Language': 'pt-PT,pt;q=0.9,en-US;q=0.8,en;q=0.7',
'Connection': 'keep-alive',
'Content-Length': '957',
'content-type': 'text/plain',
'Cookie': '_gcl_au=1.1.911606195.1646064658; OptanonAlertBoxClosed=2022-02-28T16:45:11.586Z; _ga=GA1.2.1147601977.1646064657; _ga_8WVEJF7X11=GS1.1.1646305654.3.1.1646309750.0; _ga_63QCVBV1V3=GS1.1.1646305679.1.1.1646309750.0; ASP.NET_SessionId=wlfbf2dx4oatlio0vl1ftinq; _gid=GA1.2.449121330.1646650093; calc-cookie=; OptanonConsent=isGpcEnabled=0&datestamp=Mon Mar 07 2022 11:38:48 GMT+0000 (Hora padrão da Europa Ocidental)&version=6.30.0&isIABGlobal=false&consentId=6caccc97-6af1-4b55-9049-5694835d9f7a&interactionCount=2&landingPath=NotLandingPage&groups=C0001:1,C0002:1,C0003:1,C0004:1&hosts=H10:1,H20:1,H7:1,H8:1,H23:1,H11:1,H24:1,H13:1,H25:1&genVendors=&geolocation=ES;&AwaitingReconsent=false; _gali=slider-container; _gat_UA-186811106-6=1',
'Host': 'simuladores.bancomontepio.pt',
'Origin': 'https://simuladores.bancomontepio.pt',
'Referer': 'https://simuladores.bancomontepio.pt/',
'sec-ch-ua': '" Not A;Brand";v="99", "Chromium";v="90", "Google Chrome";v="90"',
'sec-ch-ua-mobile': '?0',
'Sec-Fetch-Dest': 'empty',
'Sec-Fetch-Mode': 'cors',
'Sec-Fetch-Site': 'same-origin',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36'}

payload = {'CCRDCalculateInput':{},'MultifunctionsCalculateInput':{},'Device':{'Browser':'chrome','BrowserVersion':'90.0.4430.212','Device':'Desktop','Os':'windows','OsVersion':'windows-10','UserAgent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36'},'IsCustomer':'true','Ammount':7500,'Term':60,'ConditionCode':'26B1129900X-01-I-129900-F','CreditDestinationCode':'129900','ProductCode':'26B1129900X','FinancedExpenses':'false','FrequencyTrancheCode':'null','GoalCode':'C006','GoalDescription':'PROJETOS PESSOAIS','FrequencyTypeCode':'M','FamilyCode':'CP','Proponents':[{'Position':1,
'Birthday':'1992-03-07T13:03:30.000Z','State':'true','EntityType':{'ID':1,'CompanyID':1,'Code':'P','Description':'Proponente','Value':'null','ValueString':'null','State':'true','Imported':'null'},'ExpenseCodes':['009']}],'Counterparts':0,'OptionalExpenses':[{'Code':'009','Factor':1}],'ResidualValue':0,'InterestOnly':0,'Deferment':0}

Now I'm getting a empty json()... Response 200 but I got this structure:

{'Status': 'Unknown',
 'Error': {'VisibleToHuman': False, 'Code': '0', 'Message': ''},
 'Result': None}

As far as I know, the status should be "OK" to get some info on the Result.

Cheers

CodePudding user response:

Looks like you need to expand the payload to include more (all) of the parameters (including the cookies, specifically the ASP.NET_SessionId).

import requests

import warnings
warnings.filterwarnings("ignore")

term=24
amount=5000

url = 'https://simuladores.bancomontepio.pt/ITSCredit.External/Calculator/ITSCredit.Calculator.UI.External/gateway/Calculator/api/Calculator/Calculate?hash=-1359629931'

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.122 Safari/537.36',
'Accept-Language': 'pt-PT,pt;q=0.9,en-US;q=0.8,en;q=0.7',
'Cookie':'ASP.NET_SessionId=fhkyn1vn5knlw3uhdnh50nii;'}


payload = {
    "CCRDCalculateInput":{},
    "MultifunctionsCalculateInput":{},
    "Device":{
        "Browser":"chrome",
        "BrowserVersion":"96.0.4664.110",
        "Device":"Desktop",
        "Os":"windows",
        "OsVersion":"windows-10",
        "UserAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"},
    "IsCustomer":'true',
    "Ammount":amount,
    "Term":term,
    "ConditionCode":"26B1129900X-01-I-129900-F",
    "CreditDestinationCode":"129900",
    "ProductCode":"26B1129900X",
    "FinancedExpenses":'false',
    "FrequencyTrancheCode":'null',
    "GoalCode":"C006",
    "GoalDescription":"PROJETOS PESSOAIS",
    "FrequencyTypeCode":"M",
    "FamilyCode":"CP",
    "Proponents":[{
        "Position":'1',
        "Birthday":"1992-03-10T13:03:24.000Z",
        "State":'true',
        "EntityType":{
            "ID":'1',
            "CompanyID":'1',
            "Code":"P",
            "Description":"Proponente",
            "Value":'null',
            "ValueString":'null',
            "State":'true',
            "Imported":'null'},
        "ExpenseCodes":[
            "009"]}],
    "Counterparts":'0',
    "OptionalExpenses":[{
        "Code":"009",
        "Factor":'1'}],
    "ResidualValue":'0',
    "InterestOnly":'0',
    "Deferment":'0'}

jsonData = requests.post(url, headers=headers, json=payload, verify=False).json()
results = jsonData['Result']

mtic = results['MTIC']
installment = results['PeriodInstallment'][0]['Installment']
taeg = results['TAEG']
tan = results['PeriodInstallment'][0]['TAN']


print(f'Installment: {installment}\nTAEG: {taeg}\nTAN: {tan}\nMTIC: {mtic}')

Output:

Installment: 224.5
TAEG: 14.8
TAN: 7.0
MTIC: 5708.2

CodePudding user response:

import requests
import sys, json
import warnings
warnings.filterwarnings("ignore")

term=24
amount=5000

url = 'https://simuladores.bancomontepio.pt/ITSCredit.External/Calculator/ITSCredit.Calculator.UI.External/calculator/CPV2'

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.122 Safari/537.36',
'Accept-Language': 'pt-PT,pt;q=0.9,en-US;q=0.8,en;q=0.7'}

payload = {'Amount': amount,'Term': term,'ProductCode':"26B1129900X"}

response = requests.post(url, headers=headers, data=payload, verify=False)

<Response [200]> I only changed the link

  • Related