Python requests form-data method-CodePudding

I am trying to automate retrieval of data from this site.

My code goes:

import requests
a_session = requests.Session()
a_session.get('https://www.kmcgov.in/KMCPortal/jsp/KMCBirthRecordSearch.jsp')
session_cookies = a_session.cookies
cookie = session_cookies.get_dict()
ablu = {
    'FatherName': 'Cha',
    'dateofBirth': '19/08/2004',
    'MotherName': 'Cha',
}
referer = 'https://www.kmcgov.in/KMCPortal/jsp/KMCBirthRecordSearch.jsp'
url = 'https://www.kmcgov.in/KMCPortal/KMCBirthRegistrationAction.do?var=getVal'
r = requests.post(url, cookies=cookie, headers={"referer": referer}, data=ablu)
print(r.text)

The command line outputs {failure:true,ERRORKEY:{"ERRORMESSAGE":"Please Contact your system administrator"}}

I wonder what I am doing wrong. This is the first time I am trying to automate sending form-data.

Update 1:

import requests
a_session = requests.Session()
a_session.get('https://www.kmcgov.in/KMCPortal/jsp/KMCBirthRecordSearch.jsp')
session_cookies = a_session.cookies
cookie = session_cookies.get_dict()
ablu = {
    'FatherName': 'Cha',
    'dateofBirth': '19/08/2004',
    'MotherName': 'Cha',
}
referer = 'https://www.kmcgov.in/KMCPortal/jsp/KMCBirthRecordSearch.jsp'
url = 'https://www.kmcgov.in/KMCPortal/KMCBirthRegistrationAction.do?var=getVal'
r = a_session.post(url, cookies=cookie, headers={"Referer": referer, "Accept": '*/*',
    "Accept-Encoding": 'gzip, deflate, br',
    "Accept-Language": 'en-US,en;q=0.9,bn;q=0.8',
    "Connection": 'keep-alive',
    "Content-Length": '56',
    "Content-Type": 'application/x-www-form-urlencoded; charset=UTF-8',
    "DNT": '1',
    "Host": 'www.kmcgov.in',
    "Origin": 'https://www.kmcgov.in',
    "sec-ch-ua": '"Google Chrome";v="95", "Chromium";v="95", ";Not A Brand";v="99"',
    "sec-ch-ua-mobile": '?0',
    "sec-ch-ua-platform": '"Windows"',
    "Sec-Fetch-Dest": 'empty',
    "Sec-Fetch-Mode": 'cors',
    "Sec-Fetch-Site": 'same-origin',
    "User-Agent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36',
    "X-Requested-With": 'XMLHttpRequest'}, data=ablu)
print(r.text)

CodePudding user response：

Check this code, it works for me, maybe you're unnecessarily providing cookies to requests.post(...):

import requests

url = 'https://www.kmcgov.in/KMCPortal/KMCBirthRegistrationAction.do?var=getVal'

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:93.0) Gecko/20100101 Firefox/93.0",
    "Accept": "*/*",
    "Accept-Language": "pl,en-US;q=0.7,en;q=0.3",
    "X-Requested-With": "XMLHttpRequest",
    "Content-Type": "application/x-www-form-urlencoded; charset=UTF-8",
    "Sec-Fetch-Dest": "empty",
    "Sec-Fetch-Mode": "cors",
    "Sec-Fetch-Site": "same-origin"
}

payload = {
    'FatherName': 'CHA',
    'dateOfBirth': '19/08/2004',
    'MotherName': 'CHA'
}

response = requests.post(url, headers=headers, data=payload)

print(response.text)

EDIT:

Ok, it sometimes works and sometimes doesn't.

CodePudding user response：

Here is a working example.

First of all, note that we are using a session, so the cookies are automatically collected for you. There is no need to handle them manually.

Also, we can send additional headers when making a request on a case by case basis, but the session headers are included too and merged with your request. Please see here: Python requests - Advanced Usage.

In this case it's not even necessary to visit the page by issuing a get request first. You can send your POST request right away, it will work and it will return JSON-like data.

I think the crucial bit here was the parameter dateOfBirth. Capitalization is not the same in your original code. That must have been the reason why your requests failed on the server-side.

Also note that some headers like:

Content-Type: application/x-www-form-urlencoded
Content-Length: 56

will be populated automatically by the requests module as necessary.

import requests

a_session = requests.Session()
a_session.headers = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:93.0) Gecko/20100101 Firefox/93.0",
    "Accept": "*/*",
    "Accept-Language": "en-US,en;q=0.5",
    "Accept-Encoding": "gzip, deflate",
    "DNT": "1",
    "Connection": "keep-alive",
    "Upgrade-Insecure-Requests": "1"
}
# r = a_session.get('https://www.kmcgov.in/KMCPortal/jsp/KMCBirthRecordSearch.jsp')
# print(r.status_code)

ablu = {
    'FatherName': 'CHA',
    'dateOfBirth': '19/08/2004',
    'MotherName': 'CHA'
}

# additional headers
headers = {
    "X-Requested-With": "XMLHttpRequest",
    "Origin": "https://www.kmcgov.in",
    "Referer": 'https://www.kmcgov.in/KMCPortal/jsp/KMCBirthRecordSearch.jsp',
    "Sec-Fetch-Dest": "empty",
    "Sec-Fetch-Mode": "cors",
    "Sec-Fetch-Site": "same-origin"
}
url = 'https://www.kmcgov.in/KMCPortal/KMCBirthRegistrationAction.do?var=getVal'
r = a_session.post(url, headers=headers, data=ablu)
print(r.text)