How to send a request which includes search box content while scraping a website with search box?-CodePudding

I am trying to scrape a website using beautiful soup. For that, while making a request I need to send the Instagram user ID which is supposed to be entered in a search box & scrape the response HTML. How should I send the user ID while sending a request? Thanks.

import requests
from bs4 import BeautifulSoup

URL = "https://product.influencer.in/price-index"
r = requests.get(URL)

soup = BeautifulSoup(r.content, 'lxml') 
print(soup.prettify())

CodePudding user response：

It's a simple post request to the api to get that data. You'll need to enter your email in the value below

import requests

instagramHandles = ['leomessi','cristiano']

URL = "https://team.influencer.in/api/v1/price-index/"
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'}

for instagramHandle in instagramHandles:
    payload = {
        'email': "[email protected]",
        'insta_handle': instagramHandle}
    
    jsonData = requests.post(URL, headers=headers, data=payload).json()
    cost_min = jsonData['data']['cost_min']
    cost_max = jsonData['data']['cost_max']
    
    print(f'{instagramHandle}: {cost_min} - {cost_max}')

Output:

leomessi: 5.4 Cr. - 6.5 Cr.
cristiano: 8.5 Cr. - 10.1 Cr.