How to randomize Header with Users Agent list?-CodePudding

i have a question,

in my script .py i have:

    headers = {'Connection': 'keep-alive',
              'Cache-Control': 'max-age=0',
              'Upgrade-Insecure-Requests': '1',
              'User-Agent': {agents},
              'Accept': 'text/html,application/xhtml xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
              'Accept-Encoding': 'gzip, deflate',
              'Accept-Language': 'en-US,en;q=0.9,fr;q=0.8',
              'referer': 'bing.com'}

On the .json file

    { 
      "user_agent_list": [
          "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36",
          "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36"
      ]
    }

and when i send the request:

find_file = (grequests.get(url_a, headers={headers}, timeout=None, stream=True, verify=False, allow_redirects=False) for url_a in [f'{site_link}{x}' for x in file_rootl])

how i can, randomize headers={headers} for each requests ? of course " for agents in user_agent_list "

more better:

'User-Agent': {agents}, this i need randomize inside headers...

Some idea ? Thanks Max

CodePudding user response：

You can have a random one by having it passed by a random function:

import random

user_agent_list = [
          "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36",
          "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36"
      ]

def get_headers():
    headers = {'Connection': 'keep-alive',
              'Cache-Control': 'max-age=0',
              'Upgrade-Insecure-Requests': '1',
              'User-Agent': random.choice(user_agent_list),
              'Accept': 'text/html,application/xhtml xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
              'Accept-Encoding': 'gzip, deflate',
              'Accept-Language': 'en-US,en;q=0.9,fr;q=0.8',
              'referer': 'bing.com'}
    return headers

#then
find_file = (grequests.get(url_a, headers=get_headers(), timeout=None, stream=True, verify=False, allow_redirects=False) for url_a in [f'{site_link}{x}' for x in file_rootl])

where user_agent_list has the values imported from the json.