web scrapping / web crawling showing 403 error on the site i want to crawl-CodePudding

import requests
from bs4 import BeautifulSoup
url ='https://www.vesselfinder.com/vessels'
headers= {'User-Agent': 'Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)'}
response = requests.get(url)
soup = BeautifulSoup(response.content, 'html.parser')
response.status_code

i tried different user agent but still not working, i tried other sites it work but this website not working, help me to crawl all vessel data from this site. thanks in advance!!!

CodePudding user response：

Server wants an additional header for language

import requests

headers = {
    'user-agent': 'Mozilla/5.0',
    'accept-language': 'en-GB,en-US;q=0.9,en;q=0.8',
}

response = requests.get('https://www.vesselfinder.com/vessels', headers=headers)
response.status_code