I tested a script with Selenium Python to retrieve information from a web page. It works, at least until my IP is found. I would like to try using a proxy. I have tried the following two alternatives.

First way:

PROXY = "176.31.68.252:20213"

webdriver.DesiredCapabilities.CHROME['proxy'] = {
    "httpProxy": PROXY,
    "ftpProxy": PROXY,
    "sslProxy": PROXY,
    "noProxy": None,
    "proxyType": "MANUAL",
    "class": "org.openqa.selenium.Proxy",
    "autodetect": False
}

url = 'https://www.ufficiocamerale.it/'

driver = webdriver.Chrome(desired_capabilities=webdriver.DesiredCapabilities.CHROME)
wait = WebDriverWait(driver, 20)
driver.get(url)

# rest of the code

Second way:

PROXY = "176.31.68.252:20213"

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--proxy-server=https://%s' % PROXY)

url = 'https://www.ufficiocamerale.it/'

driver = webdriver.Chrome(chrome_options=chrome_options)
wait = WebDriverWait(driver, 20)
driver.get(url)

# rest of the code

But in the first case the webpage returns the error ERR_EMPTY_RESPONSE and in the second case it returns the error ERR_TIMED_OUT.

I found the proxy in an online free proxy list.

Is this a problem with my chosen proxy? Or is it a code problem? What is the standard way to proceed in these cases?

CodePudding user response：

Google Restriction

You are getting an error message because of the proxy,

Google blocks all free proxy's

Proxy Spider

I also coded a proxy spider that tests all proxy's on: https://free-proxy-list.net/

Code: https://github.com/xtekky/proxy-spider

Setup Own Proxy

Here is an article that tells you how to create your own proxy server.

Setup Selenium with proxy

• Try this script:

Github Code: https://github.com/xtekky/selenium-tutorials/tree/main/selenium proxy

from selenium import webdriver
from selenium.webdriver.common.proxy import ProxyType, Proxy #proxy module
import time

proxy_ip = 'ip:port' #get a free proxy from the websites in the description

#setting up proxy
proxy =Proxy()
proxy.proxy_type = ProxyType.MANUAL
proxy.http_proxy = proxy_ip
proxy.ssl_proxy = proxy_ip

#linking proxy and setting up driver
capabilities = webdriver.DesiredCapabilities.CHROME
proxy.add_to_capabilities(capabilities)
driver = webdriver.Chrome('CHROMEDRIVER PATH', desired_capabilities=capabilities) # replace the chromedriver path

#loading test page
driver.get('https://httpbin.org/ip')
time.sleep(8)
driver.quit()