How to fix selenium error "ERROR: The requested URL could not be retrieved"?-CodePudding

In a docker in a gitlab CI I am running a very short test code with python selenium

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

chrome_options = Options()
        
chrome_options.add_argument("--headless")
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
chrome_options.add_argument('--start-maximized')
driver = webdriver.Chrome(options=chrome_options)

driver.quit()

But it fails with the error

selenium.common.exceptions.WebDriverException: Message: <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html><head>
<meta type="copyright" content="Copyright (C) 1996-2016 The Squid Software Foundation and contributors">
<meta http-equiv="Content-Type" CONTENT="text/html; charset=utf-8">
<title>ERROR: The requested URL could not be retrieved</title>
<style type="text/css"><!-- 
 /*
 * Copyright (C) 1996-2016 The Squid Software Foundation and contributors
....

Used packages:

selenium 4.0.0
chrome browser 92.0.4515.107
chromedriver 92.0.4515.43

Anything I can try to fix this problem?

HINT

The error appears when trying to get the selenium driver!!

CodePudding user response：

This mean you can't access to the targeted URL ("https://www.google.com"). Check if you don't have any proxy (Running Selenium Webdriver with a proxy in Python) needed on your network or a VPN running.

CodePudding user response：

The problem is that selenium 4.0.0 seems to have a bug with respect to the proxy. Do not use selenium 4.0.0, use 3.141.0.

Possible related ticket: https://github.com/SeleniumHQ/selenium/issues/9925