Connection to google.com has a security policy called HTTP Strict Transport Security (HSTS), which m-CodePudding

While running my test scripts with selenium == 4.2.0 like this:

from selenium.webdriver import Firefox, FirefoxOptions
from selenium.webdriver.firefox.service import Service
from selenium.webdriver.common.proxy import Proxy, ProxyType


options = FirefoxOptions()
service = Service()
options.headless = True
options.accept_insecure_certs = True 
proxy = Proxy({
    'httpProxy': proxy_addr,
    'sslProxy': proxy_addr,
    'proxyType': ProxyType.MANUAL
})

options.proxy = proxy

wd = Firefox(service=service, options=options)
wd.execute("get", {'url': 'http://google.com'})

I'm getting the following error:

An error occurred during a connection to www.google.com has a security policy called HTTP Strict Transport Security (HSTS), which means that Firefox can only connect to it securely. You can’t add an exception to visit this site. Please contact the website owners to inform them of this problem. This website might not support the TLS 1.2 protocol, which is the minimum version supported by Firefox. Enabling TLS 1.0 and TLS 1.1 might allow this connection to succeed.

I think the problem is that I'm using a proxy which is running on localhost. Since I use the browser in headless mode and configured accept_insecure_certs = True I don't see how to find a workaround for this error. I'll be grateful if someone advises what else I can try to do.

CodePudding user response：

HTTP Strict-Transport-Security(HSTS)

The HTTP Strict-Transport-Security response header (commonly known as HSTS) informs browsers that the site should only be accessed using HTTPS and any future attempts to access it using HTTP should automatically be converted to HTTPS. This approach is more secure than simply configuring a HTTP to HTTPS (301) redirect on your server, where the initial HTTP connection is still vulnerable to man-in-the-middle attack.

Incase a website accepts a connection through HTTP and redirects to HTTPS, visitors may initially communicate with the non-encrypted version of the site before being redirected, as an example, if a visitor types http://www.example.com/ or even just example.com. This creates an opportunity for a man-in-the-middle attack. The redirect could be exploited to direct visitors to a malicious site instead of the secure version of the original site.

The HTTP Strict Transport Security header informs the browser that it should never load a site using HTTP and should automatically convert all attempts to access the site using HTTP to HTTPS requests instead.

However, it is to be noted that the Strict-Transport-Security header is ignored by the browser when your site is accessed using HTTP; this is because an attacker may intercept HTTP connections and inject the header or remove it. When your site is accessed over HTTPS with no certificate errors, the browser knows your site is HTTPS capable and will honor the Strict-Transport-Security header.

Solution

In short you need to replace http with https within the url_string. Effectively, the url_string needs to be:

https://google.com

CodePudding user response：

I'm assuming that you're using a MITM that allows you to intercept the TLS traffic. If so, then this is exactly the scenario that HSTS preload is intended to prevent ;)

Your MITM will be generating a fake certificate on the fly, but because it does not match the HSTS preload list that is baked into the browser, then this is why you get presented with an error (rather than a dialog that asks if you want to continue).

You may be able to get around this by configuring the proxy to strip the HSTS header on all responses (check the documentation for the particular MITM that you are using).