Selenium mitmproxy-CodePudding

I am trying to use mitmproxy to capture traffic generated by a python script using Selenium. My script manages to pass my requests through the proxy but the traffic doesn't look the way I would expect (see below). My script is as follows:

#!/usr/bin/python3

from selenium import webdriver
from selenium.webdriver.common.proxy import *
from selenium.webdriver.firefox.options import Options


def main():

    options = Options()
    options.accept_untrusted_certs = True
    options.add_argument("--headless")
    myProxy = "0.0.0.0:8080"
    options.proxy = Proxy({
        'proxyType': ProxyType.MANUAL,
        'httpProxy': myProxy,
        'sslProxy': myProxy,
        'noProxy': ''
    })
    driver = webdriver.Firefox(options=options)

    driver.get('http://example.com')

    driver.quit()


if __name__=="__main__":
    main()

To observe the response generated by the above script, I am using mitmdump with the following script:

from mitmproxy import ctx

def response(flow):
    print(flow.response.data.content)

Which produces the following output:

127.0.0.1:35966: clientconnect
b'\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\xff}TMs\xdb \x10\xbd\xfbWl\xd5K2#$'i\x1a\x8f-i\xfa\x99i\x0fi\x0fi\x0f=\x12\xb1\xb2\x98\x08P\x01\xc9\xf6t\xf2\xdf\xbbB\x8e#7\x99\x9a\x91\x81]x\xbb\xef\xb1\x90\xbd\x12\xa6\xf4\xbb\x16\xa1\xf6\xaa)f\xd9c\x87\\x143\xa0_\xe6\xa5o\xb0\xf8\xbc\xe5\xaam\x10>\x19\xc5\xa5\xce\xd2\xd1:\x1b\x97(\xf4\x1c\xca\x9a[\x87>\x8f:\xb1E\x04i1q\xd6\xde\xb7\x0c\x7fw\xb2\xcf\xa3\x8fF{\xd4\x9e\ra#(\xc7Y\x1ey\xdc\xfat\x08\xbf:@\xbd\x84\xa4\xb9\xc2<\xea%nZc\xfdd\xffF\n\xe7\x02{Y"\x0b\x93\x18\xa4\x96^\xf2\x86\xb9\x927\x98\x9f=A9\xbf#2C\x06\xfb\xc0\xa5s\xd1\xe8\xbb3b\x07\x7f\xc20Lyy\xbf\xb6\xa6\xd3\x82\x95\xa61v\t\xaf\xab9\xb5\xf3\xd5a\x89\xe2v-\xf5\x12\xe6O\xa6\x96\x0b!\xf5\xfa\xc8VQ\xa6\xac\xe2J6\xbb%0\xde\x92\x9c\xcc\xed\x9cG\x15\xc3\xd8\xb3N\xc6\xf0\xa1\x91\xfa\xfe\x86\x97\xb7\xc1tM\x9bb\x88nqm\x10~~\x8dh\xfc\xbdE\r\xb7\\xbba\xf2\x05\x9b\x1e\xbd,9|\xc3\x0e\xc9r0\xc4\xf0\xde\x12w\xc2\xa6\xa5\xcc\xa1\x95\xd5S.a\xf0\x10\xfe\x85\xec't\x83pKx;\x9f\xb7\xdb\xe7\x0c/Q\x01\xef\xbcy\x81\xe89\xaa\xd5\x7fE\x13\xd4&\x19\xdc\x19 \xd02\xcb\x85\xec\x1c\xe9\x94\\x1e\x01\x98-s5\x17fC\xc8\xed\x16.\xe8\xbb\xa2o\x18\xdb\xf5\x1d\x99\xc7\xa1%\xf3\xf3\xd3\xd5\x84\x0c_\x0e\xea\xc5\xd4\xf7\xd2I\x8fbB\xed1\x93\x8b\xc5\x9b\xc5b\x92\xc9p\xfeL`i,\xf7\xd2\x10Km4NA\xdf)\x14\x92\xc3\x89\xe2[\xb6\xd7\xe7j\xd0\xe7t\x02~\xac\xe2QU\xfc\xa3\xd8D\xe5c\xc7\xc3$d\x96\x86\n-\xc2Ye\xe9x\x1dg\xd9P\x9bt;)\xd8\xbe\x8e\xeb\xb3g7\x93L\xa3\xaf-~\xd4\xd2\x81\x08v\xa0Qe,t\x0ea\x985M\xe7\xfc@\xb8G\xc0\x11\xc1\r\x0ez\x0e:E\xf7\xc9%\xf0\xcbtDb\x17\xb6xB\x1a\xabe\x8f\xa6\xa1!y\t\xa0\xb3Ht|m:\x0f\xad\x95\x14\xa24t\xb4R\x071\x81\xe6\xdc\xddS\x85\x84\xe8-Z%\x9d#G\x92\xa5\xed!\xcf\x8c\x1e\x08\x8bU\x1e\r\xcf\x84[\xa6\xe9f\xb3I$\xd7<1v\x9d\x8e!]\xbaO3*n\x8c\x1dH\x10\xa0\nA\x92\x84\xd0x\x11\x10\xb34\x88\x93\xa5{\xa9\xd2\xf1A\xfb\x0b(\xeb|o\xe8\x04\x00\x00'
127.0.0.1:35956: GET http://example.com/
<< 200 OK 648b

The flow.response.data.content appears to be encrypted. Confusingly, if I simply run mitmproxy (not mitmdump) and open the corresponding flow I can observe the unencrypted response in the interactive UI. Also, when I make this request via the curl command:

curl --proxy http://0.0.0.0:8080 https://example.com

I get the unencrypted traffic to display through both mitmproxy and mitmdump as I had expected. It is only through selenium mitmdump that I encounter this issue.

I have added the mitmproxy certificate to my firefox profile and to my system to no avail.

CodePudding user response：

The issue is not one of encryption but of encoding. Looking at the flow.response.data.headers it is revealed that the response has gzip encoding. Using the following script with mitmdump:

from mitmproxy import ctx
import zlib

def response(flow):

    try:
        content = zlib.decompress(flow.response.data.content, 16 zlib.MAX_WBITS)
    except:
        content = flow.response.data.content

    print(content)

Correctly decode the data to print the expected result. In a more robust script we can decide how to decode the data by reading the encoding from the header.

CodePudding user response：

The problem is that you print flow.response.data.content (.data is not a public API, see the docs). If you'd use flow.response.content instead, mitmproxy should decompress the contents automatically:

def response(flow):
    print(flow.response.content)