Mitmproxy as a windows service fails to SSL

[illera88]

Problem Description

Mitmproxy works fine if run directly but fails to intercept SSL requests when run as a windows service. I run the code as Administrator.

Steps to reproduce the behavior:

This code works fine:

class AddHeader:
    def __init__(self):
        self.num = 0

    def response(self, flow):
        self.num = self.num + 1
        flow.response.headers["count"] = str(self.num)

if __name__ == '__main__':
    options = Options(listen_host='127.0.0.1', listen_port=8081)
    config = ProxyConfig(options)
    master = ProxyMaster(options, with_termlog=DEBUG, with_dumper=DEBUG)
    master.server = ProxyServer(config)
    master.addons.add(AddHeader())
    master.run()

But this code that uses mitmproxy within a service does not work and produces an error:

import socket

import win32serviceutil

import servicemanager
import win32event
import win32service

from mitmproxy.options import Options
from mitmproxy.proxy.config import ProxyConfig
from mitmproxy.proxy.server import ProxyServer
from mitmproxy.tools.dump import DumpMaster

class SMWinservice(win32serviceutil.ServiceFramework):
    '''Base class to create winservice in Python'''

    _svc_name_ = 'pythonService'
    _svc_display_name_ = 'Python Service'
    _svc_description_ = 'Python Service Description'

    @classmethod
    def parse_command_line(cls):
        '''
        ClassMethod to parse the command line
        '''
        win32serviceutil.HandleCommandLine(cls)

    def __init__(self, args):
        '''
        Constructor of the winservice
        '''
        win32serviceutil.ServiceFramework.__init__(self, args)
        self.hWaitStop = win32event.CreateEvent(None, 0, 0, None)
        socket.setdefaulttimeout(60)

    def SvcStop(self):
        '''
        Called when the service is asked to stop
        '''
        self.stop()
        self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
        win32event.SetEvent(self.hWaitStop)

    def SvcDoRun(self):
        '''
        Called when the service is asked to start
        '''
        #self.ReportServiceStatus(win32service.SERVICE_RUNNING)
        self.start()
        servicemanager.LogMsg(servicemanager.EVENTLOG_INFORMATION_TYPE,
                              servicemanager.PYS_SERVICE_STARTED,
                              (self._svc_name_, ''))
        self.main()

    def start(self):
        '''
        Override to add logic before the start
        eg. running condition
        '''
        pass

    def stop(self):
        '''
        Override to add logic before the stop
        eg. invalidating running condition
        '''
        pass

    def main(self):
        '''
        Main class to be overridden to add logic
        '''
        pass

class AddHeader:
    def __init__(self):
        self.num = 0

    def response(self, flow):
        self.num = self.num + 1
        flow.response.headers["count"] = str(self.num)

class PythonCornerExample(SMWinservice):
    _svc_name_ = "service"
    _svc_display_name_ = "just a service"
    _svc_description_ = "doesn't do much"

    def start(self):
        self._install()
        self.isrunning = True

    def stop(self):
        self.isrunning = False
        self.master.shutdown()

    def main(self):
        options = Options(listen_host='127.0.0.1', listen_port=8081, confdir=r"C:\Users\username\.mitmproxy")
        config = ProxyConfig(options)
        self.master = ProxyMaster(options, with_termlog=True, with_dumper=True)
        self.master.server = ProxyServer(config)
        self.master.addons.add(AddHeader())

        try:
            self.master.run()
        except Exception as e:
            servicemanager.LogErrorMsg("Some error happened running MitmProxy. Is port 8080 free?\n{}".format(e))

if __name__ == '__main__':
    PythonCornerExample.parse_command_line()

The error happens when an SSL packet gets to mitmproxy:
<< Cannot establish TLS with example.com:443 (sni: example.com): TlsException('SSL handshake error: WantReadError()')

And can be run with:

python script.py debug

I set confdir=r"C:\Users\username\.mitmproxy" just in case since it runs as Administrator it is not looking for the certs in the folder where I have them but still fails.
What am I missing?
Thank you in advance

System Information

Mitmproxy: 5.3.0
Python:    3.7.1
OpenSSL:   OpenSSL 1.1.1h  22 Sep 2020
Platform:  Windows-10-10.0.20241-SP0

[mhils]

May I ask why you want to run mitmproxy as a Windows service?

[illera88]

I need mitmproxy to be run on Windows startup for other programs to use it and the proper way to do that in a Windows environment is by using a service.

Does anything come to your mind on where the problem may be?

Thanks @mhils

[mhils]

My guess would have been on ~/.mitmproxy resolving differently, no other idea. Are you getting the same cert?

[illera88]

I don't think that's the problem. Check this out:
pyca/pyopenssl#168 (comment)

import socket
import OpenSSL
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.settimeout(100) # If I comment this it does not raise the OpenSSL.SSL.WantReadError exception
ctx = OpenSSL.SSL.Context(OpenSSL.SSL.TLSv1_METHOD)
conn = OpenSSL.SSL.Connection(ctx, s)
conn.connect(("93.184.216.34", 443))
try:
    conn.do_handshake()
except OpenSSL.SSL.WantReadError:
    print("badness")

Do you know if anywhere in mitmproxy you are setting a timeout? A workaround could be not setting any

[mhils]

I think we generally don't, also there would be no reason why this ony happens as a service? Do you get this with any TLS client, or just particular ones? What happens if you just run curl -k -x localhost:8081 https://example.com/?

[illera88]

I'm using chrome to test this because of #4335 (comment).

I made my previous code worke with:

import select
import socket
import OpenSSL
import ssl
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.settimeout(100)
ctx = OpenSSL.SSL.Context(OpenSSL.SSL.TLSv1_METHOD)
conn = OpenSSL.SSL.Connection(ctx, s)
conn.connect(("93.184.216.34", 443))

while True:
    try:
        conn.do_handshake()
    except OpenSSL.SSL.WantReadError:
        rd, _, _ = select.select([s], [], [], s.gettimeout())
        if not rd:
            raise ssl.SSLError("select timed out of {} seconds".format(s.gettimeout()))
        continue
    except OpenSSL.SSL.Error as e:
       raise ssl.SSLError('bad handshake: %r' % e)
    break

You have a couple of places in your code where you use do_handshake(). I'm testing if I can include my workaround in your code because it actually seems to be related with other mitmproxy crashes I've been experiencing:

127.0.0.1:12342: Traceback (most recent call last):
  File "C:\Python37-x64\lib\site-packages\mitmproxy\net\tcp.py", line 118, in read
    data = self.o.read(rlen)
  File "C:\Python37-x64\lib\socket.py", line 589, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Python37-x64\lib\site-packages\mitmproxy\proxy\server.py", line 121, in handle
    root_layer()
  File "C:\Python37-x64\lib\site-packages\mitmproxy\proxy\modes\http_proxy.py", line 9, in __call__
    layer()
  File "C:\Python37-x64\lib\site-packages\mitmproxy\proxy\protocol\tls.py", line 285, in __call__
    layer()
  File "C:\Python37-x64\lib\site-packages\mitmproxy\proxy\protocol\http1.py", line 100, in __call__
    layer()
  File "C:\Python37-x64\lib\site-packages\mitmproxy\proxy\protocol\http.py", line 203, in __call__
    if not self._process_flow(flow):
  File "C:\Python37-x64\lib\site-packages\mitmproxy\proxy\protocol\http.py", line 257, in _process_flow
    request: http.HTTPRequest = self.read_request_headers(f)
  File "C:\Python37-x64\lib\site-packages\mitmproxy\proxy\protocol\http1.py", line 13, in read_request_headers
    return http1.read_request_head(self.client_conn.rfile)
  File "C:\Python37-x64\lib\site-packages\mitmproxy\net\http\http1\read.py", line 52, in read_request_head
    host, port, method, scheme, authority, path, http_version = _read_request_line(rfile)
  File "C:\Python37-x64\lib\site-packages\mitmproxy\net\http\http1\read.py", line 243, in _read_request_line
    line = _get_first_line(rfile)
  File "C:\Python37-x64\lib\site-packages\mitmproxy\net\http\http1\read.py", line 230, in _get_first_line
    line = rfile.readline()
  File "C:\Python37-x64\lib\site-packages\mitmproxy\net\tcp.py", line 160, in readline
    ch = self.read(1)
  File "C:\Python37-x64\lib\site-packages\mitmproxy\net\tcp.py", line 136, in read
    raise exceptions.TcpTimeout()
mitmproxy.exceptions.TcpTimeout: None
Traceback (most recent call last):
  File "C:\Python37-x64\lib\site-packages\mitmproxy\net\tcp.py", line 118, in read
    data = self.o.read(rlen)
  File "C:\Python37-x64\lib\socket.py", line 589, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Python37-x64\lib\site-packages\mitmproxy\proxy\server.py", line 121, in handle
    root_layer()
  File "C:\Python37-x64\lib\site-packages\mitmproxy\proxy\modes\http_proxy.py", line 9, in __call__
    layer()
  File "C:\Python37-x64\lib\site-packages\mitmproxy\proxy\protocol\tls.py", line 285, in __call__
    layer()
  File "C:\Python37-x64\lib\site-packages\mitmproxy\proxy\protocol\http1.py", line 100, in __call__
    layer()
  File "C:\Python37-x64\lib\site-packages\mitmproxy\proxy\protocol\http.py", line 203, in __call__
    if not self._process_flow(flow):
  File "C:\Python37-x64\lib\site-packages\mitmproxy\proxy\protocol\http.py", line 257, in _process_flow
    request: http.HTTPRequest = self.read_request_headers(f)
  File "C:\Python37-x64\lib\site-packages\mitmproxy\proxy\protocol\http1.py", line 13, in read_request_headers
    return http1.read_request_head(self.client_conn.rfile)
  File "C:\Python37-x64\lib\site-packages\mitmproxy\net\http\http1\read.py", line 52, in read_request_head
    host, port, method, scheme, authority, path, http_version = _read_request_line(rfile)
  File "C:\Python37-x64\lib\site-packages\mitmproxy\net\http\http1\read.py", line 243, in _read_request_line
    line = _get_first_line(rfile)
  File "C:\Python37-x64\lib\site-packages\mitmproxy\net\http\http1\read.py", line 230, in _get_first_line
    line = rfile.readline()
  File "C:\Python37-x64\lib\site-packages\mitmproxy\net\tcp.py", line 160, in readline
    ch = self.read(1)
  File "C:\Python37-x64\lib\site-packages\mitmproxy\net\tcp.py", line 136, in read
    raise exceptions.TcpTimeout()
mitmproxy.exceptions.TcpTimeout: None

mitmproxy has crashed!
Please lodge a bug report at: https://github.com/mitmproxy/mitmproxy

[mhils]

FYI, everything in mitmproxy.proxy is about to be swapped out with a new sans-io implementation in a few days/weeks.
You may want to take a look at the sans-io branch, which could also give you a better error message here. See #1775 and https://github.com/mitmproxy/mitmproxy/projects/4. I think we're unlikely to merge nontrivial changes to mitmproxy.proxy until then.

[illera88]

It's fixed in 7.0