For some reason I get an Errno 110 when my flow list reaches 1019 items (which may be a limitation of my ubuntu vm or my computer). I know I can manually press ‘z’ to clear the flow list, I’d like to do it via script periodically. I see there’s a ‘clear_events’ in mitmproxy.tools.console. Is there any special magic to calling that function?
Uh… this should not happen. Do you happen to have a traceback? 
I’ve ran into the same error on both a VirtualBox vm and when booted directly into Ubuntu. The traceback seems to vary slightly each time, but the same general thing happens. On or about item 1019, I get an Errno 110 “failed to connect” message in red in the flowlist, and the traceback says something about too many open files. Behavior of the script is simply to infinitely replay the one flow.
Edit: I should also mention that it does not crash back to terminal, and when I quit mitmproxy, there’s no traceback. The screenshot is the best I can do.
Finally got something useful:
Exception in thread RequestReplayThread:
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/mitmproxy/net/tcp.py", line 750, in connect
connection = self.makesocket()
File "/usr/local/lib/python3.5/dist-packages/mitmproxy/net/tcp.py", line 746, in makesocket
return socket.socket(self.address.family, socket.SOCK_STREAM)
File "/usr/lib/python3.5/socket.py", line 134, in __init__
_socket.socket.__init__(self, family, type, proto, fileno)
OSError: [Errno 24] Too many open files
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/mitmproxy/proxy/protocol/http_replay.py", line 75, in run
server.connect()
File "/usr/local/lib/python3.5/dist-packages/mitmproxy/connections.py", line 230, in connect
tcp.TCPClient.connect(self)
File "/usr/local/lib/python3.5/dist-packages/mitmproxy/net/tcp.py", line 768, in connect
(self.address.host, err)
mitmproxy.exceptions.TcpException: Error connecting to "flex-capacity-na.amazon.com": [Errno 24] Too many open files
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/local/lib/python3.5/dist-packages/mitmproxy/proxy/protocol/http_replay.py", line 100, in run
self.channel.ask("error", self.f)
File "/usr/local/lib/python3.5/dist-packages/mitmproxy/controller.py", line 35, in ask
raise exceptions.Kill()
mitmproxy.exceptions.Kill
Exception in thread RequestReplayThread:
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/mitmproxy/net/tcp.py", line 763, in connect
connection.connect(self.address())
TimeoutError: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/mitmproxy/proxy/protocol/http_replay.py", line 75, in run
server.connect()
File "/usr/local/lib/python3.5/dist-packages/mitmproxy/connections.py", line 230, in connect
tcp.TCPClient.connect(self)
File "/usr/local/lib/python3.5/dist-packages/mitmproxy/net/tcp.py", line 768, in connect
(self.address.host, err)
mitmproxy.exceptions.TcpException: Error connecting to "flex-capacity-na.amazon.com": [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/local/lib/python3.5/dist-packages/mitmproxy/proxy/protocol/http_replay.py", line 100, in run
self.channel.ask("error", self.f)
File "/usr/local/lib/python3.5/dist-packages/mitmproxy/controller.py", line 35, in ask
raise exceptions.Kill()
mitmproxy.exceptions.Kill
Exception in thread RequestReplayThread:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/local/lib/python3.5/dist-packages/mitmproxy/proxy/protocol/http_replay.py", line 116, in run
if server.connected():
AttributeError: 'NoneType' object has no attribute 'connected'
Exception in thread RequestReplayThread:
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/mitmproxy/net/tcp.py", line 750, in connect
connection = self.makesocket()
File "/usr/local/lib/python3.5/dist-packages/mitmproxy/net/tcp.py", line 746, in makesocket
return socket.socket(self.address.family, socket.SOCK_STREAM)
File "/usr/lib/python3.5/socket.py", line 134, in __init__
_socket.socket.__init__(self, family, type, proto, fileno)
OSError: [Errno 24] Too many open files
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/mitmproxy/proxy/protocol/http_replay.py", line 75, in run
server.connect()
File "/usr/local/lib/python3.5/dist-packages/mitmproxy/connections.py", line 230, in connect
tcp.TCPClient.connect(self)
File "/usr/local/lib/python3.5/dist-packages/mitmproxy/net/tcp.py", line 768, in connect
(self.address.host, err)
mitmproxy.exceptions.TcpException: Error connecting to "flex-capacity-na.amazon.com": [Errno 24] Too many open files
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/local/lib/python3.5/dist-packages/mitmproxy/proxy/protocol/http_replay.py", line 100, in run
self.channel.ask("error", self.f)
File "/usr/local/lib/python3.5/dist-packages/mitmproxy/controller.py", line 35, in ask
raise exceptions.Kill()
mitmproxy.exceptions.Kill
Exception in thread RequestReplayThread:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/local/lib/python3.5/dist-packages/mitmproxy/proxy/protocol/http_replay.py", line 116, in run
if server.connected():
AttributeError: 'NoneType' object has no attribute 'connected'
This is what it did in mitmproxy:
Thanks. Can you send SIGUSR1 and SIGUSR2 to mitmproxy after it has been running for a while and post the output here?
I assume you mean pkill SIGUSR1 mitmproxy or something similar, yes? If so, I’ve tried that, and all mitmproxy does is flash a bunch of information on the screen for about .1 seconds and then close.
@Mk1Escort This probably happens because your script is making a lot of requests in such short time that the threads can’t complete their work and linux probably treats working threads as open files. You can check the limit of opened files at time by ulimit -n, it may be 1024 which is why you see the error at around 1020.
you can increase the limit by ulimit -n 2048
I still don’t know why this behavior occurs, but I did find the command I was looking for: ctx.master.view.clear(). Now I just need to figure out how to trigger it when the flow count reaches a certain number.
Edit: Found my answer: ctx.master.view.focus.index. Unfortunately I may never know the exact reason why mitmproxy locks up on me. It definitely only happnes when large numbers of flows are created in a short time (about 1k in 5 minutes). I’ve received over 3k requests in half that time with no issue. So, while ujjwal96’s solution didn’t work out, he’s probably right in that it’s most likely just my setup.