Thread problem with har dump example

help
#1

I’m starting from the har dump complex example to create my script but when I try to test the system with some load (20 users) after 10 minutes mitmproxy fails with a lot of “Can’t start new thread” python error.
I studied the example code and after various test I found that the delete of the SERVERS_SEEN variable resolves thread errors and the system can go on until 1 hour. Without this variable in the har dump I can’t put the ssl and connect time.
How can I get this values or use a global variable correctly in this heavy thread scenario?

Thanks.

#2

Hi @Gigiarum,

Sorry for the late reply (and thanks for helping out in other threads). I guess the problem we are having here is that we don’t properly clean up server connection sockets if we still keep a reference to the socket object around (as part of SERVERS_SEEN). One fix would be to just track server_conn.id in there instead of the full object.
That being said, har dump with plenty of load will run into scaling issues at some point anyway.

Hope that helps!

Max

#3

Hi @mhils,
thanks for your reply.

I try to save connection id but in my version (2.0.2) the property in the object is not present. I read source code and in the class ServerConnection I found

_stateobject_attributes = dict(
        address=tcp.Address,
        ip_address=tcp.Address,
        source_address=tcp.Address,
        ssl_established=bool,
        cert=certs.SSLCert,
        sni=str,
        alpn_proto_negotiated=bytes,
        timestamp_start=float,
        timestamp_tcp_setup=float,
        timestamp_ssl_setup=float,
        timestamp_end=float,
)

when I switch to master branch instead

_stateobject_attributes = dict(
        id=str,
        address=tuple,
        ip_address=tuple,
        source_address=tuple,
        ssl_established=bool,
        cert=certs.SSLCert,
        sni=str,
        alpn_proto_negotiated=bytes,
        tls_version=str,
        timestamp_start=float,
        timestamp_tcp_setup=float,
        timestamp_ssl_setup=float,
        timestamp_end=float,
)

How can I get a version of mitmproxy with the id field or can I compile directly from sources?

#4

It’s python, so you should be able to just into the source location of where you install mitmproxy and make the change as needed.