Hi all—just thought I’d pipe up here to let you know I’m doing a project using mitmproxy for Australia’s national news service.
Would love your input/feedback if you have any: https://www.abc.net.au/news/2018-10-25/my-phone-is-spying-on-me-so-i-decided-to-spy-on-my-phone/10306586
I’m also keen to hear about any tips or techniques for analysing big dumps of request data, mostly from a privacy perspective.
Hi @drzax,
Very cool project! I love to see more of these “I want to see where my data goes”-style projects. Are you on Android or on iOS? Android is unfortunately recently implemented changes that make these kinds of analyses needlessly difficult [1], but iPhones are still behaving very reasonable in that regard.
Please make sure that you add yourself to https://mitmproxy.org/publications/! 
Also, if your copyeditors allow it, keep us as “mitmproxy” and not “MITM Proxy”.
Please keep us updated and feel free to drop by our developer Slack channel if you need any help!
Thanks @mhils. I’m doing it on iPhone for now, but have plans for an Android comparison, so I might need some pointers on that—is the dev slack a reasonable place to ask about that?
I’m sure I can get the name updated in the copy. Cheers.
I’m doing it on iPhone for now, but have plans for an Android comparison, so I might need some pointers on that—is the dev slack a reasonable place to ask about that?
Sure - feel free to ping me there or continue here, whatever works better for you.
In a nutshell, the problem with Android Nougat and above is that apps do not trust user-added certificate authorities (CAs) by default, which makes it impossible to intercept their traffic in normal use. In contrast, all other certificate validation policies I know of mark user-added CAs as particularly trustworthy as they represent explicit user intent¹. We have asked the folks at Google on multiple occasions which threat this would defend against, but we have not received any satisfactory explanation. As it stands, this only makes privacy analyses needlessly difficult.
NCC Group’s blog post on the topic describes the two main ways how this can be bypassed: First, one can recompile individual apps (I’m not sure if that works with preinstalled apps); second, one can root their phone (which reduces your security level) and install some custom hooks. Both of course is far from ideal if you as a concerned citizen just want to see what your phone is sending out.
¹) For example, certificates signed by user-added CAs overrule HTTP Public Key Pinning in Firefox.