OS X Terminal traffic doesn't automatically appear in mitmproxy

help
1

Hi there! Just recently started using mitmproxy. I’m an amateur with the tool and everything surrounding it, so please forgive any misunderstandings or misuses of terminology in advance and feel free to swiftly correct any mistakes!

I’m in OS X 10.11.6, and I’m using mitmproxy 0.17.1. I’ve got a Network Location configured to send traffic through mitmproxy with what I believe are the default settings: 127.0.0.1, port 8080 for both HTTP and HTTPS. My mitmproxy root certificate is installed and working as intended for intercepting secure requests.

I was using the proxy all morning to troubleshoot some HTML/JavaScript/Ruby that I’m stumbling through writing, and everything was working great, as it has been for a few weeks. When I started, I was mostly using curl to send requests directly to a development virtual machine on my Mac running my Ruby application server. With those requests and the subsequent responses displayed in mitmproxy, I could ensure that my Ruby worked as I intended.

However, I must’ve changed something, because at some point, any HTTP or HTTPS requests I make via Terminal – with brew, curl, or git – simply stopped appearing in mitmproxy. I’ve tried a variety of hosts, and nothing shows up.

All of the HTTP and HTTPS traffic that originates in other apps seems fine: I see stuff flying by from Safari and Spotlight, as well as requests to iCloud and some other third-party apps I use, for example.

I’ve tried switching Network Locations back and forth to my day-to-day configuration and back to my mitmproxy configuration, tried quitting Terminal and starting it fresh, and tried rebooting my Mac, ensuring that apps that were running aren’t re-launched after logging back in.

Any ideas? Thanks for reading!

2

I had the idea that perhaps Terminal wasn’t using my systemwide proxy settings. I had a look when I was connected to my usual Network Location without mitmproxy and saw no proxy enabled…

mbp:~ mpollard$ scutil --proxy
<dictionary> {
  ExceptionsList : <array> {
    0 : *.local
    1 : 169.254/16
  }
  FTPPassive : 1
}

…and then again when using my mitmproxy Network Location with the proxy enabled and verified as being functional via other applications sending HTTP and HTTPS traffic:

mbp:~ mpollard$ scutil --proxy
<dictionary> {
  ExceptionsList : <array> {
    0 : *.local
    1 : 169.254/16
  }
  FTPPassive : 1
  HTTPEnable : 1
  HTTPPort : 8080
  HTTPProxy : 127.0.0.1
  HTTPSEnable : 1
  HTTPSPort : 8080
  HTTPSProxy : 127.0.0.1
}

Additionally:

mbp:~ mpollard$ networksetup -getwebproxy Wi-Fi
Enabled: Yes
Server: 127.0.0.1
Port: 8080
Authenticated Proxy Enabled: 0

Looks like it’s enabled to me… but I’m not sure of a way to test and make sure that traffic is, in fact, going through the proxy or not other than checking to see whether it appears in mitmproxy or not.

3

And finally, if I use the proxy option for curl like so and explicitly point it to mitmproxy like so…

curl --proxy localhost:8080 'https://mitmproxy.org'

…the connection does appear in the mitmproxy interface. But as for how this changed and why connections originating in Terminal suddenly don’t appear there, I’m still lost!

4

Hi @mpollard - that indeed looks weird. Do you possibly have a http_proxy env var defined? If not, can you fix things by setting one? Maybe it’s also helpful to take a look with WireShark and see where your curl requests are currently going to. :slight_smile:

5

That worked, thanks! Exporting http_proxy and https_proxy in my session forced traffic originating there to go through the proxy (and thus, to appear in mitmproxy) by default.

I have no idea how I managed to break this… but I’ll take your solution :wink: I’ll write up a quick check for this to add to my .profile and post it here in case anyone else runs across this problem in OS X or macOS.

Thank you, Maximilian – both for mitmproxy and the helping hand!

1 Like
6

It’s ugly, and especially so since El Capitan uses bash 3, but I hope this helps if someone runs into the same problem.

Updated: Since my Terminal no longer intelligently detects proxies, it got to be a pain if I was switching back and forth between using mitmproxy for all traffic and using no proxy. I created a detect-proxy function and a disable-proxy to solve this.

Add this to ~/.profile:

# Proxy Detection

alias disable-proxy="export http_proxy= ; export https_proxy="

function detect-proxy {
  declare -a proxy_protocols=("HTTP" "HTTPS")

  for protocol in "${proxy_protocols[@]}"
  do
    if $(scutil --proxy | grep -q "${protocol}Enable : 1"); then
      proxy_ip=$(scutil --proxy | grep "${protocol}Proxy" | awk '{print $3}')
      proxy_port=$(scutil --proxy | grep "${protocol}Port" | awk '{print $3}')
      export "$(echo $protocol | tr '[:upper:]' '[:lower:]')=${proxy_ip}:${proxy_port}"
    else
      disable-proxy
    fi
  done
}

detect-proxy

New Terminals will now automatically pick up your systemwide proxy configuration; you can use detect-proxy or disable-proxy to do this on demand or bypass the proxy altogether per bash process.

(Oh, and if anyone figures out how I’ve gotten myself into a situation where I need this hack, please, let me know!)