3.0.3 crashing on updated arch linux

help
1

Good afternoon mitmproxy community,

I seem to be experiencing an issue while trying to run mitmproxy v3.0.3 transparently on a fully updated version of arch linux. I have installed mitmproxy from both pip and from pacman, both seem to be affected. I start mitmdump or mitmproxy in transparent mode with the firewall rules in place to redirect traffic to port 8080 and I can see the first connection start, but then the program halts and I can’t sigterm kill it. From top, it doesn’t appear that there are any resources being utilized any more, and the browser that was redirected does seem to load the page, but I never get any more output from mitmdump.

I should mention that v2.0.2 was running without encountering any such issues a week ago before I updated.

Mitmproxy: 3.0.3
Python: 3.6.4
OpenSSL: OpenSSL 1.1.0g 2 Nov 2017
Platform: Linux-4.15.6-1-ARCH-x86_64-with-arch

Thoughts?

2

Uh - complete halt is weird. You can’t interact with the mitmproxy UI at all anymore? Just to absolutely make sure that is a regression, could you test the precompiled binaries for 2.0 and 3.0?

3

That is correct, it will prevent any further interaction and keyboard interrupt doesn’t work. The same behaviour occurs on command line or with the gui.

I’m experiencing the same behaviour with 3.0 and 2.0. Some other update must be giving me grief.

4

I’ll try to provide as much information as I can.

  • I have two identical machines running arch linux
  • The deployment and configuration of these machines is performed using an archiso image that I threw together so they should be identical in that respect
  • Upon performing and update (pacman -Syu) on one of these machines, I noticed my mitmproxy script broke, no surprise as there was a new major release of the platform
  • upon reinstallation/reconfiguration of the updated archlinux machine, I am still experiencing the same issue

An exhaustive list of package differences between the two machines:

  • bash 4.4.018-1 -> 4.4.019-1
  • ddrescue 1.22-1 -> 1.23-1
  • dhclient 4.4.0-1 -> 4.4.1-1
  • dhcpcd 6.11.5-1 -> 7.0.1-1
  • git 2.16.1-1 -> 2.16.2-1
  • gnupg 2.2.4-2 -> 2.2.5-1
  • gnutls 3.5.17-1 -> 3.5.18-1
  • gpgme 1.10.0-1 -> 1.10.0-2
  • iana-etc 20180131-1 -> 20180221-1
  • iptables 1.6.1-2 -> 1.6.2-2
  • irssi 1.1.0-1 -> 1.1.1-1
  • json-glib 1.4.2-1 -> 1.4.2-2
  • less 487-1 -> 530-1
  • libdrm 2.4.89-1 -> 2.4.90-3
  • libepoxy 1.4.3-1 -> 1.5.0-1
  • libnghttp2 1.29.0-1 -> 1.30.0-1
  • libsystemd 237.31-1 -> 237.64-1
  • libxshmfence 1.2-1 -> 1.3-1
  • linux 4.15.3-1 -> 4.15.6-1
  • mesa 17.3.3-2 -> 17.3.6-1
  • mkinitcpio-busybox 1.27.2-1 -> 1.28.1-1
  • mpfr 4.0.0-1 -> 4.0.1-1
  • ncurses 6.0+20170902-3 -> 6.1-3
  • pacman-mirrorlist 20180203-1 -> 20180224-1
  • pango 1.40.14-1 -> 1.40.14-2
  • pcre2 10.30-1 -> 10.31-1
  • perl-io-socket-ssl 2.048-3 -> 2.055-1
  • procps-ng 3.3.12-2 -> 3.3.12-3
  • slang 2.3.1a-1 -> 2.3.1a-2
  • s-nail 14.9.6-1 -> 14.9.7-1
  • systemd 237.31-1 -> 237.64-1
  • systemd-sysvcompat 237.31-1 -> 237.64-1
  • vim 8.0.1476-1 -> 8.0.1542-2
  • vim-runtime 8.0.1476-1 -> 8.0.1542-2
  • xorgproto 2018.2-1 -> 2018.4-1

An exhaustive list of python package differences between the two machines:

  • argh 0.26.2 -> NONE
  • brotlipy 0.6.0 -> 0.7.0
  • cffi 1.11.4 -> 1.11.5
  • construct 2.8.22 -> NONE
  • cryptography 1.8.2 -> 2.1.4
  • cssutils 1.0.2 -> NONE
  • EditorConfig 0.12.1 -> NONE
  • h11 NONE -> 0.7.0
  • h2 2.6.2 -> 3.0.1
  • html2text 2016.9.19 -> NONE
  • hyperframe 4.0.2 -> 5.1.0
  • jsbeautifier 1.6.14 -> NONE
  • kaitaistruct 0.6 -> 0.8
  • ldap3 NONE -> 2.4.1
  • mitmproxy 2.0.2 -> 3.0.3
  • pathtools 0.1.2 -> NONE
  • pyasn1 0.2.3 -> 0.4.2
  • pyOpenSSL 16.2.0 -> 17.5.0
  • pyperclip 1.5.32 -> 1.6.0
  • PyYAML 3.12 -> NONE
  • ruamel.yaml 0.13.14 -> 0.15.35
  • tornado 4.4.3 -> 4.5.3
  • urwid 1.3.1 -> 2.0.1
  • watchdog 0.8.3 -> 0.11.0

I know some of those are more useful than others, but for the sake of completeness I wanted to include them all.

I’m pretty stumped here and can’t figure out what my next step should be.

Any help would be greatly appreciated!

5

Thanks so much. I’m still a bit ¯\_(ツ)_/¯, but this is really helpful. Some more things to try:

  1. This affects both mitmproxy and mitmdump, correct? In this case we can rule out an urwid/console issue.
  2. What’s the complete output of mitmdump -v until it halts?
  3. You mentioned that SIGTERM doesn’t work, so this probably won’t either: How about SIGUSR1/SIGUSR2 to get debug info?
  4. Does this happen for both HTTP and HTTPS? Does passing --tcp .* help with anything?
6

I will test these tonight/tomorrow and get back to you. At this time all I can say is that it affects both mitmdump and mitmproxy and I’ve only tested with http. It only occurs with transparent mode though :confused:

I’ll get back in the near future with details of the other steps. Thanks!

7

Transparent mode only is interesting, maybe you can add print statements around https://github.com/mitmproxy/mitmproxy/blob/master/mitmproxy/platform/linux.py and see if it hangs there somewhere (make sure to set PYTHONUNBUFFERED env var to true). That’s really the only transparent-mode specific code we have.

8

Hello, I can second that:

  • Archlinux 4.15.6-1
  • mitmproxy 3.0.3
  • python 3.6.4
  • openssl 1.1.0g
  1. mitmproxy and mitmdump hang in transparent mode. Process cannot be killed even with SIGKILL/TERM.
  2. mitmdump displays little information in log:
Proxy server listening at http://*:8080
192.168.136.157:46938: clientconnect

  1. USR1/2 don’t seem to add anything to log file or term output

  2. Passing --tcp .* doesn’t seem to help. Tried only with HTTP.

  3. dmesg output after mitm* hangs:

[ 1844.156810] INFO: task mitmdump:2074 blocked for more than 120 seconds.
[ 1844.157148]       Tainted: G           O     4.15.6-1-ARCH #1
[ 1844.157506] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1844.157847] mitmdump        D    0  2074    387 0x00000004
[ 1844.157850] Call Trace:
[ 1844.157857]  ? __schedule+0x24b/0x8c0
[ 1844.157860]  ? get_futex_key+0x303/0x3f0
[ 1844.157862]  schedule+0x32/0x90
[ 1844.157865]  __lock_sock+0x79/0xc0
[ 1844.157868]  ? wait_woken+0x80/0x80
[ 1844.157870]  lock_sock_nested+0x50/0x60
[ 1844.157875]  getorigdst+0x5a/0x240 [nf_conntrack_ipv4]
[ 1844.157877]  ? preempt_count_add+0x49/0xa0
[ 1844.157880]  nf_getsockopt+0x47/0x70
[ 1844.157883]  ip_getsockopt+0x7f/0xc0
[ 1844.157885]  ? SyS_getsockname+0xa9/0xe0
[ 1844.157887]  ipv6_getsockopt+0x4a/0x110
[ 1844.157890]  SyS_getsockopt+0x76/0xd0
[ 1844.157892]  do_syscall_64+0x74/0x190
[ 1844.157895]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[ 1844.157897] RIP: 0033:0x7fd0066d1d3a
[ 1844.157898] RSP: 002b:00007fcffcab7be8 EFLAGS: 00000206 ORIG_RAX: 0000000000000037
[ 1844.157900] RAX: ffffffffffffffda RBX: 00007fcffd2d8108 RCX: 00007fd0066d1d3a
[ 1844.157901] RDX: 0000000000000050 RSI: 0000000000000000 RDI: 0000000000000004
[ 1844.157901] RBP: 00007fcffcab7bf0 R08: 00007fcffcab7bf0 R09: 0000000000000000
[ 1844.157902] R10: 00007fcfff264a98 R11: 0000000000000206 R12: 00007fcffd2d8108
[ 1844.157903] R13: 0000000000000000 R14: 00007fd0022df980 R15: 00007fcffd2d1b80

I’m fairly sure mitmproxy worked with a 4.14.?.? kernel, I’ll add more information tonight.

Thank you for your awesome work on mitmproxy :slight_smile:

9

Downgrading to 4.15.3-1 solves this issue.

10

Line 27 is crashing:
dst = csock.getsockopt(socket.SOL_IP, SO_ORIGINAL_DST, 16)
poking around to see if I can find out anything.

I’ve taken the liberty of creating an issue and linking to this thread.

11

This commit is bundled into 4.15.7 and will likely correct the issue. Should hit the main arch repo shortly. Thanks for the help and amazing work!

12

You guys rock - thanks so much! :heart: :cake: :rocket: