Mitmproxy not working with urls which have 'under_scores'

help
1

Hi,
I have been using mitmproxy to test our iOS app and till now it was working really well. But with the latest build of our app the developers have made some changes to the way device talks to the API end points and now mitmproxy is blocking the traffic to new endpoints.
Here is the change which our devs have made. As an example, till now iOS device used to talk to our endpoints with urls which looked like this - 'https://api.myservice.com/cpp/link/RegisterClient’
But now with the latest changes, the endpoint has been pre-fixed at runtime with deviceID, arch and OS info. So the new url looks like this - 'https://a3jt99zj340-52f8b8d-os9-3-x86_64-ver3_9X.api.myservice.com/cpp/link/RegisterClient’
and a call to this endpoint with mitmproxy enabled fails. I did a bit of experiment and noticed that if I replace the underscores in the above endpoint with hyphens then it works - ‘https://a3jt99zj340-52f8b8d-os9-3-x86-64-ver3-9X.api.myservice.com/cpp/link/RegisterClient’

Would someone help me in understanding why mitmproxy is not working with endpoints which have underscores ?
Here is the error message which it throws. The error is in requests library but then why does it work fine with mitmproxy disabled ?

Traceback (most recent call last):
File “/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py”, line 1531, in
globals = debugger.run(setup[‘file’], None, None, is_module)
File “/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py”, line 938, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File “/Users/kapshar/dev/AIViOSIntegration/Testing/test.py”, line 70, in
params=params,
File “/Library/Python/2.7/site-packages/requests/api.py”, line 57, in request
return session.request(method=method, url=url, **kwargs)
File “/Library/Python/2.7/site-packages/requests/sessions.py”, line 475, in request
resp = self.send(prep, **send_kwargs)
File “/Library/Python/2.7/site-packages/requests/sessions.py”, line 585, in send
r = adapter.send(request, **kwargs)
File “/Library/Python/2.7/site-packages/requests/adapters.py”, line 440, in send
raise ProxyError(e, request=request)
requests.exceptions.ProxyError: HTTPSConnectionPool(host=’/a3jt99zj340-52f8b8d-os9-3-x86-64-ver3-9X.api.myservice.com’, port=443): Max retries exceeded with url: /cdp/link/RegisterClient?geolocation=US&pageId=default&format=json&acceptedTerms=PreAccepted&deviceTypeID=AK6OCP77UJI1&pageType=browse&decorationScheme=ios-browse&featureScheme=ios-features-v1&firmware=5.80.66.5&version=ios-v3&deviceID=F1A735F4D86444443835508US (Caused by ProxyError(‘Cannot connect to proxy.’, error(‘Tunnel connection failed: 400 Bad Request’,)))

2

I looked at the verbose logs from mitmdump and can see that it gives a HTTPSyntaxException. Looks like mitmproxy doesn’t consider ‘under_scores’ in urls a valid character. Here is the stacktrace -

File “/Library/Python/2.7/site-packages/netlib/http/http1/read.py”, line 38, in read_request_head
form, method, scheme, host, port, path, http_version = _read_request_line(rfile)
File “/Library/Python/2.7/site-packages/netlib/http/http1/read.py”, line 239, in _read_request_line
host, port = _parse_authority_form(path)
File “/Library/Python/2.7/site-packages/netlib/http/http1/read.py”, line 266, in _parse_authority_form
raise HttpSyntaxException(“Invalid host specification: {}”.format(hostport))
ProtocolException: Error in HTTP connection: HttpSyntaxException(‘Invalid host specification: test_test.api.myservice.com:443’,)

3

I think the fix should go into netlib/utils.py at line#211 ->
_label_valid = re.compile(b"(?!-)[A-Z\d-]{1,63}(?<!-)$", re.IGNORECASE)
In regex we should have a case for underscore also.

4

Hi there. Underscores are definitely not a valid character in domain name. The question is what mitmproxy should do about it - we can be liberal and pass on errors like this upstream, or we can show an error and refuse to pass the traffic. This is a choice we face again and again, and the “right” answer is different in different situations.

On balance, in this case I feel we should be liberal and pass the invalid character upstream. Either way, we certainly shouldn’t show a traceback for this sort of issue. I’ve opened a bug on our tracker for this here:

https://github.com/mitmproxy/mitmproxy/issues/1529