# Mitmproxy not working with urls which have 'under\_scores'

**URL:** <https://discourse.mitmproxy.org/t/mitmproxy-not-working-with-urls-which-have-under-scores/156>\
**Category:** help\
**Created:** [August 30, 2016, 11:50am UTC](https://discourse.mitmproxy.org/t/mitmproxy-not-working-with-urls-which-have-under-scores/156 "2016-08-30T11:50:58Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![qakapil](https://avatars.discourse-cdn.com/v4/letter/q/9f8e36/32.png) [@qakapil](https://discourse.mitmproxy.org/u/qakapil)\
**Post date:** [August 30, 2016, 11:50am UTC](https://discourse.mitmproxy.org/t/mitmproxy-not-working-with-urls-which-have-under-scores/156/1 "2016-08-30T11:50:58Z")

</div>

Hi,  
I have been using mitmproxy to test our iOS app and till now it was working really well. But with the latest build of our app the developers have made some changes to the way device talks to the API end points and now mitmproxy is blocking the traffic to new endpoints.  
Here is the change which our devs have made. As an example, till now iOS device used to talk to our endpoints with urls which looked like this - '[https://api.myservice.com/cpp/link/RegisterClient](https://api.myservice.com/cpp/link/RegisterClient)’  
But now with the latest changes, the endpoint has been pre-fixed at runtime with deviceID, arch and OS info. So the new url looks like this - 'https://a3jt99zj340-52f8b8d-os9-3-x86\_64-ver3\_9X.api.myservice.com/cpp/link/RegisterClient’  
and a call to this endpoint with mitmproxy enabled fails. I did a bit of experiment and noticed that if I replace the underscores in the above endpoint with hyphens then it works - ‘[https://a3jt99zj340-52f8b8d-os9-3-x86-64-ver3-9X.api.myservice.com/cpp/link/RegisterClient](https://a3jt99zj340-52f8b8d-os9-3-x86-64-ver3-9X.api.myservice.com/cpp/link/RegisterClient)’

Would someone help me in understanding why mitmproxy is not working with endpoints which have underscores ?  
Here is the error message which it throws. The error is in requests library but then why does it work fine with mitmproxy disabled ?

Traceback (most recent call last):  
File “/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py”, line 1531, in   
globals = debugger.run(setup[‘file’], None, None, is\_module)  
File “/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py”, line 938, in run  
pydev\_imports.execfile(file, globals, locals) # execute the script  
File “/Users/kapshar/dev/AIViOSIntegration/Testing/test.py”, line 70, in   
params=params,  
File “/Library/Python/2.7/site-packages/requests/api.py”, line 57, in request  
return session.request(method=method, url=url, \*\*kwargs)  
File “/Library/Python/2.7/site-packages/requests/sessions.py”, line 475, in request  
resp = self.send(prep, \*\*send\_kwargs)  
File “/Library/Python/2.7/site-packages/requests/sessions.py”, line 585, in send  
r = adapter.send(request, \*\*kwargs)  
File “/Library/Python/2.7/site-packages/requests/adapters.py”, line 440, in send  
raise ProxyError(e, request=request)  
requests.exceptions.ProxyError: HTTPSConnectionPool(host=’/a3jt99zj340-52f8b8d-os9-3-x86-64-ver3-9X.api.myservice.com’, port=443): Max retries exceeded with url: /cdp/link/RegisterClient?geolocation=US&pageId=default&format=json&acceptedTerms=PreAccepted&deviceTypeID=AK6OCP77UJI1&pageType=browse&decorationScheme=ios-browse&featureScheme=ios-features-v1&firmware=5.80.66.5&version=ios-v3&deviceID=F1A735F4D86444443835508US (Caused by ProxyError(‘Cannot connect to proxy.’, error(‘Tunnel connection failed: 400 Bad Request’,)))

---

<div class="post-metadata">

**Author:** ![qakapil](https://avatars.discourse-cdn.com/v4/letter/q/9f8e36/32.png) [@qakapil](https://discourse.mitmproxy.org/u/qakapil)\
**Post date:** [August 31, 2016, 12:41pm UTC](https://discourse.mitmproxy.org/t/mitmproxy-not-working-with-urls-which-have-under-scores/156/2 "2016-08-31T12:41:24Z")

</div>

I looked at the verbose logs from mitmdump and can see that it gives a HTTPSyntaxException. Looks like mitmproxy doesn’t consider ‘under\_scores’ in urls a valid character. Here is the stacktrace -

File “/Library/Python/2.7/site-packages/netlib/http/http1/read.py”, line 38, in read\_request\_head  
form, method, scheme, host, port, path, http\_version = \_read\_request\_line(rfile)  
File “/Library/Python/2.7/site-packages/netlib/http/http1/read.py”, line 239, in \_read\_request\_line  
host, port = \_parse\_authority\_form(path)  
File “/Library/Python/2.7/site-packages/netlib/http/http1/read.py”, line 266, in \_parse\_authority\_form  
raise HttpSyntaxException(“Invalid host specification: {}”.format(hostport))  
ProtocolException: Error in HTTP connection: HttpSyntaxException(‘Invalid host specification: test\_test.api.myservice.com:443’,)

---

<div class="post-metadata">

**Author:** ![qakapil](https://avatars.discourse-cdn.com/v4/letter/q/9f8e36/32.png) [@qakapil](https://discourse.mitmproxy.org/u/qakapil)\
**Post date:** [August 31, 2016, 1:07pm UTC](https://discourse.mitmproxy.org/t/mitmproxy-not-working-with-urls-which-have-under-scores/156/3 "2016-08-31T13:07:17Z")

</div>

I think the fix should go into netlib/utils.py at line#211 -\>  
\_label\_valid = re.compile(b"(?!-)[A-Z\d-]{1,63}(?\<!-)$", re.IGNORECASE)  
In regex we should have a case for underscore also.

---

<div class="post-metadata">

**Author:** ![cortesi](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mitmproxy.org/cortesi/32/1_2.png) [@cortesi](https://discourse.mitmproxy.org/u/cortesi)\
**Post date:** [September 4, 2016, 11:12pm UTC](https://discourse.mitmproxy.org/t/mitmproxy-not-working-with-urls-which-have-under-scores/156/4 "2016-09-04T23:12:50Z")

</div>

Hi there. Underscores are definitely not a valid character in domain name. The question is what mitmproxy should do about it - we can be liberal and pass on errors like this upstream, or we can show an error and refuse to pass the traffic. This is a choice we face again and again, and the “right” answer is different in different situations.

On balance, in this case I feel we should be liberal and pass the invalid character upstream. Either way, we certainly shouldn’t show a traceback for this sort of issue. I’ve opened a bug on our tracker for this here:

[https://github.com/mitmproxy/mitmproxy/issues/1529](https://github.com/mitmproxy/mitmproxy/issues/1529)