First of all, I’m very sorry for asking questions about this, and I’ve seen the issue tracker filled up with problems very similar to mine.
However, I can’t find the solution after hours of tinkering, so hopefully somebody can point me in the right direction.
I’m running mitmproxy in transparent mode (mitmproxy -T --host) on a separate machine that I’ve set up as a router, and I’m routing trafic to mitmproxy via iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 8080 (same with port 443). My client connects to this machine directly.
Connecting to HTTP(S) sites works perfectly, but I’ve found a piece of software that makes some requests to the internet and, after a few of those, starts giving me the dreadful “HttpError(‘Invalid HTTP request form (expected: relative, got: absolute)’,”.
I’m at a loss, as everything works fine for the first 10 requests (1 of them HTTPS, the rest HTTP) but then it starts failing when the client starts doing some HTTP requests again. Capturing the requests I see that they have the port explicitly set (http://example.com:80) but I fail to see how that can be affecting anything.
Is there something I should be configuring, or something I can enable to find out what’s happening?
Looks like you have some application that is making absolute-form HTTP requests (e.g. GET http://example.com/foo HTTP/1.1 instead of GET /foo HTTP/1.1. In 99% of the cases, this means the application actually wants to talk to a proxy, which it should not want to do in case of a transparent proxy.
Using absolute-form to request resources from a normal HTTP server is permitted by the relevant RFCs, but it’s a bit painful for us as we now have two conflicting pieces of information about the target host - the unmalleable IP and the spoofable hostname in the URL. Adding the fact that this usually indicates a misconfiguration, we currently decide to just fail in this case, even though it technically could be valid HTTP.
Long story short, you can probably work around this by adjusting the validation here. We’d be happy to accept a PR that adds a command line switch for allowing this behaviour, but I’d be reluctant to make this the default behaviour as it usually indicates a configuration error.
Thanks for your response. I understand this might be an edge case, as the
app is properly configured (basically there is nothing to configure, it’s
just doing it’s own thing :D) but some of the requests didn’t work.
In the end I used a different MITM proxy, and all is good again :).