I am trying to emulate Charles Proxy’s Map Local feature. I have tried a few different things which seem relatively simple but I cannot seem to get them to work.
To test out what I am doing I am trying to access http://getbootstrap.com/ and replace bootstrap.min.css with a local test file that has the contents:
{
background: orange !important;
}
From mitmproxy I’ve gone into the replacements interface (o then R) and added a row that includes
Filter: ~a bootstrap.min.css
Regex: .*
Replacement: ~/test.css
using this method, every new line in bootstrap.min.css is replaced with the text “~/.test.css” as opposed to the file contents. I can’t seem to figure out how to tell mitmproxy to read from file.
When using mitmdump with the following launch command(s) bootstrap.min.css never seems to be replaced:
mitmdump "–replace-from-file :~a bootstrap.min.css::~/test.css"
mitmdump --replace-from-file :~a:bootstrap.min.css:$HOME/test.css
mitmdump “–replace-from-file :~a:bootstrap.min.css:~/test.css”
I have a feeling that I am missing something simple and just don’t understand exactly what it is that I need to do to get it to read from file appropriately.
Thank you so much, dufferzafar. That appears to be capturing the file and replacing it with the contents I’m expecting, I can’t quite say for certain what it’s doing though because it replaces the content 14 times. I checked versus the original bootstrap.min.css and it only has 6 lines of content so it doesn’t appear to be correlated to that.
To draw a strange correlation, it looks like the response contains 14 headers, so I’m wondering if it’s replacing the content once per each header? This would make me believe that it’s an issue with the filter or a bug in mitmdump. Could it be that the filter is triggering multiple times on the same response (once for each header)?
I was not able to reproduce this because I was using some other content in the test.css. But now I can. And I’m out of ideas as to why this is happening.
And since I’m on 0.18 - I’m getting something more weird:
I tried with your supplied regex and am still getting multiple replacements like in my and @dufferzafar’s screenshots above.
The only reason I suggested that it might be related to headers was because of the number of repetitions of content of test.css that I am getting versus the number of headers on the request. However, it looks like @dufferzafar might be getting more repetitions than I was so the header bit might be a red herring.
My mistake @mkagenius. It does appear to be working with your regex. I had been using a bash alias to run the command and after updating it to your regex I forgot to source my alias file to update the command.
Thanks for this thread - this is super helpful!
Looks like we have a couple of usability problems:
Replacements are applied to both headers and content. The longer I think about it, the more I believe we should have separate switches for that - maybe --replace and --replace-header
.* works on a per-line basis. it looks like passing re.DOTALL for FBod/FBodRequest/FBodResponse may be the more reasonable default. I’d be happy to merge a PR that changes this in mitmproxy/filt.py
File-Replacements cannot be specified from within mitmproxy. I see three ways we could implement this:
3.1 Implement a separate editor for file replacements
3.2 Introduce a magic file: prefix for replacements. If a replacement starts with file:, the remainder is treated as a file path. We need to be a bit careful here for mitmweb as that allows reading arbitrary files from the UI.
3.3 add a boolean flag in the grid editor that indicates file replacements. We need to be a bit careful here for mitmweb as that allows reading arbitrary files from the UI.
One additional thing that I’m noticing as I start to use this on a larger (2.7mb) file replace. I haven’t had the opportunity to dig too deeply into it just yet, but using the same filter/regex/file-replace it seems like the response gets stuck. Initially I thought it was due to the multiple replace that was happening with the first regex, however, even with the new regex the response seems to take an extremely long time. For that I’m also using a javascript file, which I don’t think would make a huge difference, but who knows. I’ll see if I can get a test scenario set-up to reproduce the issue when I have some more time.
Thank you everyone for sticking with this. I feel like options 1 and 3 may be safer overall (if my opnion holds any weight)
Option 2 seems like it could break regexes that people already have in place.
