Emulate Charles Proxy's Map Local

I am trying to emulate Charles Proxy’s Map Local feature. I have tried a few different things which seem relatively simple but I cannot seem to get them to work.

To test out what I am doing I am trying to access http://getbootstrap.com/ and replace bootstrap.min.css with a local test file that has the contents:

  • {
    background: orange !important;

From mitmproxy I’ve gone into the replacements interface (o then R) and added a row that includes
Filter: ~a bootstrap.min.css
Regex: .*
Replacement: ~/test.css

using this method, every new line in bootstrap.min.css is replaced with the text “~/.test.css” as opposed to the file contents. I can’t seem to figure out how to tell mitmproxy to read from file.

When using mitmdump with the following launch command(s) bootstrap.min.css never seems to be replaced:
mitmdump "–replace-from-file :~a bootstrap.min.css::~/test.css"
mitmdump --replace-from-file :~a:bootstrap.min.css:$HOME/test.css
mitmdump “–replace-from-file :~a:bootstrap.min.css:~/test.css”

I have a feeling that I am missing something simple and just don’t understand exactly what it is that I need to do to get it to read from file appropriately.


AFAIK, replace from file will NOT work from the replacement interface (o then R) so you’ll have to use the --replace-from-file option.

This is the command that works for me:

mitmproxy --replace-from-file ":~a bootstrap.min.css:[^\Z]*:/full/path/to/test.css"


replace-from-file expects 3 things:

  1. A filter that can match a flow: ~a bootstrap.min.css
  2. A regex that can match the content in the flow: [^\Z]*
  3. Full path to a file. /full/path/to/test.css
    (tilde ~ expansion won’t work!)

Now, about that nasty looking regex!

As you’ve experienced yourself .* matches everything but only in a single line - so the replacement will be repeated for every line in the file.

To match everything in the file you need to be crafty :stuck_out_tongue:

Just like \w matches a word, \Z matches the end-of-string. (reference)

So, [^\Z]* matches all characters except the end-of-string (essentially everything.)

1 Like

Thank you so much, dufferzafar. That appears to be capturing the file and replacing it with the contents I’m expecting, I can’t quite say for certain what it’s doing though because it replaces the content 14 times. I checked versus the original bootstrap.min.css and it only has 6 lines of content so it doesn’t appear to be correlated to that.

To draw a strange correlation, it looks like the response contains 14 headers, so I’m wondering if it’s replacing the content once per each header? This would make me believe that it’s an issue with the filter or a bug in mitmdump. Could it be that the filter is triggering multiple times on the same response (once for each header)?

@synaestheory Sorry, but I couldn’t reproduce the 14 replacements thing!

Can you please share a screenshot etc. of what you’re getting?

The contents of test.css are:

body {
  background: orange;

Running this command:
mitmdump --replace-from-file “:~a bootstrap.min.css:[^\Z]*:$HOME/test.css”

I’ve also tried the fully qualified path (instead of using $HOME) and get the same result.

The output from chrome DevTools for bootstrap.min.css:

I’m running mitmdump v0.17.1 on OSX El Capitan 10.11.6

Hm, so that’s bad!

I was not able to reproduce this because I was using some other content in the test.css. But now I can. And I’m out of ideas as to why this is happening.

And since I’m on 0.18 - I’m getting something more weird:

(mitmproxy --replace-from-file ":~a bootstrap.min.css:[^\Z]*:test.css")

One workaround for this is to use a newline character in the regex (assuming, the original reponse body has atleast one newline character)

Use this then, this should preserve the headers:

mitmproxy --replace-from-file ":~a bootstrap.min.css:[^\Z]*\n[^\Z]*:hhh.css"

mitmproxy v18

I tried with your supplied regex and am still getting multiple replacements like in my and @dufferzafar’s screenshots above.

The only reason I suggested that it might be related to headers was because of the number of repetitions of content of test.css that I am getting versus the number of headers on the request. However, it looks like @dufferzafar might be getting more repetitions than I was so the header bit might be a red herring.

This now needs some love from brains bigger than ours.

@cortesi @mhils @kriechi - can you please have a look?

New regex works for me fine - 0.17 and 0.18 (OSX el capitan)

Try clearing your cache, browser must have cached the older replacement. (css fetch should not have 304 status code)

My mistake @mkagenius. It does appear to be working with your regex. I had been using a bash alias to run the command and after updating it to your regex I forgot to source my alias file to update the command.

Thanks for this thread - this is super helpful! :slight_smile:
Looks like we have a couple of usability problems:

  1. Replacements are applied to both headers and content. The longer I think about it, the more I believe we should have separate switches for that - maybe --replace and --replace-header
  2. .* works on a per-line basis. it looks like passing re.DOTALL for FBod/FBodRequest/FBodResponse may be the more reasonable default. I’d be happy to merge a PR that changes this in mitmproxy/filt.py :smiley:
  3. File-Replacements cannot be specified from within mitmproxy. I see three ways we could implement this:
    3.1 Implement a separate editor for file replacements
    3.2 Introduce a magic file: prefix for replacements. If a replacement starts with file:, the remainder is treated as a file path. We need to be a bit careful here for mitmweb as that allows reading arbitrary files from the UI.
    3.3 add a boolean flag in the grid editor that indicates file replacements. We need to be a bit careful here for mitmweb as that allows reading arbitrary files from the UI.

I’d favor option 3, but I don’t have a strong opinion on this. @dufferzafar, @kriechi, @cortesi, thoughts?

One additional thing that I’m noticing as I start to use this on a larger (2.7mb) file replace. I haven’t had the opportunity to dig too deeply into it just yet, but using the same filter/regex/file-replace it seems like the response gets stuck. Initially I thought it was due to the multiple replace that was happening with the first regex, however, even with the new regex the response seems to take an extremely long time. For that I’m also using a javascript file, which I don’t think would make a huge difference, but who knows. I’ll see if I can get a test scenario set-up to reproduce the issue when I have some more time.

Thank you everyone for sticking with this. I feel like options 1 and 3 may be safer overall (if my opnion holds any weight) :stuck_out_tongue:

Option 2 seems like it could break regexes that people already have in place.