Control caching and streaming via scripting

help
1

Hey there,

I stumbled across mitmproxy a few weeks ago and started to develop my own software which utilize mitmproxy.

So far everything works as expected. I run mitmproxy behind an Apache Webserver in transparent mode. During the development two problems occurred which I couldn’t solved yet.

First one is caching. About 75% of all requests which are processed by mitmproxy could be cached. But I would like to implement the logic which response should be cached and for how long in my Python software.

As far as I have seen it, there is no official way of doing this via python. I am fine with implementing the logic myself but do you have any hints how I could tell mitmproxy that it should’t pass the response to the requested server but serve the cached response?

My second problem is streaming. Some of the files passed through mitmproxy are pretty large (1-10GB). I have ready the page about response streaming: http://docs.mitmproxy.org/en/stable/features/responsestreaming.html

When I use the parameter --stream 1m everything works fine (either its a simple request with just a few kb or its a large file and therefore streamed). How could I do this via code? Setting flow.response.stream in the response function is to late because mitmproxy would try to load the whole file into RAM.

Thank you very much for your help and this awesome project!

Best regards,
Jens

2

I make a simple response myself with

flow.response = mitmproxy.http.HTTPResponse.make(503, b"ERROR", { “Content-Type”: “text/html” })

and then mitmproxy will not call the server returning client my response.

You can also serve a cached response in a file: you can find info in the docs.

1 Like
3

How could I do this via code? Setting flow.response.stream in the response function is to late because mitmproxy would try to load the whole file into RAM.

Well as you said, that’s too late. It works better if you modify it in the request function. :wink:

Other than that: What @Gigiarum said, thanks! :slight_smile:

1 Like
4

Thank you very much for the reply @Gigiarum and @mhils

I tried to modify the flow.response.stream option in my request function but the response isn’t available at that time so I got an error. Would be great if you can give me a hint on that too :smile:

5

Well damn, good point. :smile:
responseheaders() should work.