Test scenario with multiple clients needing the same "live" data


#1

Hello everyone!

I am trying to set up a text environment where I am getting data from an external web service from multiple clients on my network. These clients are test code that are configured in a specific way and uses the incoming data to calculate some output I want to analyze. I need the exact same data to be distributed to the clients.

Sadly the web service is really not liking it when I gear up the amount of clients requesting data from it, and blocks my API-KEY almost immediately after starting up more than 2. I need around 100 at least.

So I have an idea of setting up a proxy that cached all the server responses from 1 of the clients requests and serves that up to the others.

The clients can be configured to direct the traffic directly to mitmproxy, so I dont need a transparent proxy. I am thinking a reverse proxy is the way to go here, but maybe a forward proxy will work just as well, if not better (easier)?

Some caveats:

  • The server requires HTTPS calls
  • The server responses have all the no-cache rules set

Im pretty sure mitmproxy can handle this for me. Am I right about that?

A few questions:

  • Can I let mitmproxy handle the https part, and just let the client communicate to the proxy with http?
  • How do I make mitmproxy ignore all the no-cache directives in the response and control when it should refresh the response with new data?

I have read the docs, but cant seem to find much about cache …in fact. Is there any more documentation than what is served at https://docs.mitmproxy.org… ? It doesnt seem to cover all the options mitmproxy system gives?

Thanks for any responses!

/Tomas