Using mitmproxy to record unit tests

help
1

We develop client libraries in numerous languages and would like to have server record/replay support across all our testing. Our scenarios are that third party contributors are able to download and test against recordings without incurring the cost of testing against our service as well as better support for GitHub Continuous Integration, which works better if running tests does not require any network requests. Ideally, we’d be able to have our test suites invoke this proxy in another process with some configurations and then have our tests run through the proxy and then shut down the proxy when our tests complete.

I have played with the proxy (and mitmdump, which seems more appropriate for our large scale usage) and looked at the docs, but I have had some difficulty determining if we would actually be able to meet all of our requirements and, beyond that, how to set it up and which options and commands to use to do so. Our requirements for this use case are as follows:

  • Ability to record client requests and server responses
  • Ability to run in replay mode in which client requests are matched to a recorded flow and the serve response is replayed
  • Ability to run all tests in replay mode sequentially (no need for concurrency, but we shouldn’t have to individually play every test)
  • Ability to save the flows from each test in their own files
  • Ability to ignore certain headers when matching, such as date headers which will always be obsolete
  • Ability to scrub sensitive information like Authorization headers and account names in a way that does not interfere with flow matching. This could be in the form of some callback which is called to scrub right before recording and right before matching. Something to that effect.

If you guys have any insight on how to get started with all of this, that would be really appreciated. Thank you so much!

2

Hi Rick,

This definitely is something that can be done with mitmproxy. Some folks at ustwo even wrote a framework on top of mitmproxy for that purpose (GitHub - ustwo/mastermind: Man in the middle testing), although that is quite a bit outdated nowadays unfortunately. Replies inline below.

Ability to record client requests and server responses

You run mitmproxy and then save at some point, or also run mitmdump -w filename.

Ability to run in replay mode in which client requests are matched to a recorded flow and the serve response is replayed

That pretty much describes mitmproxy’s server replay mode: Features

The entire server replay implementation is in an isolated mitmproxy addon: https://github.com/mitmproxy/mitmproxy/blob/master/mitmproxy/addons/serverplayback.py
This should give you a rough idea on how you can customize things.

Ability to run all tests in replay mode sequentially (no need for concurrency, but we shouldn’t have to individually play every test)

There are a bunch of ways to do this - you either add your own instrumentation, or just restart mitmproxy with individual recordings to be replayed, etc.

Ability to save the flows from each test in their own files

You can select a few flows in mitmproxy and only save those to a single file.

Ability to ignore certain headers when matching, such as date headers which will always be obsolete

See docs, there’s a server_replay_use_headers setting.

Ability to scrub sensitive information like Authorization headers and account names in a way that does not interfere with flow matching. This could be in the form of some callback which is called to scrub right before recording and right before matching. Something to that effect.

You can use the replacements functionality, write your own custom addon that scrubs sensitive data, or directly roll your own customized server replay addon.

Best,
Max

3

Thank you so much for the help and quick response! I’ll give this a try in an upcoming sprint and follow up if I run into any problems.