Best way to parse HTTP messages programatically for automated tests

help
1

Hello,

I am evaluating software options for my automation tests in order to replace test cases that are normally preformed manually through a similar mitm tool Charles. I was a little surprised to find the interactive nature of mitmproxy and I am not sure the best way to automate my cases.

I need to use mitmproxy so I can search HTTP messages for certain URLS, various GET/POST parameters, and other similar tests. I immediately set out trying to dump flows to a file and then parse that to get my results. This didn’t work out as half of the files are garbage values (unless I did something wrong in the dump).

What’s the best way to do this? Should I use this to turn into the dump into har files and then do my parsing? Or are their enough tools within mitmproxy to where I can get the answers I need? I don’t have a very good grasp with how to filter the messages and the options there.

Some example test cases:
did the call happen?
is this call pointed to the correct feed?
Is the call pointed to the correct URL?

2

Hi @farnett44,

You want mitmproxy’s non-interactive counterpart, mitmdump. :slight_smile:

We retain all traffic as-is, and that may include gzip compression. This is probably what you describe as “garbage”. :wink: As a rule of thumb, you usually do not want to process our dumpfile format yourself.

Looking at your requirements, I would recommend using our scripting interface to build what you need. Here is a very simple example that checks if a certain request has been made in a recording:

example-script.py:

import sys

def request(flow):
    if flow.request.url == "http://example.com/foo":
        sys.exit(0)  # exit right away

def done():
    sys.exit(1)  # we did not see the request

Here’s how you would use it:

mitmdump -n -r flows.mitm -s example-script.py

The example script is in the mitmproxy 0.18 syntax, which we will release very soon. For mitmproxy 0.17, you’d just need to add a “context” argument to the functions (as shown here).

Does that help? :slight_smile:

3

Hello @mhils,

Thanks for such an informative response.

There were a few aspects that I was not understanding and that really helped clear some things up. The more I read, the more it seemed like using the scripting interface is the way to go.

This is exactly what I was trying to do since I don’t know python, but from examples I’ve read it seems like the scripting needed to parse the dumps is pretty basic.

I’ll try out the scripting interface and see how that goes.
Would you recommend http://docs.mitmproxy.org/en/stable/scripting/overview.html for examples and getting started?

4

Yes, that’s the place to look at! If you have any specific questions, we’re happy to help. :slight_smile: