Can't connect to the flow when using as library


#1

Hello!
I’m new in programming and have a hard time reading code documentaton. Usually I work with examples, but it didn’t work for me now.
I would like to use methods described here, using mitmproxy as library, but don’t understand how: http://docs.mitmproxy.org/en/v0.18.2/scripting/api.html#mitmproxy.models.http.HTTPRequest

Earlier this year I wrote code that works with mitmproxy 0.15 or 0.16 (start mitmproxy, find right request, print it and replace its body from file).

import os
from mitmproxy import controller, proxy
from mitmproxy.proxy.server import ProxyServer


class MyProxy(controller):

    with open("ad.xml", "r") as ad:
        ad_data = ad.read()
        print ("open file: ", ad_data)

    def __init__(self, server):
        controller.Master.__init__(self, server)

    def run(self):
        try:
            return controller.Master.run(self)
        except KeyboardInterrupt:
            self.shutdown()

    def handle_request(self, flow):
        if 'ad.mail.ru/vast' in flow.request.url:
            print ("request: ", flow.request.host)
            print (flow.request.url, flow.request.path_components)
        flow.reply()

    def handle_response(self, flow):
        if 'ad.mail.ru/vast' in flow.request.url:
            print ("response: ", self.ad_data)
            flow.response.content = (self.ad_data)
        flow.reply()


config = proxy.ProxyConfig(port=8080)
server = ProxyServer(config)
ProxyInstance = MyProxy(server)
ProxyInstance.run()

But now it doesn’t work and I can’t write something similar to work with flow.
Right now I work with code from example, but can’t understand how to get access to methods like flow.url etc: github.com/mitmproxy/mitmproxy/blob/0.18.x/examples/flowbasic

from mitmproxy import flow, controller, options
from mitmproxy.proxy import ProxyServer, ProxyConfig


class MyMaster(flow.FlowMaster):
    def run(self):
        try:
            flow.FlowMaster.run(self)
        except KeyboardInterrupt:
            self.shutdown()

    @controller.handler
    def request(self, f):
        print("request", f)

    @controller.handler
    def response(self, f):
        print("response", f)


opts = options.Options(app_port=8080)
config = ProxyConfig(opts)
state = flow.State()
server = ProxyServer(config)
m = MyMaster(opts, server, state)
m.run()

#2

Ok. Looks like I made it.

from mitmproxy import flow, controller, options
from mitmproxy.proxy import ProxyServer, ProxyConfig

class MProxy(flow.FlowMaster):

    with open("ad.xml", "r") as ad:
        ad_data = ad.read()

    def run(self):
        try:
            flow.FlowMaster.run(self)
        except KeyboardInterrupt:
            self.shutdown()

    @controller.handler
    def request(self, f):
        #print("request", f)
        if 'ad.mail.ru' in f.request.url:
            print ("request: ", f.request.host)
            print (f.request.url, f.request.path_components)
            print (f.request.headers)

    @controller.handler
    def response(self, f):
        #print("response", f)
        if 'ad.mail.ru' in f.request.url:
            print ("response: ", self.ad_data)
            f.response.text = (self.ad_data)
            f.response.headers["Test-Header"] = "TEST"

opts = options.Options(cadir="~/.mitmproxy/")
config = ProxyConfig(opts)
server = ProxyServer(config)
state = flow.State()
m = MProxy(opts, server, state)
m.run()

#3

And now I have a strange feeling that I do it all wrong.
What the best way (tool) to write some scripts for automated tests?

I need to run instance of proxy, then execute scripts like:

  1. Navigate and make some actions with Selenium then verify specific request\response.
  2. Navigate somewhere and replace specific response from file then use Selenium to verify state.
  3. Add\remove specific header then make item 1-2.
    etc.

If I continue using mitmproxy as library - do I need experience to work with its code (right now I lack of experience with programming)?


#4

Still need some help. Anyone?

Right now I can start proxy in background with this code, but don’t understand how to manipulate with it (using api?):

from mitmproxy import master, controller, options
from mitmproxy.proxy import ProxyServer, ProxyConfig
import time
import threading

def background(f):
    '''
    a threading decorator
    use @background above the function you want to run in the background
    '''
    def bg_f(*a, **kw):
        threading.Thread(target=f, args=a, kwargs=kw).start()
    return bg_f

class MProxy(master.Master):

    @background
    def run(self):
        try:
            master.Master.run(self)
        except KeyboardInterrupt:
            self.shutdown()

    def shutdown(self):
            self.shutdown()

    @controller.handler
    def request(self, f):
        print("request", f)

    @controller.handler
    def response(self, f):
        print("response", f)

opts = options.Options(cadir="~/.mitmproxy/")
config = ProxyConfig(opts)
server = ProxyServer(config)
m = MProxy(opts, server)
m.run()

time.sleep(60) # there is should be another code
m.shutdown()

#5

Hi,

Sorry for the slow response - :christmas_tree:! :slight_smile:

I would strongly recommend upgrading to 1.0 and using our offical API instead of subclassing (Flow)Master : http://docs.mitmproxy.org/en/stable/scripting/overview.html
You can find examples at https://github.com/mitmproxy/mitmproxy/tree/master/examples/simple.

You can execute mitmproxy in headless mode using mitmdump -s script.py. If you are writing tests, you could for example sys.exit(exitcode) from your script to signal test success/failure.


#6

Hi!

Thanks for reply. I have some questions for inline scripts then:

  1. Is there some way to start script within PyCharm for easier debugging?
  2. Can it be combined with testing framework like py.test?

Right now i see it this way (but it seems very complicated):
I have a file with hundred of tests belongs to some functionality (e.g. GA counters), it uses Selenium for navigation and Proxy for verification (e.g. open main page, check presence of GA counters).

  1. Start mitmdump with inline script script.py
  2. Execute tests with py.test
  3. Each test modify script.py and make navigation
  4. Script.py make verification and store result in file
  5. Test check that file and make test pass\fail.

#7

Yes! If you just run mitmdump in debug mode, you can place regular breakpoints in your scripts. :slight_smile:

Yes, see http://docs.mitmproxy.org/en/stable/scripting/overview.html#testing.

Indeed. If you just want to check if some requests have been made, you could also interact with mitmproxy using mitmweb’s API: https://github.com/mitmproxy/mitmproxy/blob/88f3459c7d7fc44bf0e247cf7f018f8c53431e8d/mitmproxy/tools/web/app.py#L439-L458
/clear and /flows might be particularly helpful for your case. :slight_smile:

This API is internal though and may change at some point, even though we don’t have any plans here. You can also expose your own API to instruct mitmproxy, e.g. as in https://github.com/mitmproxy/mitmproxy/blob/master/examples/simple/wsgi_flask_app.py.


#8
  1. Is this actual guide to run in Debug mode? https://github.com/mitmproxy/mitmproxy/blob/master/examples/complex/remote_debug.py

  2. Unfortunately, I need to check request\response, modify response (status code, headers, body), drop connection, simulate lags etc. Looks like I can’t do this through Mitmweb’s API.

  3. I mean a little different thing (may be I still don’t understand correctly how it all works):
    I can access Mitmproxy API only when load my code as inline script (mitmdump -s script.py)?
    But this way I can’t load it through py.test (pytest sript.py).
    Each file with python scripts will contain a hundred of tests like these:

    def Test1:
    Mitmproxy: change response body if request satisfies condition
    Selenium: open page
    Selenium: make action
    Mitmproxy: check for correct request1 presence

    def Test2:
    Mitmproxy: change response body if request satisfies condition
    Selenium: open page
    Selenium: make action
    Mitmproxy: check for correct request2 presence


#9

I used a little helper file (mitmstarter.py) to run mitmdump + scripts from within PyCharm:

if __name__ == "__main__":
    import sys
    from mitmproxy.main import mitmdump
    sys.argv = ['mitmdump', '-s', 'my_script.py']
    mitmdump()

But that was with Py 2.x & mitmproxy 0.16 or so… didn’t have enough time since then to fiddle / update to current versions.
If it doesn’t work anymore, it still might be the right approach to run / debug from within PyCharm or any other IDE.


#10

Thank you! It works after a little import modifying:

if __name__ == "__main__":
    import sys
    from mitmproxy.tools.main import mitmdump
    sys.argv = ['mitmdump', '-s', 'script.py']
    mitmdump()

#11

Sorry, I should have been more clear. To debug mitmdump (or mitmproxy/mitmweb) in PyCharm, you have to pick venv/bin/mitmdump-script.py (venv/Scripts/mitmdump-script.py on Windows) as the file to run/debug.

Mitmproxy’s functionality is a bit limited when it comes to simulating lagging (you have to use the concurrent decorator for that, and that’s more of a workaround), but other than that all those should be possible.

You can also run mitmproxy in a separate thread of your application. https://github.com/mitmproxy/mitmproxy/blob/master/mitmproxy/tools/main.py should give you some hints on how to do that.