Error while using external Python module in mitmproxy project

help
1

I have grabbed your development version of mitmproxy (on master branch now). And it works just fine. Mitmproxy and mitmdump functionalities are awesome. But I got some strange error behavior. I want to use some external module in my custom addon. For instance I want to use module lxml to parse some traffic content. But script fails to start.
I performed some investigation. Problem is next:
I have this little custom script (independent for now from mitmproxy) that works fine if script allocated outside mitmproxy folder. But as soon as I move it to mitmproxy addon folder it fails to start from command line.
I run it from cmd like python3 my-custom-script.py and got the error:

    Original exception was:
Traceback (most recent call last):
  File "parse_response_html_tree.py", line 1, in <module>
    from lxml import etree
  File "src/lxml/etree.pyx", line 102, in init lxml.etree
  File "/usr/lib/python3.5/functools.py", line 22, in <module>
    from types import MappingProxyType
  File "/home/mykola/git-repos/ga/mitmproxy/mitmproxy/types.py", line 2, in <module>
    import glob
  File "/usr/lib/python3.5/glob.py", line 5, in <module>
    import fnmatch
  File "/usr/lib/python3.5/fnmatch.py", line 38, in <module>
    @functools.lru_cache(maxsize=256, typed=True)
AttributeError: module 'functools' has no attribute 'lru_cache'

That could be lack of my Python knowledge. But any sort of help will be invaluable!
Thanks

2

Hi,

Your problem is that you execute your script in the mitmproxy folder, which happens to have a file called types.py, which clashes with the builtin types module. One solution should be calling python3 -m mitmproxy.yourscript or something along those lines so that it doesn’t take the script’s folder as a path for importing modules. That being said, if you are writing an addon for mitmproxy, there’s no need to move it into the mitmproxy tree, you can keep it outside and add it to mitmproxy using mitmproxy -s scriptname.

1 Like
3

Thanks a lot. You saved my day!

4

Can you, please, help with next item. Most probably for you this will be several minutes and for me that could be bunch of hours.
I want to use mitmpoxy with my custom script. If this custom script is single one it works well. But if it includes some imports form my custom modules (f.e. in the beginning of my custom script include from custom_module import CustomRequestInterceptor) it fails with exception ImportError: No module named 'custom_module'.
How to solve import problem in such situation?

5

You can import your custom module but their files have to stay in the same folder with the mitmproxy script.

folder
 -> my_script.py
 -> custom_module.py

mitmproxy -s folder/my_script.py

6

Yeah, with such structure it’s OK. But I need multipackage structure. I was able to solve that importing issue. I such case:

  • allocate my_script.py is in the root of the project. In such way root goes to PYTHONPATH
  • everywhere use absolute imports like from package_name.custom_module import CustomRequestInterceptor
  • run mitmporoxy as you mention mitmproxy -s project_root/my_script.py

But thank you so much for the response.

1 Like