Issue importing BeautifulSoup into a script

help
1

I’m trying to get BeautifulSoup imported into a mitmdump script and am stuck. I’ve verified that b4 is installed in the same python runtime that mitmdump is executing and even tried forcing the path using os.environ[‘PYTHONPATH’] inside the script. doesn’t work.

Fails with this error:

Script error: Traceback (most recent call last):
  File "script.py", line 9, in <module>
    from b4 import BeautifulSoup
ImportError: No module named 'b4'

Please help!

import sys
from mitmproxy import io
import os

    #import pdb; pdb.set_trace()

    from b4 import BeautifulSoup

    class Writer:
        def __init__(self, path):
            self.f = open(path, "wb")

        def response(self, flow):
            self.f.write( bytes(flow.request.url,'utf-8'))
            soup = BeautifulSoup(flow.response.content, 'html.parser')
            self.f.write( flow.response.content )

    def start():
        return Writer("out")
2

Never mind, I’m an idiot, looking at the mitmscript that brew installs, I saw “cellar” in the file path and skipped to the assumption that it was brew’s copy of python rather than it’s own executable.

putting
/usr/local/Cellar/mitmproxy/1.0.2/libexec/bin/

first in my path and correctly setting PYTHONPATH to
/usr/local/Cellar/mitmproxy/1.0.2/libexec/lib/python3.6

and then running
pip3 install BeautifulSoup4

fixed the issue.