Replay_request does not seem to trigger request or response scripts

help
1

Hi, I’ve had a longish script from mitmproxy’s 0.8 days that I’m trying to migrate to the current stable 0.17 version. My script had some logic that would do some processing based on a particular request and response, duplicate the existing flow, modify the request and replay it. The response to the replayed request would be processed by the same response handler and go through the whole sequence again.

In 0.17, the replayed request and response do not seem to trigger the request and response handlers of the inline script anymore. Is this intentional, or did I fudge something up? Is there a way to mark the replay_request’d flow so that it is processed by the scripts again?

  1. Simple way to reproduce, which I thought should duplicate endless requests:

    def response(context,flow):
    newflow=context.duplicate_flow(flow)
    newflow.request.path="/"
    context.replay_request(newflow)
    `

  2. On a related note, can a script create a new flow from scratch, instead of having to duplicate an existing one?

Thanks!

1 Like
2

Hi @twaway!

  1. context.replay_request does not trigger script hooks anymore, but it is a blocking operation. You can immediately inspect newflow.response after calling replay_request.
  2. We don’t have that feature currently, but it should be straightforward to add. Let me know if you’d like to get your hands dirty and give it a go. :slight_smile:
3

Ahh, thanks.

  1. I suppose I could recursively call response on the newflow, but the call stack would be quite large as my existing script goes through a chain of several thousand requests/responses before it completes. So I’d really want a way to regain the previous behaviour, unless there is some reason that is not advisable.
    Looking at the source code, it seems that there are params block and run_scripthooks which default to blocking and not running hooks. I suppose that setting them to False and True respectively should do what I require… ie, I could call replace my calls to context.replay_request with context._master.replay_request(newflow)
  2. While looking at the flow.py, I realised there is also a create_request function. So the solution seems to be
    context._master.create_request(method='GET',scheme='http',host='host.domain.com',path='/path/to/resource',port=80)

Do you see any problem with my solution? Thanks!

4

This looks fine, but be warned that going through _master this is not part of the official API and subject to change. :wink:

5

I think we’re trying to do very similar things. I need to take an already created request, replay it until I get a particular response, then go through a chain of GETs and POSTs created from scratch using information from the initial request.

Is there any chance I can take a look at where you’re at? Any help would be super appreciated.

6

I took the easy way out and replaced calls to context.replay_request with
context._master.replay_request

7

Did you notice any strange behavior when you used the create_request? When I use it, instead of creating a single request, it creates the 75 copies of the same one.

8

No, it worked fine for me…

9

twaway, any chance you can post your code for this? I’ve been trying to get response hook retries working in 0.18 with no luck. (My use case is to retry a few times inside the proxy if the destination returns a flaky 503 response).

10
  • Current behaviour; replayed flow does not trigger further the response handler again
def response(context,flow):
      newflow=context.duplicate_flow(flow)
      newflow.request.path="/"
      context.replay_request(newflow)

Pasted image879×193 2.57 KB

11
  • unofficial way to return to 0.8 behaviour; replayed flow triggers request and response handlers again
def response(context,flow):
      newflow=context.duplicate_flow(flow)
      newflow.request.path="/"
      context._master.replay_request(newflow)

Pasted image875×289 4.32 KB

The above works for me on mitmproxy 0.17