1

I have a scenario where I have to upload a file from flask app to a third party API. I have wrapped all API requests in Flask to control API usage. For this I have redirected the traffic from main route towards the api wrapper route with http 307 status to preserve the request body and in the API wrapper I have used request to post into third party API endpoint.

The problem is only file < 100KB gets send through the redirection request, having a file larger than 100 KB gets somehow terminated in the sending phase.

Is there any limit in the 307 redirection and payload size?

I tried debugging by watching the network timing stack trace, from there it seems the request is dropped in the sending phase.

Main Blueprint

@main.route('/upload/',methods=['POST','GET'])  
def upload(): 
    #for ajax call
    if request.method == 'POST'
        return redirect(url_for('api.file_push'),code=307)
    else:
        return render_template('file-upload.html')

API Blueprint 

@api.route('/upload/',methods=['POST'])
def file_push():
    upload_file = request.files['file']
    filename = urllib.parse.quote(upload_file.filename)
    toUpload = upload_file.read()
    result=requests.post(apiInterfaces.FILE_UPLOAD_INTERFACE+'/'+filename,files{'file':toUpload})
    return result

Yes, I can directly send post request to API endpoint from main route but I don't want to, it will destroy my system design and architecture.

Dsujan
  • 411
  • 3
  • 14
  • would you mind to explain how it will destroy your "system design and architecture"? – Srikanth Chekuri Dec 11 '19 at 05:47
  • I want a separate blueprint to handle all third API usages so that I could use it for mobile development. – Dsujan Dec 11 '19 at 05:55
  • Then don't add the upload route in the main, you already have it in API blueprint, register that blueprint with the app. Why did you introduce a hop? what was the reason to put the wrapper in main and then redirect? – Srikanth Chekuri Dec 11 '19 at 06:07
  • I have updated the code have a look. I need main to render a template in case of GET request. For POST(from ajax) I have to carry the request payload to the API wrapper so that it would send the post request to third party API with the same request payload. – Dsujan Dec 11 '19 at 06:15
  • You are not getting my point. You could eliminate that route in main and have upload and render code in API blueprint itself. – Srikanth Chekuri Dec 11 '19 at 06:22
  • 1
    Yes I could have. I can solve that problem that way. But its strange for a redirection to terminate for a large file size. I wanted to know why, how and when? It could be bug or something that I may have been missing out. – Dsujan Dec 11 '19 at 06:34
  • I am also experiencing this. Do you have a solution yet? – ndtreviv Jul 28 '20 at 16:09
  • 1
    @ndtreviv No. I tried debugging, searched here and there but no luck. So opted direct use of API with no redirection. – Dsujan Jul 30 '20 at 04:50

1 Answers1

0

I assume you're using Python, and possibly requests so this answer will be based on what I've learned figuring this out (debugging with a colleague). I filed a bug report with psf/requests. There is a related answer here which confirms my suspicions.

It seems that when you initiate a PUT request using requests (and urllib3), the entire request is sent before a response from the server is looked at, but some servers can send a HTTP 307 during this time. One of two things happen:

  1. the server closes the connection by sending the response, even if the client has not finished sending the entire file. In this case, the client might see a closed connection and you won't have a response you can use for redirect (this happens with urllib3>1.26.10 (roughly)) but requests is not handling this situation correctly
  2. the server sends the response and you re-upload the file to the second location (urllib3==1.26.3 has this behavior when using requests). Technically, there is a bug in urllib3 and it should have failed, but silently lets you upload...

However, it seems that if you are expecting a redirect, the most obvious solution might be to send a null byte via PUT first, get a valid response back for the new URL [don't follow redirects], and then use that response to do the PUT of the full file. With requests, it's probably something like

>>> import requests
>>> from io import BytesIO
>>> data = BytesIO(b'\x00')
>>> response = request.put(url, data=data, allow_redirects=False)
>>> request.put(response.headers['Location'], data=fp, allow_redirects=False)

and at that point, you'll be ok (assuming you only expect a single redirect here).

kratsg
  • 600
  • 1
  • 5
  • 17