1

I am trying to upload videos to my Flask app but I am getting errors every time i try to upload larger files

When making this request

curl --location --request POST 'http://192.168.1.217:5000/api/fileservice/0.1/files/upload_file' \
--form 'mime_type="video/mp4"' \
--form 'uuid="12345"' \
--form 'file=@"/home/sean/Videos/long.mp4"' \
--form 'file_name="my-great-file.mp4"''

I first receive a 308 response

192.168.1.217 - - [30/Oct/2022 13:23:17] "POST /api/fileservice/0.1/files/upload_file HTTP/1.1" 308 -

But after this I keep getting the following response back indicating that the remote side of the stream being written to has been closed.

POST http://192.168.1.217:5000/api/fileservice/0.1/files/upload_file
Error: write EPIPE

My Postman logs show this below the error

mime_type: "video/mp4"
uuid: "12345"
file: undefined
file_name: "my-great-file.mp4"

Interesting that the file is undefined. I assume because Flask isn't processing the large file properly.

Also there were a few occasions where it managed to hit the endpoint code before failing, and it appeared to fail here (based on the print statements that were/weren't outputted)

@blueprint.route("/files/upload_file/", methods=["POST"])
def upload_file():
    """Upload the file meta data and return the file upload location. Accepts a multipart/form-data request"""
    print("test")
    form_data = flask.request.form
    file = flask.request.files["file"] ## FAILS HERE
    print("file", file)

    if not file:
        raise Exception("No file provided")

    byte_stream = file.read()

    request = FileCreateRequest(
        uuid=form_data["uuid"],
        file_name=form_data["file_name"],
        mime_type=form_data["mime_type"],
        bytes=byte_stream,
    )

    result = flask.current_app.conns.file_service.create_file(request=request)

    response = {}
    response["file"] = vars(result.file)

    response = flask.current_app.response_class(
        response=json.dumps(response), status=200, mimetype="application/json"
    )

    return response

I think the issue is because the file is large (1.6MB) because requests like this with a much smaller mp4 file(390KB) work fine

curl --location --request POST 'http://192.168.1.217:5000/api/fileservice/0.1/files/upload_file' \
--form 'mime_type="video/mp4"' \
--form 'uuid="12345"' \
--form 'file=@"/home/sean/Videos/Peek 2022-10-30 10-08.mp4"' \
--form 'file_name="my-great-file.mp4"'

Any ideas on how i can solve this issue?

Sean
  • 587
  • 4
  • 20

1 Answers1

0

I wasn't to familiar with 308 redirects before, but this seemed to be the actual cause of the issue.

The original reuqest was made to http://192.168.1.217:5000/api/fileservice/0.1/files/upload_file

It resulted in a 308 with a Location header set to http://192.168.1.217:5000/api/fileservice/0.1/files/upload_file/

This was the correct url defined in my flask app.

However, the second follow up request made by the client (Postman), as mentioned above, had file: undefined in the multipart form. This resulted in a parsing error being thrown by Flask, but in was caught and silenced due to the following in Flasks formparser.py file:

class FormDataParser:
    """This class implements parsing of form data for Werkzeug.  By itself
    it can parse multipart and url encoded form data.  It can be subclassed
    and extended but for most mimetypes it is a better idea to use the
    untouched stream and expose it as separate attributes on a request
    object.
    ...

    :param silent: If set to False parsing errors will not be caught.
    """

    def __init__(
        self,
        ...
        silent: bool = True,
    ) -> None:

This was silencing the following error which i only saw after changing this silent param to False:

[2022-10-30 17:40:54,736] ERROR in app: Exception on /api/fileservice/0.1/files/upload_file/ [POST]
Traceback (most recent call last):
  File "/home/sean/.pyenv/versions/3.10.5/envs/musicApp-api-3.10.5/lib/python3.10/site-packages/flask/app.py", line 2463, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/sean/.pyenv/versions/3.10.5/envs/musicApp-api-3.10.5/lib/python3.10/site-packages/flask/app.py", line 1760, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/sean/.pyenv/versions/3.10.5/envs/musicApp-api-3.10.5/lib/python3.10/site-packages/flask_cors/extension.py", line 165, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/home/sean/.pyenv/versions/3.10.5/envs/musicApp-api-3.10.5/lib/python3.10/site-packages/flask/app.py", line 1758, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/sean/.pyenv/versions/3.10.5/envs/musicApp-api-3.10.5/lib/python3.10/site-packages/flask/app.py", line 1734, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/home/sean/DevProjects/MusicApp/api/api/rest/file_service_api.py", line 16, in upload_file
    form_data = flask.request.form
  File "/home/sean/.pyenv/versions/3.10.5/envs/musicApp-api-3.10.5/lib/python3.10/site-packages/werkzeug/utils.py", line 109, in __get__
    value = self.fget(obj)  # type: ignore
  File "/home/sean/.pyenv/versions/3.10.5/envs/musicApp-api-3.10.5/lib/python3.10/site-packages/werkzeug/wrappers/request.py", line 426, in form
    self._load_form_data()
  File "/home/sean/.pyenv/versions/3.10.5/envs/musicApp-api-3.10.5/lib/python3.10/site-packages/flask/wrappers.py", line 112, in _load_form_data
    super()._load_form_data()
  File "/home/sean/.pyenv/versions/3.10.5/envs/musicApp-api-3.10.5/lib/python3.10/site-packages/werkzeug/wrappers/request.py", line 264, in _load_form_data
    data = parser.parse(
  File "/home/sean/.pyenv/versions/3.10.5/envs/musicApp-api-3.10.5/lib/python3.10/site-packages/werkzeug/formparser.py", line 255, in parse
    return parse_func(self, stream, mimetype, content_length, options)
  File "/home/sean/.pyenv/versions/3.10.5/envs/musicApp-api-3.10.5/lib/python3.10/site-packages/werkzeug/formparser.py", line 132, in wrapper
    return f(self, stream, *args, **kwargs)
  File "/home/sean/.pyenv/versions/3.10.5/envs/musicApp-api-3.10.5/lib/python3.10/site-packages/werkzeug/formparser.py", line 282, in _parse_multipart
    form, files = parser.parse(stream, boundary, content_length)
  File "/home/sean/.pyenv/versions/3.10.5/envs/musicApp-api-3.10.5/lib/python3.10/site-packages/werkzeug/formparser.py", line 412, in parse
    event = parser.next_event()
  File "/home/sean/.pyenv/versions/3.10.5/envs/musicApp-api-3.10.5/lib/python3.10/site-packages/werkzeug/sansio/multipart.py", line 232, in next_event
    raise ValueError(f"Invalid form-data cannot parse beyond {self.state}")
ValueError: Invalid form-data cannot parse beyond State.PREAMBLE

So setting the original request to use the complete url with the trailing backslash prevented this second request being made where the file was undefined, preventing this error. The file now uploads successfully

Sean
  • 587
  • 4
  • 20