Uploading large video file to Google App Engine

Question

I am trying to upload a large video into Google Cloud Storage via Google App Engine.

I followed the principles in this post which refers to uploading a large spreadsheet. Can't upload large files to Python + Flask in GCP App Engine

I have set up chunking using dropzone.js

I have set up an upload in main.py which I had hoped would upload the file chunks into the app's tmp directory and have put in logic which moves the completed file into Google Cloud Storage once all parts are in place.

I get the following error message:

TypeError: write() argument must be str, not bytes

Here is my back end code

from flask import Flask, render_template, request, redirect, url_for
from google.cloud import storage
from flask_dropzone import Dropzone
from werkzeug.utils import secure_filename
import os
import base64


app = Flask(__name__, template_folder='./templates', static_folder="./static")
dropzone = Dropzone(app)
app.config['UPLOAD_PATH'] = '/tmp'

@app.route('/', methods=['GET', 'POST'])

def index():
    return render_template('index.html')


@app.route('/upload', methods=['POST', 'GET'])
def upload():

    if request.method == 'POST':

    upload_file = request.files.get('file')

    tmp_file_path = '/tmp/' + upload_file.filename

    with open(tmp_file_path, 'a') as f:
        f.write(uploaded_file.read())

    chunk_index = int(flask.request.form.get('dzchunkindex')) if 
(flask.request.form.get('dzchunkindex') is not None)  else 0

    chunk_count = int(flask.request.form.get('dztotalchunkcount'))      if (flask.request.form.get('dztotalchunkcount') is not None)  else 1


    if (chunk_index == (chunk_count - 1)):
        print('Saving file to storage')
        print( chunk_count )

        storage_client = storage.Client()

        storage_bucket = storage_client.get_bucket('percy-277618.appspot.com')

        blob = storage_bucket.blob(upload_file.filename)

        blob.upload_from_filename(tmp_file_path, client=storage_client)
        print('Saved to Storage')

        print('Deleting temp file')
        os.remove(tmp_file_path)

              

if __name__ == '__main__':

    app.run(host='127.0.0.1', port=8080, debug=True)

This is my front end code.

<form id="uploadwidget" method="post" action="/upload" class="dropzone"></form>

<script type="text/javascript">

Dropzone.options.uploadwidget = {

       paramName: 'file',
       forceChunking: true,
       timeout: 300000,
       chunking: true,
       url: '/upload',
       chunkSize: 10485760,
       maxFilesize: 1025,

};

</script>

I found some StackOverflow links [1](https://stackoverflow.com/questions/50542683/typeerror-write-argument-must-be-str-not-bytes-while-saving-npy-file), [2](https://stackoverflow.com/questions/21689365/python-3-typeerror-must-be-str-not-bytes-with-sys-stdout-write) that explain the cause of this error and might help you. The suggestions that are included is to open files in binary mode (include 'b' in mode argument) or convert bytes to str. I hope this helps and your error gets eliminated. — Artemis Georgakopoulou, Aug 05 '20 at 12:57

Dustin Ingram · Accepted Answer · 2020-08-05T19:42:07.703

2

Using uploaded_file.read() produces bytes, not a string. You should open the file in binary mode:

with open(tmp_file_path, 'ab') as f:

edited Aug 05 '20 at 19:42

answered Aug 05 '20 at 15:28

Dustin Ingram

20,502
7
59
82

Thanks Dustin. When I try this, I get the following error in app engine UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8c in position 5: invalid start byte . As it is a video that is being uploaded, would it be utf-8 that would be the decode setting and not another setting. Does utf-8 not only apply to text? – stephen251 Aug 05 '20 at 19:08
Oops, read the question too fast and missed that it was video. Updated my answer! – Dustin Ingram Aug 05 '20 at 19:42
Thanks Dustin, that works. I had to alter the yaml file to instance_class: F4_1G to give it enough processing power for larger video files. – stephen251 Aug 06 '20 at 08:09

Uploading large video file to Google App Engine

1 Answers1