uplad file to google drive with progress bar with python requests

Question

This is my code for uploading to google drive with python requests using google-drive-api.

import sys
import json
import requests
from tqdm import tqdm
import requests_toolbelt
from requests.exceptions import JSONDecodeError

class ProgressBar(tqdm):
    def update_to(self, n: int) -> None:
        self.update(n - self.n)

def upload_file(access_token:str, filename:str, filedirectory:str):

    metadata = {
        "title": filename,
    }

    files = {}
    session = requests.session()

    with open(filedirectory, "rb") as fp:
        files["file"] = fp
        files["data"] = ('metadata', json.dumps(metadata), 'application/json')
        encoder = requests_toolbelt.MultipartEncoder(files)
        with ProgressBar(
            total=encoder.len,
            unit="B",
            unit_scale=True,
            unit_divisor=1024,
            miniters=1,
            file=sys.stdout,
        ) as bar:
            monitor = requests_toolbelt.MultipartEncoderMonitor(
                encoder, lambda monitor: bar.update_to(monitor.bytes_read)
            )

            r = session.post(
                "https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart",
                data=monitor,
                allow_redirects=False,
                headers={"Authorization": "Bearer " + access_token},
            )

    try:
        resp = r.json()
        print(resp)
    except JSONDecodeError:
        sys.exit(r.text)

upload_file("access_token", "test.txt", "test.txt")

When i am trying send file with data attribute in post request then file name did not send and with files attribute in post request then requests-toolbelt not working. How to fix this error ?

any reason why you dont want to use the python client libary? — Linda Lawton - DaImTo, Nov 03 '22 at 11:45

Tanaike · Accepted Answer · 2022-11-03T13:20:36.147

When I saw your script, I thought that the content type is not included in the request header. In this case, I think that the request body is directly shown in the uploaded file. I thought that this might be the reason for your current issue. In order to remove this issue, how about the following modification?

From:

r = session.post(
    url,
    data=monitor,
    allow_redirects=False,
    headers={"Authorization": "Bearer " + access_token},
)

To:

r = session.post(
    url,
    data=monitor,
    allow_redirects=False,
    headers={
        "Authorization": "Bearer " + access_token,
        "Content-Type": monitor.content_type,
    },
)

In this case, from metadata = { "title": filename }, it supposes that url is https://www.googleapis.com/upload/drive/v2/files?uploadType=multipart. Please be careful about this.
When you want to use Drive API v3, please modify metadata = { "title": filename } to metadata = { "name": filename }, and use the endpoint of https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart.
- When the file is uploaded with Drive API v3, the value of {'kind': 'drive#file', 'id': '###', 'name': 'test.txt', 'mimeType': 'text/plain'} is returned.
By the way, when an error like badContent occurs in your testing, please try to test the following modification. When in the request body of multipart/form-data the file content is put before the file metadata, it seems that an error occurs. I'm not sure whether this is the current specification. But, I didn't know the order of request body is required to be checked.
- From
```
  files = {}
  files["file"] = fp
  files["data"] = ('metadata', json.dumps(metadata), 'application/json')
```
- To
```
  files = collections.OrderedDict(data=("metadata", json.dumps(metadata), "application/json"), file=fp)
```

Note:

I thought that in your script, an error might occur at file_size = os.path.getsize(filename). Please confirm this again.
When I tested your script by modifying the above modifications, I could confirm that a test file could be uploaded to Google Drive with the expected filename. In this case, I also modified it as follows.
```
  files = collections.OrderedDict(data=("metadata", json.dumps(metadata), "application/json"), file=fp)
```

References:

The change works for me :- set "Content-Type" header and `files = collections.OrderedDict(data=("metadata", json.dumps(metadata), "application/json"), file=fp)` . Thanks for answer. — jak bin, Nov 03 '22 at 13:58
@jak bin About `The change works for me`, welcome. Thank you for letting me know. I'm glad your issue was resolved. If your question was solved, please push an accept button. Other people who have the same issue with you can also base your question as a question that can be solved. And I think that your issue and solution will be useful for them. If you don't find the button, feel free to tell me. https://stackoverflow.com/help/accepted-answer — Tanaike, Nov 03 '22 at 22:34
@jak bin About your new question of `how i can upload file to specific folder on google drive ?`, I would like to support you. So, can you post it as a new question? Because that is different from your question. So can you post it as new question by including the detailed information? Because when your initial question is changed by a comment, other users who see your question are confused. By posting it as new question, users including me can think of it. If you can cooperate to resolve your new issue, I'm glad. Can you cooperate to resolve your new question? — Tanaike, Jul 11 '23 at 04:57
@jak bin Now, I guess that in this case, your new question might be able to be achieved by modifying `metadata` to `metadata = {"title": filename,"parents":[{"id": "###folderId###"]}`. But, unfortunately, I cannot understand your situation, because this question is on Nov 3, 2022. So, if this was not the solution of your current situation, can you post it as a new question? If you can cooperate to resolve your new issue, I'm glad. Can you cooperate to resolve your new question? — Tanaike, Jul 11 '23 at 05:02
i have posted the question [Here](https://stackoverflow.com/questions/76662800/upload-file-to-google-drive-on-specific-folder-with-progress-bar-and-python-requ) — jak bin, Jul 11 '23 at 15:55

Linda Lawton - DaImTo · Answer 2 · 2022-11-03T11:44:26.123

0

Metadata needs to be sent in the post body as json.

Python Requests post() Method

data = Optional. A dictionary, list of tuples, bytes or a file object to send to the specified url
json = Optional. A JSON object to send to the specified url

 metadata = {
    "name": filename,
  }


  r = session.post(
            url,
            json=json.dumps(metadata),
            allow_redirects=False,
            headers={"Authorization": "Bearer " + access_token},
        )

edited Nov 03 '22 at 11:44

answered Nov 03 '22 at 09:39

Linda Lawton - DaImTo

106,405
32
180
449

this did not works. this also did not send file name to server. file name only send with files attribute. – jak bin Nov 03 '22 at 11:19
make sure you are setting the file name. In v3 its name in v2 it was title. – Linda Lawton - DaImTo Nov 03 '22 at 11:44

score 0 · Answer 3 · answered Feb 05 '23 at 16:16

Future readers can find below a complete script that also contains details on how to get access to the bearer token for HTTP authentication.

Most of the credit goes to the OP and answers to the OPs question.

"""
Goal: For one time upload of a large file (as the GDrive UI hangs up)

Step 1 - Create OAuth 2.0 Client ID + Client Secret 
- by following the "Authentication" part of https://pythonhosted.org/PyDrive/quickstart.html

Step 2 - Get Access Token 
- from the OAuth playground -> https://developers.google.com/oauthplayground/ 
    --> Select Drive API v3 -> www.googleapis.com/auth/drive --> Click on "Authorize APIs" 
    --> Click on "Exchange authorization code for tokens" --> "Copy paste the access token" 
    --> Use it in the script below

Step 3 - Run file as daemon process
- nohup python -u upload_gdrive.py > upload_gdrive.log 2>&1 &
- tail -f upload_gdrive.log
"""

import sys
import json
import requests
from tqdm import tqdm
import requests_toolbelt # pip install requests_toolbelt
from requests.exceptions import JSONDecodeError

import collections

class ProgressBar(tqdm):
    def update_to(self, n: int) -> None:
        self.update(n - self.n)

def upload_file(access_token:str, filename:str, filepath:str):

    metadata = {
        "name": filename,
    }

    files = {}
    session = requests.session()

    with open(filepath, "rb") as fp:
        
        files = collections.OrderedDict(data=("metadata", json.dumps(metadata), "application/json"), file=fp)

        encoder = requests_toolbelt.MultipartEncoder(files)
        with ProgressBar(
            total=encoder.len,
            unit="B",
            unit_scale=True,
            unit_divisor=1024,
            miniters=1,
            file=sys.stdout,
        ) as bar:
            monitor = requests_toolbelt.MultipartEncoderMonitor(
                encoder, lambda monitor: bar.update_to(monitor.bytes_read)
            )

            r = session.post(
                "https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart",
                data=monitor,
                allow_redirects=False,
                headers={
                    "Authorization": "Bearer " + access_token
                    , "Content-Type": monitor.content_type
                    },
            )

    try:
        resp = r.json()
        print(resp)
    except JSONDecodeError:
        sys.exit(r.text)

upload_file("<access_token>"
            , "<upload_filename>", "<path_to_file>")

uplad file to google drive with progress bar with python requests

3 Answers3

From:

To:

Note:

References: