3

Currently, I am using PyDrive to upload my backup (.tar file) to google drive.

Is there a special thing to do with this library to upload a huge file to Google Drive (around 5gb). In the Google Drive API documentation, it says that we must use the Resumable upload ? https://developers.google.com/drive/web/manage-uploads

My problem is that when I try to send a huge file, the script executes without any errors, really quickly and the file does not appear in GoogleDrive. However, if I do this with a small file around 100mb, everything works perfectly fine...

My code is the following:

def upload(self, backupFile, backupFileName):

    json_data=open(os.path.join(__location__, 'client_secrets.json'))

    data = json.load(json_data)

    """Email of the Service Account"""
    SERVICE_ACCOUNT_EMAIL = data['client_email']

    """Path to the Service Account's Private Key file"""
    SERVICE_ACCOUNT_PKCS12_FILE_PATH = os.path.join(__location__, 'key.p12')

    f = file(SERVICE_ACCOUNT_PKCS12_FILE_PATH, 'rb')
    key = f.read()
    f.close()

    credentials = SignedJwtAssertionCredentials(SERVICE_ACCOUNT_EMAIL, key,
        scope='https://www.googleapis.com/auth/drive', sub='email')
    http = httplib2.Http()
    credentials.authorize(http)

    gauth = GoogleAuth()
    gauth.credentials = credentials

    drive = GoogleDrive(gauth)

    file1 = drive.CreateFile({'title': backupFileName, "parents" : [{"id":"0B7FoN03AUUdZVlNETEtWLS1VTzQ"}]} )  # Create GoogleDriveFile instance with title 'Hello.txt'

    file1.SetContentFile(backupFile);
    file1.Upload()

When I try to send a large file, no errors are returned whatsoever. The python script simply ends without anything being shown...

Etienne Noël
  • 5,988
  • 6
  • 48
  • 75
  • not an answer, but I would avoid sending large files to Drive. Way too many issues. Esp for a tar, I would split(1) it into 100MB chunks and then reassemble on download using cat(1) – pinoyyid Dec 05 '14 at 03:42
  • I think you are right, I'll probably use Amazon Glacier. – Etienne Noël Dec 05 '14 at 11:22

2 Answers2

1

There are 100MB limits on some types of files

https://support.google.com/drive/answer/37603?hl=en

Size limits

Documents: 1,024,000 characters, regardless of the number of pages or font size. If using the new version of Drive (with the red "New" button on the left side), uploaded document files that are converted to the Google documents format can’t be larger than 50 MB. If using the classic version of Drive, you can't convert document files that are larger than 10 MB.

Spreadsheets: 400,000 cells, with a maximum of 256 columns per sheet. Uploaded spreadsheet files that are converted to the Google spreadsheets format can’t be larger than 100 MB, and need to be under 400,000 cells and 256 columns per sheet.

More information about spreadsheet size limits All spreadsheet limits mentioned above have been removed in the new version of Google Sheets. The new version of Google Sheets should support 2 million cells of data, though please note that extremely large spreadsheets may have slower performance. Learn more about switching to the new version of Google Sheets. Presentations: Presentations created in Google Slides can be up to 100 MB. Uploaded presentation files that are converted to Google Slides can also be up to 50 MB.

Drawings: We’ve never seen anyone make a drawing that was too big (but that’s not a dare).

Other files: Files that you upload but don’t convert to a Google Docs, Sheets, or Slides format can be up to 5 TB each.

Across Google Drive, Gmail, and Google+ Photos, every user is given 15 GB of free storage space, and can purchase additional storage as well.

John La Rooy
  • 295,403
  • 53
  • 369
  • 502
  • How is this even an answer? Any time Google APIs return errors they'll throw an exception on pyDrive. This is most certainly a pyDrive bug since it's not even getting to the upload part yet. – Allison Jul 30 '17 at 03:40
0

You can upload any size by passing a file like object to .content of GoogleDriveFile

file1.content=fileobj

from my experience it reads in 8192 bytes at at ime first it seeks 0,2, then seeks 0,0, then calls tell() before every read(8192) and it also calls read(8192) at seek(0,2)+1

so you have to put read(),tell() and seek() methods in the object this is probably from the http.py portion of the api

Edo Edo
  • 164
  • 2
  • 9