1MB quota limit for a blobstore object in Google App Engine?

Question

I'm using App Engine (version 1.4.3) direct write the blobstore in order to save images. when I try to store an image which is larger than 1MB I get the following Exception

com.google.apphosting.api.ApiProxy$RequestTooLargeException: The request to API call datastore_v3.Put() was too large.

I thought that the limit for each object is 2GB

Here is the Java code that stores the image

private void putInBlobStore(final String mimeType, final byte[] data) throws IOException {
    final FileService fileService = FileServiceFactory.getFileService();
    final AppEngineFile file = fileService.createNewBlobFile(mimeType);
    final FileWriteChannel writeChannel = fileService.openWriteChannel(file, true);
    writeChannel.write(ByteBuffer.wrap(data));
    writeChannel.closeFinally();
}

looks like splitting the data into smaller chunks made the trick. I still got the exception when I tried to store large dataStore record (which has hard limit of 1MB). since the exception stack trace was in a different thread, I thought it is the blobStore that make the problems. Google: you owe me several hours of debugging — Yonatan Maman, Apr 02 '11 at 18:41
If you'd included the stacktrace (or looked at it closely), we could've helped. — Nick Johnson, Apr 04 '11 at 00:55
*UPDATE* the above code seems to work for me. It seems there is no longer a 1 mb limit... — itgiawa, Aug 17 '12 at 05:14

matt burns · Answer 1 · 2012-11-22T18:20:48.380

Here is how I read and write large files:

public byte[] readImageData(BlobKey blobKey, long blobSize) {
    BlobstoreService blobStoreService = BlobstoreServiceFactory
            .getBlobstoreService();
    byte[] allTheBytes = new byte[0];
    long amountLeftToRead = blobSize;
    long startIndex = 0;
    while (amountLeftToRead > 0) {
        long amountToReadNow = Math.min(
                BlobstoreService.MAX_BLOB_FETCH_SIZE - 1, amountLeftToRead);

        byte[] chunkOfBytes = blobStoreService.fetchData(blobKey,
                startIndex, startIndex + amountToReadNow - 1);

        allTheBytes = ArrayUtils.addAll(allTheBytes, chunkOfBytes);

        amountLeftToRead -= amountToReadNow;
        startIndex += amountToReadNow;
    }

    return allTheBytes;
}

public BlobKey writeImageData(byte[] bytes) throws IOException {
    FileService fileService = FileServiceFactory.getFileService();

    AppEngineFile file = fileService.createNewBlobFile("image/jpeg");
    boolean lock = true;
    FileWriteChannel writeChannel = fileService
            .openWriteChannel(file, lock);

    writeChannel.write(ByteBuffer.wrap(bytes));
    writeChannel.closeFinally();

    return fileService.getBlobKey(file);
}

score 3 · Accepted Answer · answered Apr 02 '11 at 13:52

3

The maximum object size is 2 GB but each API call can only handle a maximum of 1 MB. At least for reading, but I assume it may be the same for writing. So you might try to split your writing of the object into 1 MB chunks and see if that helps.

answered Apr 02 '11 at 13:52

Brummo

1,030
2
12
18

I tried to split the writing to several writeChannel.write invocations, but got same results – Yonatan Maman Apr 02 '11 at 17:07
What does it mean "each API call can only handle a maximum of 1 MB" ? which API ? does it mean 1MB per (my app) request ? – Yonatan Maman Apr 02 '11 at 17:13
No, I would guess that it means more or less each function call, not the web request triggering you're code (otherwise there would really be no way to handle more than 1 MB). – Brummo Apr 05 '11 at 21:23

score 3 · Answer 3 · answered Jul 16 '11 at 10:35

As Brummo suggested above if you split it into chunks < 1MB it works. Here's some code.

public BlobKey putInBlobStoreString(String fileName, String contentType, byte[] filebytes) throws IOException {
    // Get a file service
    FileService fileService = FileServiceFactory.getFileService();
    AppEngineFile file = fileService.createNewBlobFile(contentType, fileName);
    // Open a channel to write to it
    boolean lock = true;
    FileWriteChannel writeChannel = null;
    writeChannel = fileService.openWriteChannel(file, lock);
    // lets buffer the bitch
    BufferedInputStream in = new BufferedInputStream(new ByteArrayInputStream(filebytes));
    byte[] buffer = new byte[524288]; // 0.5 MB buffers
    int read;
    while( (read = in.read(buffer)) > 0 ){ //-1 means EndOfStream
        ByteBuffer bb = ByteBuffer.wrap(buffer);
        writeChannel.write(bb);
    }
    writeChannel.closeFinally();
    return fileService.getBlobKey(file);
}

There is a constant for the max blob fetch size in BlobstoreService.MAX_BLOB_FETCH_SIZE = 1015808. I tested this in a local unit test and it works for both reading and writing. — bigspotteddog, Sep 29 '11 at 17:37

1MB quota limit for a blobstore object in Google App Engine?

3 Answers3

Linked