Fetching blob from Google Blobstore without knowing size before hand

Question

I need to fetch a blob from the Blobstore programmatically, without knowing the size before hand. Does anyone know how to do that?

I have tried using

BlobstoreService blobStoreService = BlobstoreServiceFactory.getBlobstoreService();
byte[] picture = blobStoreService.fetchData(blobKey, 0, Integer.MAX_VALUE);

but I get an error since (at least seemingly) Integer.MAX_VALUE is too big.

java.lang.IllegalArgumentException: Blob fetch size 2147483648 it larger than maximum size 1015808 bytes.
at com.google.appengine.api.blobstore.BlobstoreServiceImpl.fetchData(BlobstoreServiceImpl.java:250)

So does anyone know how to do this correctly? Also if you could tell me in passing, is it better to same images into the blobstore as "jpeg" or as "png"?

Not that it would seem to matter anyway, but you know that the given limits are inclusive and therefore `Integer.MAX_VALUE` was likely to fail. Not that it matters much here. — Konsol Labapen, Apr 11 '13 at 16:45
An `inclusive` API, unbelievable. `app-engine` is not taken seriously I think. — Kamel, May 22 '15 at 09:28

score 3 · Accepted Answer · answered Apr 11 '13 at 16:59

Hope this helps, this is the way I have been doing it for a while:

        BlobstoreService blobstoreService = BlobstoreServiceFactory.getBlobstoreService();
        BlobKey blobKey = new BlobKey(KEY);

        // Start reading
        ByteArrayOutputStream out = new ByteArrayOutputStream();

        long inxStart = 0;
        long inxEnd = 1024;
        boolean flag = false;

        do {
            try {
                byte[] b = blobstoreService.fetchData(blobKey,inxStart,inxEnd);
                out.write(b);

                if (b.length < 1024)
                    flag = true;

                inxStart = inxEnd + 1;
                inxEnd += 1025;

            } catch (Exception e) {
                flag = true;
            }

        } while (!flag);

        byte[] filebytes = out.toByteArray();

I used to use:

BlobInfo blobInfo = blobInfoFactory.loadBlobInfo(blobKey);
filesize = blobInfo.getSize();

to obtaint the size, but for some reason, sometimes this info was null.

Maybe all this can give you an idea.

You can fetch up to `BlobstoreService.MAX_BLOB_FETCH_SIZE` bytes at a time using `fetchData()`. The constant is defined [in the javadoc](https://developers.google.com/appengine/docs/java/javadoc/com/google/appengine/api/blobstore/BlobstoreService) but its use for `fetchData` is only documented [in the Python doc](https://developers.google.com/appengine/docs/python/blobstore/functions#fetch_data). — Sébastien Trottier, Jun 28 '13 at 18:34

Kamel · Answer 2 · 2015-05-22T08:54:11.920

def blob_fetch(blob_key):
  blob_info = blobstore.get(blob_key)
  total_size = blob_info.size
  unit_size = blobstore.MAX_BLOB_FETCH_SIZE
  pos = 0
  buf = cStringIO.StringIO()
  try:
    while pos < total_size:
      buf.write(blobstore.fetch_data(blob_key, pos, min(pos + unit_size - 1, total_size)))
      pos += unit_size
    return buf.getvalue()
  finally:
    buf.close()

The MAX_BLOB_FETCH_SIZE is not obvious in the document.

score 1 · Answer 3 · answered Jun 28 '13 at 18:44

1

In Python:

from google.appengine.ext.blobstore import BlobInfo
from google.appengine.api import blobstore
import cStringIO as StringIO

blobinfo = BlobInfo.get(KEY)

offset = 0
accumulated_content = StringIO.StringIO()
while True:
  fetched_content = blobstore.fetch_data(
      blobinfo.key(),
      offset,
      offset + blobstore.MAX_BLOB_FETCH_SIZE - 1)
  accumulated_content.write(fetched_content)
  if len(fetched_content) < blobstore.MAX_BLOB_FETCH_SIZE:
    break
  offset += blobstore.MAX_BLOB_FETCH_SIZE

answered Jun 28 '13 at 18:44

Sébastien Trottier

2,226
2
15
11

The `read` method of the [`BlobReader`class](https://developers.google.com/appengine/docs/python/blobstore/blobreaderclass) from `google.appengine.ext.blobstore` of the Python API will do also (instead of the while loop). – Sébastien Trottier Jun 28 '13 at 19:03

Fetching blob from Google Blobstore without knowing size before hand

3 Answers3