1

The argument has been made and settled on numerous occasions: Blobstore is better than DataStore for storing images. Now say I have an app similar to Instagram or Facebook or Yelp or any of those apps that are image intensive. In my particular case the ideal model would be

public class IdealPostModel{

  Integer userId;
  String synopsys;
  Blob image;
  ...//more data/fields about the post

}

But since I must use the BlobStore, my model does not have a blob but instead a URL or BlobKey. The heavy catch is that to send a post (i.e. save a post to server) the app must -- in that order --

  1. Send all the non-blob data to the server.
  2. Wait for server to respond with BlobstoreUtils.generateServingUrl(null) data
  3. Send the images to the BlobStore
  4. Have BlobStore send response to my server with the BlobKey or url of the image
  5. Store the BlobKey/url in my DataStore entity

That's a lot of handshakes!!!

Is there a way to send all the data to the server in step one: strings and image. And then from there have the server do everything else? Of course I am here hoping to reduce the amount of work. I image App Engine must be quite mature by now and there has to be a simpler way than my architecture.

Now of course I am here because I am experiencing situations where the data is saved but the BlobKey or URL is not being saved to the Entity. It happens about 10% of the time. Or maybe less, but it does feel like 10%. It's driving my users insane, which means it's driving me even more insane since I don't want to lose my users.

Ideally

  1. App sends everything to server in one call: image and metadata such as userId and synopsys
  2. Server somehow gets a blob key from Blobstore
  3. Server sends image to blobstore, to be stored at the provided blob key, and server sends other data to DataStore, also including the blob key in the datastore.

Update

    public static String generateServingUrl(String path) {
        BlobstoreService blobstoreService = BlobstoreServiceFactory.getBlobstoreService();
        return blobstoreService.createUploadUrl(null == path ? "/upload" : path);
    }

This is a snippet on my server.

Dan McGrath
  • 41,220
  • 11
  • 99
  • 130
Katedral Pillon
  • 14,534
  • 25
  • 99
  • 199

2 Answers2

3

The workflow is different. Given your example:

  1. There is "Post" button and "Attach an image" button under a form for a new post.

  2. When a user hits the Attach button, you ask a user to select a file and save it to the BlobStore (or Google Cloud Storage). There is no need to call BlobstoreUtils.generateServingUrl(null) - you need to get an upload URL. When a call returns a key, you store it in the Post object.

  3. When a user hits the Post button, you save the Post - including the image key (keys) in the Datastore. Thus, the entire workflow takes only two calls.

  4. If a user hits the Cancel button - remember to delete the uploaded image(s) if any, or they will be orphaned. The tricky part is to delete these images if a user simply left your app or lost connection.

If you want, you can reverse the process - save post first, then let users attach images. This is not what most users expect, but it solves the problem with orphaned images.

Andrei Volgin
  • 40,755
  • 6
  • 49
  • 58
  • 1
    Do you have an example of how to send the image to BlobStore from android or iOS and also receive the url as the response? Thanks. – Katedral Pillon Mar 24 '15 at 01:09
  • You say that `The entire workflow takes only two calls`. That seems to suggest I can get the uploadUrl right from android. Is that correct? Right now I am doing it from the server as : `blobstoreService.createUploadUrl(string)`. – Katedral Pillon Mar 24 '15 at 01:27
  • 1
    Will you clarify for me if the following are the steps in your response? 1 `app => server: get upload url`; 2 `app => BlobStore : save my blob`; 3 `BlobStore => server : here is the BlobKey`; 4 `server => app : here is the BlobKey`; 5 `app => server : here is post data associated with blobKey`. – Katedral Pillon Mar 24 '15 at 02:04
  • You can get an upload URL immediately when a form is opened and before a user decides to do something. This upload URL is valid for 10 minutes, if I remember correctly. This way the app will seem more responsive to a user as you eliminate one step from the process after a file is submitted. – Andrei Volgin Mar 24 '15 at 02:10
  • The steps in your comment are correct, except you need to add step 1a (server returns upload url) and step 6 - server responds to the post call. Technically, these are 3 round-trips between the app and the server, but the first round trip can be done before any user action if necessary. – Andrei Volgin Mar 24 '15 at 02:14
  • In that case step 4 is quite problematic. How does the server know which app instance to send the BlobKey to? Say I have 100 users (low number of purpose), how do I know which user to send the key too? do I require the waiting user to poll continuously? What if I have three waiting users, how does each know which key belong to it? My guess is you have probably done this and therefore I am missing something. – Katedral Pillon Mar 24 '15 at 04:27
  • When you request an upload URL, you need to provide a callback url. The Blobstore rewrites your request - it hits that callback url and then takes its response and delivers it to your app. So from your app's point of view, it submitted a form to the upload url, and the response arrives when upload is completed. It's up to you what to include in that response - I include the blob key and the file size. – Andrei Volgin Mar 24 '15 at 05:03
  • I am reading it, but I am kind of lost. I understand the call back from the BlobStore goes to my server, not to my android app. That's where I am lost. Your sentence `it hits that callback url and then takes its response and delivers it to your app` does not add up to me because I don't see how to do that. It would be great if you could share a code snippet. Thanks. – Katedral Pillon Mar 24 '15 at 05:20
  • 1
    I think at this point a code snippet is in order. I too am having a hard time following this. And I am just a curious bystander. – Konsol Labapen Mar 24 '15 at 05:49
2

You actually cand send everything in a single request.

Bear in mind that when you send a blob using a URL gotten by calling blobstoreService.createUploadUrl("/yourservingurl") , GAE actually makes a first request to a different URL, stores the blob and then calls /yourservingurl passing you the blobkeys which you can retrive by doing:

Map<String, List<BlobKey>> blobs = blobstoreService.getUploads(req);
List<BlobKey> blobKeys = blobs.get("myFile");

So in effect all other form values will be lost after that first request, BUT if you can build that URL dynamically eg 

blobstoreService.createUploadUrl("/yourservingurl?param1=a&param2=b")

then you can get back those parameters on your Servlet and persist everything (including the blobkey of the already stored blob) at once, making much fewer calls to the datastore.

UPDATE: the steps would be

1) Gather all parameters on the client side and create an upload URL with those params eg: blobstoreService.createUploadUrl("/yourservingurl?userId=989787") . GAE will generate a unique URL for that specific request.

when you commit the form, GAE will persist the blobs and call /yourservingurl

2) On the Servlet serving /yourservingurl you'll get all the blobkeys form files you uploaded by using : blobstoreService.getUploads(request) AND you can get the parameters you included with the standard request.getParameter("userId")

3) Now in your servlet you have all the parameters (eg userId) you sent plus the blobkey, you can persist your Post object in one call.

jirungaray
  • 1,674
  • 1
  • 11
  • 18
  • Thanks for the response but it is too condensed for me to understand. Do you mind elaborate on the steps in the manner that I have been doing? So you are saying: step 1: `app => server: here are post and image`; step 2:`server => BlobStore: save image with upleadUrl`; step 3:`server => datastore: save post entity`;step 4: `blobStore => server (callback): here is BlobKey`; step 5: `server => datastore: add blobKey to post entity`. Is this what you had in mind? – Katedral Pillon Mar 24 '15 at 15:50
  • 1
    Thanks I was missing that. +1. This helps a lot. But I still have two outstanding questions. 1) with this system android (i.e. the client) does not know immediately when the data has been persisted since there is no response to it: the call back instead goes to my server, correct? 2) since that get call belongs to me, can it be a long as I want? Because the metadata includes a comment section which can be a lot of characters. Thanks. – Katedral Pillon Mar 24 '15 at 16:13
  • 1) you do get a response to your client form you servlet/endpoint, but NOT from the blob upload itself, this is done by GAE and is completely transparent. you'll have to build the response on the servlet working on **/yourservingurl** 2)Unfortunately this is a hack and you'll be restricted to the maximum URL length (about 2000 chars, depending on the browser). If you plan to send longer text you will have to make a second call to send that text. – jirungaray Mar 24 '15 at 16:21
  • In other words do you know if blobstore limits the number of characters in an upload url? – Katedral Pillon Mar 24 '15 at 16:21
  • not on the blobstore side, the limitation comes form the HTTP protocol implementation itself. GAE has limit of 2048 chars, but there's a [request to extend that] (https://code.google.com/p/googleappengine/issues/detail?id=7053) – jirungaray Mar 24 '15 at 16:27