0

Looking through the Java "low level" API to access the Google AppEngine DataStore, it seems to do a lot of heavy lifting and tons of type-checking, parsing, flattening, object creation, etc. which my App doesn't really require (it's wayyyyy too general for my needs).

Therefore, I'd like to get "even lower-level" access to the underlying functions (e.g. by calling com.google.apphosting.api.ApiProxy.makeSyncCall(...) directly) to skip a lot of the (for me) unnecessary overhead, and (who knows) maybe even discover some additional performance-features in the process that aren't accessible otherwise.

For this, I would need to use the classes inside com.google.storage.onestore.v3 directly to build the raw binary request packets and interpret the reply packets.

Of course, it's possible to examine the way things work (roughly) by looking through the source code of the classes inside com.google.appengine.api.datastore, which use these classes, but things would be so much easier if I could also get a hold of the JavaDocs for the classes inside com.google.storage.onestore.v3. Unfortunately I can't find them anywhere (the source-code isn't available either). Has anyone here maybe come across them somewhere?

Markus A.
  • 12,349
  • 8
  • 52
  • 116
  • Nobody answered this a few minutes ago? – Kevin Bowersox Aug 14 '13 at 23:51
  • Remember much of that machinery is used to marshall data for RPC to the datastore service, and validating what is sent and received. These are sure to be Ali's that change with time, do really want to write to a potentially unstable api which won't be tracked for you from release to release. – Tim Hoffman Aug 14 '13 at 23:55
  • @TimHoffman Since most of the API code is uploaded along with my app when I do a deploy (specifically everything in `com.google.appengine.api.datastore`), it probably won't make much difference in reliably. If Google were to change the interface provided by `com.google.storage.onestore.v3` in a non-backwards-compatible way today, all apps would break and have to be re-deployed along with the updated APIs also, so I doubt this will ever happen. Also, the server-side part of my app is super-dirt-simple. That's why I'm a bit bothered by all the overhead I incur from the provided API... – Markus A. Aug 15 '13 at 00:00
  • @TimHoffman (The marshaling happens inside classes in `com.google.storage.onestore.v3`, which is why I'd like to use it. I'm not planning to go any lower than that and build my own binary requests...) – Markus A. Aug 15 '13 at 00:03
  • @MarkusA. Can you show us the benchmarks you've used to determine that the extra overhead of the current API is a bottleneck for you? – Jason C Aug 15 '13 at 00:05
  • @JasonC Clearly the bottleneck are going to be the RPC calls to the DataStore and the network latency, but should that stop me from wanting to shave off some ms from the user-experienced request time and the server-load, which I end up paying for, where I can? I started coding during the time of the 386 processor. Back then people had to worry about every clock cycle, which lead to much better software performance. And, for myself, I'd like to value that approach. As an analogy: Even on a 6 hour road-trip I don't like waiting for 3 minutes at a non-traffic-triggered red light for no reason... – Markus A. Aug 15 '13 at 00:15
  • I wouldn't even go that low myself. But I use python and it's low level api presents dictionaries, which I have used from time to time. Java deployment is different to python, none of the appengine code is deployed with python apps. Are you certain that all of the java code is deployed with your app and not stripped etc? New versions of appengine are deployed in appengine (we see the new version in the console) often ages before we see the SDK ? If what you said was true none of those rolling upgrades could occurr – Tim Hoffman Aug 15 '13 at 00:15
  • @MarkusA. Honestly, yes, it should stop you. My origins are the same as yours, and even now I have to worry about that frequently on embedded systems work. However, the premature optimization is unnecessary. Consider the system you are actually developing for, not the one you are used to developing for. Your time now would be much better spent becoming familiar with usages of the documented API. Your time later will be much better spent by optimizing real bottlenecks that you may discover. It is not worth the risk of crippling future maintenance by using undocumented, unsupported APIs. :) – Jason C Aug 15 '13 at 00:17
  • @TimHoffman Some of the Dashboard features depend on the underlying deployed API and others don't. For example in the recent release notes (https://code.google.com/p/googleappengine/wiki/SdkForJavaReleaseNotes) you can find points like "Admin console dashboard charts and reports for all users have been fully migrated to the new, more reliable backend announced in 1.7.6.". I'm not exactly sure how the deploy works in Python, but the Java deploy (as far as I know) definitely includes the used SDK (e.g. 1.8.2). Otherwise, Google couldn't ever make any changes to it without breaking apps. – Markus A. Aug 15 '13 at 00:23
  • @JasonC While I completely see your point, I have 3 issues with it in *this* case: 1. I actually have no idea what underlying hardware I am developing for, though it is probably safe to assume it's fairly fast. 2. I'm (indirectly) paying for each clock cycle, so every bit might count, especially since it's hard for me to judge how large the API overhead truly is. 3. Even though RPC and network might be the overall bottleneck, they (let's say) "run in a different thread" than my API code and so as far as using up server instance capacity goes, I might be able to make a big difference here. – Markus A. Aug 15 '13 at 03:12
  • @MarkusA. Well, here's hoping you meet your deadlines and don't get bit by the next AppEngine update! – Jason C Aug 15 '13 at 04:18
  • YOu should really build a prototype using the published api's then profile and see what/where you really see bottle necks. Ultimately you can't do anything about round trip times to the services you consume, so without profiling it you are probably spinning your wheels looking for things to optimize that may have no impact whatsoever in the overall performance. – Tim Hoffman Aug 15 '13 at 06:50

0 Answers0