23

I want to make client version of GAE app that store exact data of online version.(myapp.appspot.com) If i can use sdk instead, is any library or tools to sync online and sdk version? I try using bulkloader but i can't load downloaded data to local SDK? Please help.

Dan McGrath
  • 41,220
  • 11
  • 99
  • 130
Ivan Slaughter
  • 767
  • 1
  • 8
  • 26

2 Answers2

29

As explained in this article (link updated, thanks to Zied Hamdi)

You simply need to enable the remote api

builtins:
- remote_api: on

Update your application then run the following commands:

appcfg.py download_data -A s~YOUR_APP_NAME --url=http://YOUR_APP_NAME.appspot.com/_ah/remote_api/ --filename=data.csv
appcfg.py --url=http://localhost:8080/_ah/remote_api/ --filename=data.csv upload_data .

Edit for After April 12th 2016 on Latest AppEngine SDK:

The above works for SDK version 1.9.0 and before. However with the depreciation of ClientLogin, the above will cause an error of 

03:13 PM Uploading data records.
[INFO    ] Logging to bulkloader-log-20160909.151355
[INFO    ] Throttling transfers:
[INFO    ] Bandwidth: 250000 bytes/second
[INFO    ] HTTP connections: 8/second
[INFO    ] Entities inserted/fetched/modified: 20/second
[INFO    ] Batch Size: 10
[INFO    ] Opening database: bulkloader-progress-20160909.151355.sql3
2016-09-09 15:13:55,175 INFO client.py:578 Refreshing due to a 401 (attempt 1/2)
2016-09-09 15:13:55,176 INFO client.py:804 Refreshing access_token
2016-09-09 15:13:55,312 INFO client.py:578 Refreshing due to a 401 (attempt 2/2)

Recommended by Anssi here, we can use the API server directly without running into this error. For a typical dev_appserver startup you get the following output 

INFO     2016-09-09 19:27:11,662 sdk_update_checker.py:229] Checking for updates to the SDK.
INFO     2016-09-09 19:27:11,899 api_server.py:205] Starting API server at: http://localhost:52497
INFO     2016-09-09 19:27:11,905 dispatcher.py:197] Starting module "default" running at: http://localhost:8080
INFO     2016-09-09 19:27:11,918 admin_server.py:116] Starting admin server at: http://localhost:8000

instead of the above for upload use the API port, in this case 

appcfg.py --url=http://localhost:52497/_ah/remote_api/ --filename=data.csv upload_data .
StanleyZheng
  • 4,008
  • 3
  • 21
  • 24
Olivier.Roger
  • 4,241
  • 5
  • 40
  • 68
  • 5
    To get this to work locally for me I also needed to add the -A dev~YOUR_APP_NAME to the upload_data options. – ahanson Apr 02 '14 at 16:31
  • 2
    I got the following error when doing the upload_data step: 'google.appengine.api.datastore_errors.BadRequestError: app "dev~my_app_name" cannot access app "my_app_name"'s data'. I solved this by removing the '.' at the end of the line. So I wrote: appcfg.py --url=http://localhost:8080/_ah/remote_api/ --filename=data.csv upload_data – Martin Sherburn Aug 29 '14 at 10:26
  • 1
    Is this only for python apps? – morpheus05 Sep 22 '14 at 08:39
  • 1
    Version SDK 1.9.0 > has deprecated ClientLogin as of April, 12, 2016 which leads users using the solution above to get an access token error of `Refreshing due to a 401 (attempt 2/2)` . There are two solutions, using an older SDK version of 1.9.0 or before or use detailed following use the API port and not the admin server port ie -- appcfg.py `url=http://localhost:49658/_ah/remote_api/` instead https://code.google.com/p/googleappengine/issues/detail?id=12445 – StanleyZheng Sep 09 '16 at 19:25
15

See the docs for details on how to download and upload your entire datastore. Simply bulk download from production, then bulk upload to your local datastore.

Bear in mind, however, that the local datastore is not designed to handle large volumes of data - you may run into performance or memory issues.

Nick Johnson
  • 100,655
  • 16
  • 128
  • 198
  • 1
    Thanks Nick, where i can get more clear "how to" of doing bulk upload to local datastore? Really need to make this work for me. – Ivan Slaughter Apr 21 '10 at 19:59
  • 1
    See the section 'loading data into the development server' on that page. You simply specify an additional command line argument. – Nick Johnson Apr 22 '10 at 09:06
  • Anyway, Nick.. i've installed 1.3.3 sdk and experiencing some error of bulkload about 6000 records, that stop on record 145. And exception throw when i want to use datastore viewer on local SDK console. What's wrong? – Ivan Slaughter Apr 22 '10 at 19:33
  • I can't possibly tell unless you tell me what the error is, and what the stacktrace is. – Nick Johnson Apr 22 '10 at 20:48