0

I use GAE bulk loader to download datastore data - 

appcfg.py download_data --log_file=bulkloader.log --kind=MyKind --application=s~myappid --url=http://myappid.appspot.com/rmt_api --filename=data_downloaded.csv --db_filename=skip --config_file=bulkloader.yaml

But after some time, I am getting OverQuotaError: The API call datastore_v3.RunQuery() required more quota than is available. This is about Datastore Read Operations. Looks like I should download some data on day 1, the something on day 2, day 3 etc.

How can I do it?

Upd. The doc says

If the transfer is interrupted, you can resume the transfer from where it left off using the --db_filename=... and --result_db_filename=... arguments. These arguments are the names of the progress file and the results file created by the tool, which are either names you provided with the arguments when you started the transfer, or default names that include a timestamp. This assumes you have sqlite3 installed, and did not disable progress files with --db_filename=skip.

Does it mean that I can run appcfg.py download_data ... several times passing the same db_filename and result_db_filename values and it will continue to download remaining records each time? What will happen with my CSV file? Will it add records at the end of file?

LA_
  • 19,823
  • 58
  • 172
  • 308
  • How many entities are in that Kind? Do you have billing enabled? When you go to the dev console, click on Quota Details, what does it say? – GAEfan Jun 08 '14 at 19:40
  • Here are the quotas: https://developers.google.com/appengine/docs/quotas#Datastore Not very good, without billing enabled. Enabling billing doesn't mean you will be charged, if you stay within the quotas. – GAEfan Jun 08 '14 at 20:06
  • @GAEfan, currently I have 100,231 entries there. The billing is disabled. I would prefer to keep it disabled, otherwise I will be over free quota and will have to pay for something else. – LA_ Jun 09 '14 at 08:18
  • Daily quotas are replenished daily at midnight Pacific time. What if you started your task 5 minutes before midnight? Assuming a 10 minute job, would it put half the queries into one day's quota, and the other half into the next's? – GAEfan Jun 09 '14 at 14:30

2 Answers2

0

If you are operating within free quota now, enabling billing doesn't change anything, other than fixing this problem. Check the quotas with billing enabled. You will most likely not incur any charges, unless your app sees tremendous traffic.

GAEfan
  • 11,244
  • 2
  • 17
  • 33
  • In my case it will, in some other scenario I catch OverAPIQuota to stop processing and wait for the next day, so I can not enable the billing. Will try the approach with `db_filename` and `result_db_filename` to download everything in a few days. – LA_ Jun 09 '14 at 14:06
0

Usage of db_filename and result_db_filename allows to download data in a few days. Once daily limit is reached, downloading can be stopped and then started again on the next day - it will not download the same data again, but will continue to download remaining items. CSV file will be created once all data is downloaded (i.e. on the last day).

LA_
  • 19,823
  • 58
  • 172
  • 308