Stack Overflow
Stack Overflow

Questions tagged [bulkloader]

167 questions
4
votes
3 answers

App engine bulk loader download warning "No descending index on __key__, performing serial download"

I'm using the following to download all instances of one of my kinds: appcfg.py download_data --config_file=bulkloader.yaml --kind=ModelName --filename=ModelName.csv --url=http://appid.appspot.com/remote_api If the kind has more instances than the…
Saxon Druce
  • 17,406
  • 5
  • 50
  • 71
4
votes
2 answers

How to load data from the online GAE datastore into the local development server?

I have previously used the approach described in GAE docs to download backups of my entities on the live datastore. Currently, I have a csv file per entity kind, that I got by writing bulkloader.yaml and using this command: appcfg.py download_data…
Mohamed Khamis
  • 7,731
  • 10
  • 38
  • 58
3
votes
1 answer

Google AppEngine Bulkloader High Replication Data Delay

In the AppEngine Bulkloader Documentation, they state "If you attempt to copy from a High Replication datastore, you'll see a high_replication_warning error in the Admin Console, and the downloaded data might not include recently saved entities."…
CorrosiveKid
  • 33
  • 3
3
votes
1 answer

How can I speed up the App Engine bulk downloader?

I'm trying to use the App Engine bulkloader to download entities from the datastore (the high-replication one if it matters). It works, but it's quite slow (85KB/s). Are there some magical set of parameters I can pass it to make it faster? I'm…
DurhamG
  • 222
  • 2
  • 9
3
votes
2 answers

Which credentials should I put in for Google App Engine BulkLoader at JAVA development server?

I can't upload data to my GAE Java dev serwer. Bulkloader asks for a password, but no password matches: D:\python_google_appengine>appcfg.py upload_data --config_file=bulkloader.yaml --filename=templates.csv…
zacheusz
  • 8,750
  • 3
  • 36
  • 60
3
votes
1 answer

How to upload data with key_name by Google App Engine bulkloader

I can upload data, but key_name is empty. How can I use the 'id' in the CSV as the key_name on datastore? I'like to use 'id' as the key_name, because other data uses the 'id' as the foreign key. I'm new to Google App Engine. This is the CSV…
rawlow
  • 31
  • 1
3
votes
3 answers

Uploading data with bulkloader

In short: how can I configure bulkloader to insert data into 2 models with references? I have a person and fruit class, with person linking to fruit: class Fruit(db.Model): name = db.StringProperty() class Person(db.Model): name =…
hoju
  • 28,392
  • 37
  • 134
  • 178
3
votes
1 answer

(Google App Engine) Bulkloader upload generates new entries instead of updating existing ones

for my project I am using GAE datastore to store data. For backup purpose I decided to use the bulkloader - which downloads the whole data perfectly in a csv file. Also the upload uploads the data fine without errors. My problem is, that the…
Adam
  • 31
  • 1
3
votes
1 answer

Errors during downloading data from Google App Engine using bulkloader

I am trying to download some data from the datastore using the following command: appcfg.py download_data --config_file=bulkloader.yaml --application=myappname --kind=mykindname --filename=myappname_mykindname.csv …
abhgh
  • 308
  • 1
  • 9
3
votes
2 answers

How can I use bulkuploader to populate class with a db.SelfReferenceProperty?

I've got a class that is using db.SelfReferenceProperty to create a tree-like structure. When trying to populate the database using appcfg.py upload_data -- config_file=bulkloader.yaml --kind=Group --filename=group.csv (...) , I'm getting an…
Esteban Küber
  • 36,388
  • 15
  • 79
  • 97
3
votes
1 answer

PySpark: Can saveAsNewAPIHadoopDataset() be used as bulk loading to HBase?

We currently import data to HBase tables via Spark RDDs (pyspark) by using saveAsNewAPIHadoopDataset(). Is this function using the HBase bulk loading feature via mapreduce? In other words, would saveAsNewAPIHadoopDataset(), which imports directly…
kentt
  • 543
  • 5
  • 12
3
votes
2 answers

Upload Bulk Image in MultipuleThread android

right now i am uploading image in single thread using asynctask. but it is not efficient and it's takes lot of time to upload. so i am refering this link to download image using ThreadPoolExecuter. can i use this strategy while uploading image also…
mcd
  • 1,434
  • 15
  • 31
3
votes
2 answers

Importing a dict to an App Engine ndb.JsonProperty with bulkloader

I'm using the bulkloader to upload data into my App Engine data storage. I can't seem to store a dictionary into a JsonProperty and I am getting the following error: BadValueError: Unsupported type for property nearby_countries: My…
Erwan
  • 3,733
  • 30
  • 25
3
votes
1 answer

How to upload a repeated property into ndb datastore with appcfg.py

I've been using appcfg.py to upload_data pretty successfully, but I'm not sure how to set up the import transform in bulkloader.yaml for repeated properties or how to structure the CSV. For example: In a post model that looks like this: class…
mehulkar
  • 4,895
  • 5
  • 35
  • 55
2
votes
2 answers

In Google App Engine, how do I configure bulkloader.yaml to handle an optional foreign key on a model?

I have a model in App Engine that I am exporting in CSV format and then importing to my local dev environment using appcfg.py and bulkloader.yaml. I can import and export most models, but I am having issues with models that have a foreign key which…
Chris
  • 4,237
  • 6
  • 30
  • 42
1
2
3
…
11 12