Updating data with bulkloader

Question

I'm using python script do upload some data to my app engine backend.

Here is it's definition in bulkloader.yaml

- kind: Subcategory
  connector: csv
  connector_options:
    encoding: utf-8
  property_map:
    - property: __key__
      external_name: id
      export_transform: transform.key_id_or_name_as_string
      import_transform: transform.none_if_empty(int)

    - property: name
      external_name: name

    - property: categoryId
      external_name: categoryId    
      export_transform: transform.key_id_or_name_as_string
      import_transform: transform.none_if_empty(int) 

    - property: language
      external_name: language 

    - property: active
      external_name: active
      import_transform: bool

The problem is, that column active is dynamically changing later, and next time I upload the same data again it's being replaced with false because, column doesn't exist in csv.

I tried removing the column from bulkloader, but then the columns just dissapears. It's probably because entities are replaced and not updated. Is there a way to preserve already existing columns without replacing/deleting them when uploading?

score 1 · Answer 1 · edited May 23 '17 at 12:05

1

Sorry, there is no easy way to merge properties with the bulk loader. As you noticed, it's creating entities based on your CSV data, then storing them with known keys, overwriting existing entities. To accomplish a merge, you'll need to bulk download the existing entities, perform the merge in your dataset, then load the merged entities.

There are hooks for doing fancy things during the loader logic which might help implement a merge tool. See this related SO question: Merge multiple columns in bulkloader But you'd still need to fetch the properties to merge from the datastore somehow. The datastore can't perform the merge on its own.

edited May 23 '17 at 12:05

Community

1
1

answered Sep 14 '14 at 02:10

Dan Sanderson

2,111
12
12

I'm not sure if this will help you since it's from 2010, but here's the old bulk loader chapter from the 1st edition of my App Engine book: http://ae-book.appspot.com/chapters/bulkdata This predates the version of bulk loader that uses `bulkloader.yaml`, but maybe it'll be useful. I believe the bulk loader is generally considered an undocumented feature of the Python SDK now. (Backup/restore is still a supported feature, but no longer relies on the bulk loader utility.) – Dan Sanderson Sep 14 '14 at 02:12

Updating data with bulkloader

1 Answers1