2

I have a model in App Engine that I am exporting in CSV format and then importing to my local dev environment using appcfg.py and bulkloader.yaml.

I can import and export most models, but I am having issues with models that have a foreign key which is not always present. I can use the lambda import below to always import the foreign key as None or use the create_foreign_key() transform to import the foreign key when every line in my csv file has the foreign key.

How do I configure bulkloader.py to import the foreign key when it is present and ignore it when it is not present?

- kind: MyModel
  connector: csv
  connector_options:
  property_map:
    - property: myOtherModel
      external_name: myOtherModel
      import_transform: "lambda x: x is None and None or None"
      #import_transform: transform.create_foreign_key('MyOtherModel', key_is_id=True)
      export_transform: transform.key_id_or_name_as_string

Just uncommenting out the second import_transform in place of the lambda transform will produce this error. 

File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/bulkload/transform.py", line 127, in generate_foreign_key_lambda
    value = int(value)
ValueError: invalid literal for int() with base 10: ''

This error occurs when I run appcfg.py. All other imports without foreign keys or where foreign keys are always present are working properly.

appcfg.py upload_data --config_file=bulkloader.yaml --num_threads=1 --batch_size=50 --url=http://localhost:8080/remote_api --email=Chris --passin --kind=MyModel --filename=MyModel.csv

The myOtherModel column in the csv file sometimes contains the MyOtherModel.key().id() and sometimes not.

Eg.

myOtherModel
1234
4567

2345

5678
James
  • 773
  • 2
  • 18
  • 29
Chris
  • 4,237
  • 6
  • 30
  • 42
  • They match. If I put an id in the column for every MyOtherModel foreign key property in the CSV file for MyModel, everything will work. It seems as if I'm not getting the default to None option. – Chris Jan 23 '12 at 10:16
  • Ok. Thanks for trying. Can you assist me in writing this lambda line correctly? lambda x: int(x)>0 and x or "Create Foreign Key" – Chris Jan 23 '12 at 12:27
  • The problem seems to be that for some reason the bulkloader process is attempting to parse a blank '' as an int. That is why I'm trying to just do the lambda expression, so that I can catch the non-existent foreign keys and prevent the error. Thanks for pointing out the other StackOverflow problems. I had looked at them, but none of them involve returning keys. I need to be able to return create_foreign_key() when a number is present in the column, but I can't find the syntax. – Chris Jan 23 '12 at 13:42

2 Answers2

2

Default behivour when foreign key value does not exist, reference property takes value: None and code for this:

import_transform: transform.create_foreign_key('MyOtherModel')

As in your error message: value does not exist shows that you uses 'othercolumn','','anohtercolumn' instead of 'othercolumn',,'anohtercolumn'

So if the source '' is then to handle this:

from this link : http://eikke.com/python-ifelse-in-lambda/ as Thomas Thurman comments, lambda expression should look like:

import-transform: "lambda x: [x, None][x=='']"

I hope it works for you

asdf_enel_hak
  • 7,474
  • 5
  • 42
  • 84
  • No. That is the problem. Using the transform.create_foreign_key() line will result in the error that I have just added above. – Chris Jan 23 '12 at 09:30
0

One way to handle this would be to:

  1. Import all of the entities that don't have keys with a bulkloader.yaml that doesn't have a key property import transform specified for the model.

  2. Export all of the entities.

  3. Import the entities. (all of which will now have key values)

A slightly better variation on this:

  1. Import all of the entities that have keys using a bulkloader.yaml with a key property import transform.

  2. Import all of the entities without keys using a bulkloader.yaml without the key property import transform.

This is a bit cumbersome. I would like a way to have entities missing a key value automatically assigned one upon import.

James
  • 773
  • 2
  • 18
  • 29