GeoDjango LayerMapping & Foreign Key

Question

I am trying to import my KML file into a model using GeoDjango's LayerMapping functionality. I've run tests and had no issues when doing regular imports. However, I recently added a foreign key to my model. My model is called PlaceMark and it now has a FK to a model called Layer. I would like to either

override the import and manually set the value of the foreign key field or
update my KML file to contain a new element that connects the PlaceMark to the layer via either the pk or name field of Layer.

Here is how I am testing from the shell and the relevant error:

>>>from locator import load
>>>load.run()
...
TypeError: ForeignKey mapping must be of dictionary type.
....

Here is my load.py file:

import os
from django.contrib.gis.utils import LayerMapping
from models import PlaceMark

placemark_mapping = {
    'name' : 'Name',
    'description' : 'Description',
    # This line below is the one that is suspect #
    'layer': 'Layer',
    'geom' : 'POINT25D',
}

placemark_kml = os.path.abspath(os.path.join(os.path.dirname(__file__), 'data/claim.kml'))

def run(verbose=True):
    lm = LayerMapping(PlaceMark, placemark_kml, placemark_mapping,
                      transform=False, encoding='iso-8859-1')

lm.save(strict=True, verbose=verbose)

KML File:

<?xml version="1.0" encoding="Windows-1252"?>
<kml xmlns="http://earth.google.com/kml/2.1">
<Folder>
  <description><![CDATA[TankSafe_Claims]]></description>
  <Placemark>
    <name><![CDATA[G2184729A]]></name>
    <description><![CDATA[<br><br><br>
    <table border="1" padding="0">
    <tr><td>Policy_Number</td><td>53645645</td></tr>
    <tr><td>Claim_Number</td><td>2342342234</td></tr>
    <tr><td>Policy_Type</td><td>TSP</td></tr>
    <tr><td>Name</td><td>Al's Total</td></tr>
    <tr><td>Street_Address</td><td>555 109th Avenue</td></tr>
    <tr><td>City</td><td>Pullman</td></tr>
    <tr><td>State</td><td>NY</td></tr>
    <tr><td>Zip_Code</td><td>55555</td></tr>
    <tr><td>County</td><td>Allegan</td></tr>
        ]]></description>
    <visibility>1</visibility>
    <open>0</open>
    <Point>
      <extrude>1</extrude>
      <altitudeMode>relativeToGround</altitudeMode>
      <coordinates>-86.092641,42.483953,0</coordinates>
    </Point>
    <!--- ***Should I add the line below?*** -->
    <Layer><name>claims</name></Layer>
  </Placemark>
</Folder>
</kml>

My goal is to just get all the PlaceMarks imported with references to the relevant layer. Any ideas?

Thanks! Larry

score 8 · Answer 1 · answered Oct 11 '17 at 13:48

layer_mapping = {
    'fk': {'nm_field': 'NAME'}, # foreign key field
    'this_field': 'THIS',
    'that_field': 'THAT',
    'geom': 'POLYGON',
}

the error you're receiving that the Foreign Key field should be a dictionary is basically requesting an additional mapping to the model which the foreign key relates.

in the above snippet:

'fk' is the foreign key field name from the model the data is being loaded into (lets call it 'load model')
'nm_field' is the field name from the model the 'load model' has the foreign key relationship to (lets call it 'primary model')
'NAME' is the field name from the data being loaded into 'load model' which holds the relationship to 'primary model'

more explicitly, imagine if 'primary model' is a dataset of lakes and they have a field called 'nm_field' that is the lake name as a string.

now imagine, 'load model' is a dataset of points representing all the buoys on all the lakes and has a field name 'fk' that is a ForeignKey to 'primary model' for the assignment of the lake each buoy belongs to.

finally, the data you're loading into 'load model' has a string field called 'NAME' and it contains the pre-populated name of the lake each buoy belongs to. that string name is the relationship tie. it allows the 'load model' to use that name to identify which lake in the 'primary model' it should establish a foreign key with.

score 3 · Answer 2 · answered Feb 22 '14 at 20:03

I tricked the LayerMapper into loading the ForeignKey field as a plain data-type after creating the tables.

Give USCounty an FK "state" to USState and run manage.py syncdb
Replace the "state" with "state_id" and the real datatype, usually models.IntegerField and execute the load.run() LayerMapper.
Return the "state" FK to the USCounty model.

Use Django normally.

In my case below, the "state" keys are 2-character FIPS codes.

class USCounty(models.Model):
    state = models.ForeignKey(USState)
    ## state_id = models.CharField(max_length=2)
    ...
    geom = models.MultiPolygonField(srid=4326)
    objects = models.GeoManager()

I'm doing the same but i'm really unhappy with this hack. I would like to be able to create and load the database from my .shp file without manual intervention — Antoine Claval, Oct 06 '15 at 19:12

score 1 · Answer 3 · edited May 23 '17 at 12:22

I worked around this by manually adding a temporary pre_save callback. You can connect it just for the record creation, then disconnect as soon as LayerMapping has done its work.

See 'My Solution' here - the 'black box' method I refer to is in fact exactly this use case.

The code that works for me:

def pre_save_callback(sender, instance, *args, **kwargs):
    fkey = some_method_that_gets_the_foreign_key()
    instance.type = fkey

# other mappings defined as usual
mapping = {
    'key1': 'KEY1',
    ...,
}

lm = LayerMapping(models.MyModel, PATH_TO_SHAPEFILE, mapping, transform=True)
# temporarily connect pre_save method
pre_save.connect(pre_save_callback, sender=models.MyModel)
try:
    lm.save(strict=True)
except Exception as exc:
    optional_error_handling()
    raise
finally:
    # disconnect pre_save callback
    pre_save.disconnect(pre_save_callback, sender=models.MyModel)

score 0 · Answer 4 · answered Jan 21 '14 at 15:19

It doesn't look like there is an easy way to hook into LayerMapping for foreign key fields. I solved this by using a for loop and the get_geoms() call. Thanks to http://invisibleroads.com/tutorials/geodjango-googlemaps-build.html

Here is an example of what I did:

placemark_kml = os.path.abspath(os.path.join(os.path.dirname(locator.__file__), 'data/claim.kml'))
datasource = DataSource(placemark_kml)
lyr = datasource[0]
waypointNames = lyr.get_fields('Name')
waypointDescriptions = lyr.get_fields('Description')
waypointGeometries = lyr.get_geoms()
for waypointName, waypointGeometry, waypointDescription in itertools.izip(waypointNames, waypointGeometries, waypointDescriptions):
    placemark = PlaceMark(name=waypointName, description=waypointDescription, geom=waypointGeometry.wkt)
    placemark.layer = Layer.objects.get(pk=8)
    placemark.save()

score 0 · Answer 5 · answered Jul 12 '16 at 15:38

Not an answer but hopefully a hint.

The error thrown comes from this part of the code. line ~220 of layermapping.py

elif isinstance(model_field, models.ForeignKey):
    if isinstance(ogr_name, dict):
        # Is every given related model mapping field in the Layer?
        rel_model = model_field.rel.to
        for rel_name, ogr_field in ogr_name.items():
            idx = check_ogr_fld(ogr_field)
            try:
                rel_model._meta.get_field(rel_name)
            except models.fields.FieldDoesNotExist:
                raise LayerMapError('ForeignKey mapping field "%s" not in %s fields.' %
                                    (rel_name, rel_model.__class__.__name__))
        fields_val = rel_model
    else:
        raise TypeError('ForeignKey mapping must be of dictionary type.')

At the beginning of the for loop, it looks for a dict: ogr_name.items()

ogr_name is actually defined as the value part of the mapping dict. The dict is supposed to be composed of the org field name and the related field name from the related model.

If anyone understands the origin of that ogr_name dict, it would be of great use.

GeoDjango LayerMapping & Foreign Key

5 Answers5