reindexObject fails during FileField to BlobField migration in Plone 4.0.7

Question

I'm trying to migrate from plone 3.3.5 to plone 4.0.7 and I'm stuck on a step that converts all the FileFields to BlobFields.

Plone upgrade script successfully converts all native FileFields but I have several custom AT-based classes which have to be converted manually. I've tried two ways of doing the conversion which leads me to the same error.

1. Using schemaextender as outlined in Plone migration guide and a source code example

2. Renaming all FileFields to blob fields and then running this script:

   from AccessControl.SecurityManagement import newSecurityManager
   from AccessControl import getSecurityManager
   
   from Products.CMFCore.utils import getToolByName
   from zope.app.component.hooks import setSite
   
   from Products.contentmigration.migrator import BaseInlineMigrator
   from Products.contentmigration.walker import CustomQueryWalker
   
   from plone.app.blob.field import BlobField
   
   
   admin=app.acl_users.getUserById("admin")
   newSecurityManager(None, admin)
   
   portal = app.plone
   setSite(portal)
   
   
   def find_all_types_fields(portal_catalog, type_instance_to_search):
       output = {}
       searched = []
       for k in catalog():
           kobj = k.getObject()
           if kobj.__class__.__name__ in searched:
               continue
           searched.append(kobj.__class__.__name__)
           for field in kobj.schema.fields():
               if isinstance(field, type_instance_to_search):
                   if kobj.__class__.__name__ in output:
                       output[kobj.__class__.__name__].append(field.__name__)
                   else:
                       output[kobj.__class__.__name__] = [field.__name__]
       return output
   
   def produce_migrator(field_map):
       source_class = field_map.keys()[0]
       fields = {}
       for x in field_map.values()[0]: fields[x] = None
   
       class FileBlobMigrator(BaseInlineMigrator):
           '''Migrating ExtensionBlobField (which is still a FileField) to BlobField'''
           src_portal_type = source_class
           src_meta_type = source_class
           fields_map = fields
           def migrate_data(self):
               '''Unfinished'''
               for k in self.fields_map.keys():
                   #print "examining attributes"
                   #import pdb; pdb.set_trace()
                   #if hasattr(self.obj, k):
                   if k in self.obj.schema.keys():
                       print("***converting attribute:", k)
                       field = self.obj.getField(k).get(self.obj)
                       mutator = self.obj.getField(k).getMutator(self.obj)
                       mutator(field)
           def last_migrate_reindex(self):
               '''Unfinished'''
               self.obj.reindexObject()
       return FileBlobMigrator
   
   def consume_migrator(portal_catalog, migrator):
       walker = CustomQueryWalker(portal_catalog, migrator, full_transaction=True)
       transaction.savepoint(optimistic=True)
       walker_status = walker.go()
       return walker.getOutput()
   
   def migrate_blobs(catalog, migrate_type):
       all_fields = find_all_types_fields(catalog, migrate_type)
       import pdb; pdb.set_trace()
       for k in [ {k : all_fields[k]} for k in all_fields]:
           migrator = produce_migrator(k)
           print consume_migrator(catalog, migrator)
   
   
   catalog = getToolByName(portal, 'portal_catalog')
   
   migrate_blobs(catalog, BlobField)
   

The problem occurs on self.obj.reindexObject() line where I receive the following traceback:

    2011-08-09 17:21:12 ERROR Zope.UnIndex KeywordIndex: unindex_object could not remove documentId -1945041983 from index object_provides.  This should not happen.
    Traceback (most recent call last):
    File "/home/alex/projects/plone4/eggs/Zope2-2.12.18-py2.6-linux-x86_64.egg/Products/PluginIndexes/common/UnIndex.py", line 166, in removeForwardIndexEntry indexRow.remove(documentId)
    KeyError: -1945041983
    > /home/alex/projects/plone4/eggs/Zope2-2.12.18-py2.6-linux-x86_64.egg/Products/PluginIndexes/common/UnIndex.py(192)removeForwardIndexEntry()
        191                            str(documentId), str(self.id)),
    --> 192                            exc_info=sys.exc_info())
        193         else:

If I remove the line that triggers reindexing, the conversion completes successfully, but if I try to manually reindex catalog later, every object that's been converted can no longer be found, and I'm a bit at loss of what to do now.

The site has LinguaPlone installed, maybe it has something to do with this?

Answer 1

One option would be to run the migration without the reindexObject() call and do a "Clear and Rebuild" in the catalog ZMI Advanced tab after migrating.