2

I am trying to copy the datastore from one app to another (live to dev server). I first need to delete all the entities on the dev server (the actual dev app I'm using, not to be confused with the local dev server). I am doing so in the datastore admin console, but after the process completes, I still have _GAE_MR_MapreduceState and _GAE_MR_ShardState entities remaining. These should be temporary during the delete process as far as I understand. Besides this, occasionally when I go to the datastore admin console (about half the time) I also see a number of my deleted entities also showing up again. I have deleted them again a couple times.

I don't see them in the datastore viewer, but I don't know if this is always the case. I've just yet to see them show up in this view again. In datastore viewer there are only entities of type _AE_Backup_Information, _AE_Backup_Information_Kind_Files, _AE_Backup_Information_Kind_Type_Info and _AE_DatastoreAdmin_Operation. I don't see _GAE_MR_MapreduceState and _GAE_MR_ShardState entities in this view.

UPDATE: I just was able to catch one of my deleted entities in the datastore viewer. Although there was only one of the deleted entity types and only one of the entities visible here (before I was seeing 3 or 4 entity types and thousands of entities). Right after I caught it in the datastore viewer I went to the datastore admin console and found that the same entity type was visible here, although two of them in this case. I also didn't find _GAE_MR_MapreduceState and _GAE_MR_ShardState nor _AE_Backup_Information, _AE_Backup_Information_Kind_Files, _AE_Backup_Information_Kind_Type_Info and _AE_DatastoreAdmin_Operation entities in the datastore admin this time. END UPDATE

I have not tried deleting the _GAE_MR_MapreduceState and _GAE_MR_ShardState entities in the datastore admin console. This seems kind of unsafe as these entities are created during a delete operation, I don't know what it would do. I would attempt deleting them in the datstore viewer, but they aren't showing up there.

Those times that my deleted entities show up in the datastore admin console _AE_Backup_Information, _AE_Backup_Information_Kind_Files, _AE_Backup_Information_Kind_Type_Info and _AE_DatastoreAdmin_Operation entities are also visible there as well. So this is strange too.

There are no tasks on the task queue and no jobs are running.

Is this some sort of synchronization issue? Is a cache not being updated somewhere? Is there some way I can reset something to get these entities cleared out?

UPDATE 2: I tried flushing Memcache and this did not make a difference. The entities continue to show up in the datastore admin console, pretty much 100% of the time. I have not seen any of the deleted entities show up in the datastore viewer however since I deleted the one I caught showing up before.

UPDATE 3: I just attempted to perform the same type of datastore delete operation in another app and I am getting the same corruption as a result. I first created a backup, then selected and deleted all entities, both operations from the datastore admin console.

The delete operation jobs completed. They show up logged under completed operations. No more jobs are left in the task queue. But all of the entities are visible again in the datastore admin console! I deleted 10 entity types, most with a couple dozen entities, one with several thousand. Every one of the entity types is listed again with the same number of entities in the datastore admin console.

In the datastore viewer, two of the entity types still remain, with a smaller (but still significant) number of entities than before the delete operation. One with 2,172 from 6,855 before the delete. The other with 17 from 44 before the delete.

Again I flushed the Memcache. It made no difference.

I see that the datastore admin is experimental, but I wouldn't expect it to completely break my datastore by performing a simple delete operation with no way to reset the datastore afterwards. I now have two apps, both with billing enabled costing my client money, that are unusable because their datastores are corrupted. What are my options here? Is it possible I can get a response from someone at Google? This is the recommended forum for Google App Engine support.

SimonJ
  • 21,076
  • 1
  • 35
  • 50
Jeff Lockhart
  • 5,379
  • 4
  • 36
  • 51
  • why don't you just create a new dev server data store ? I know that doesn't solve the problem of entities turning up. – Tim Hoffman Mar 08 '13 at 23:59
  • Do you mean a new app? The datastore is associated with the app. I'd like to maintain my app version history and other history and settings associated with the current app as well as the name and URL, which would not be possible if I were to create a new app. Other than that, I don't know of a way to create a new datastore for an app. – Jeff Lockhart Mar 09 '13 at 00:02
  • You might want to set up a way to generate a datastore with representative data. Some combination of programmatically creating entities and copying over select entities from prod. You shouldn't have to deal with _GAE and _AE (private) entities. – Joel Burget Mar 09 '13 at 00:09
  • I don't understand your solution. I haven't gotten to copying the production datastore yet as I'm still trying to purge the old datstore entities from the dev app. They continuously keep showing up in the datastore admin console after being deleted. – Jeff Lockhart Mar 09 '13 at 00:58
  • @jeffdgr8 - no I meant create a new dev data store, if you start the devserver with the `--clear_datastore` option it will be empty. This wil have no impact on your code etc.. – Tim Hoffman Mar 09 '13 at 04:07
  • @TimHoffman That makes sense what you were saying now, but my dev server is an actual GAE app. I just have two apps, one I'm using as the live server and one for testing and development. When I say dev server I don't mean local dev server, but just the GAE app I'm using for development. Sorry if this wasn't clear. My troubles are with the GAE datastore, not the local one. Since the live server app has the most up-to-date datastore, I need to copy it to the dev server app. This is what I'm trying to do. – Jeff Lockhart Mar 09 '13 at 07:14
  • @Jeff- could you find any solution? I am also facing same issue where I am not able to delete entities from datastore admin consoles. They show up despite deleting. I =n my case I had removed few properties from my entity and I wanted that change to reflect in my datastore,hence was trying to delete all existing data from datastore. One possible workaround could be to delete entities from the code. – user1938357 Jun 23 '13 at 06:00
  • @user1938357 I have not found a solution, no. I have been able to deduce that both the datastore admin console and datastore viewer do not necessarily display up-to-date info about the actual current datastore state, which is annoying when performing large migration tasks because you can't get immediate feedback as to the results. It's not that none of the changes show up either, usually most of them do. But it's the fact that not all changes show up that makes the tools less than useful. The only sure way to get accurate info is to query the datastore directly in code and log the results. – Jeff Lockhart Jun 24 '13 at 00:44

1 Answers1

0

It looks like you have old style exports set up in your dev app. You should probably check your cron.yaml and disable those exports. Ideally, you would switch to the newer (and cheaper) exports using the Managed Import/Export service. For bulk deletes the current recommendation is to use Cloud Dataflow with the bulk delete template. Both of the Managed Import/Export & Cloud Dataflow based delete will not use the Cloud Datastore database in your project, where both the old style exports and deletes will use your Cloud Datastore database.

Jim Morrison
  • 2,784
  • 1
  • 7
  • 11