I'm using Mongoengine to perform query against MongoDB database and then save result as json file. My document definition looks like this:
class Foo(Document):
# fields definitions goes here
@queryset_manager
def some_fields(self, queryset):
return queryset.exclude('field1', 'field2', 'field3')
As you can see, I've created custom method to exclude some unnecessary fields from the query result. But the the whole process query->count results->convert to json is terribly slow. I've created some dummy logs to better explain my problem
query_start = datetime.datetime.now()
foos = Foo.some_fields.filter(**filter_dict)
query_end = datetime.datetime.now()
logger.debug(
"%s || Query time: %s" % (str(datetime.datetime.now()), str((query_end - query_start).total_seconds())))
counting_start = datetime.datetime.now()
foos_count = foos.count()
counting_end = datetime.datetime.now()
logger.debug("%s || Foos count: %d in %f" % (
str(datetime.datetime.now()), foos_count,(counting_end - counting_start).total_seconds(),
))
serialization_start = datetime.datetime.now()
json_response = foos.to_json()
serialization_end = datetime.datetime.now()
logger.debug("%s || Serialization time: %s" % (
str(datetime.datetime.now()), str((serialization_end - serialization_start).total_seconds())))
The example output below
Query time: 0.098477
Foos count: 0 in 27.771622
Serialization time: 34.59575
I'm guessing that result from previous step is not being used by the next part. Am I right? Because I can't find any other explanation that converting empty result to json takes over half a minute. Or maybe I'm missing something? How can I change my code to perform these operations much faster?
