So I have to implement a nearest neighbor search feature on my app, and I am currently using Parse as the backend. What I was doing till now was to perform a query and get the results and then sort them to get the top 20-30 results. But since my database has scaled up, I now have about 4,000 locations (expect to reach about 15,000) on which I have to apply the nearest neighbor search. This is not good for a real time system.
What I have thought of:
I can code an efficient solution using QuadTree, but here's a catch. I understand that I can make a job and create the QuadTree and keep it in memory, but this seems wasteful as there always will be a complete thread devoted to maintaining the tree in the memory, not to mention constant checking and balancing. And in the event that thread fails, I will probably have to manually start it.
The other solution could be to create a QuadTree object and store it in stable memory, and read that object everytime the query comes in, and serve the results. But I think this will be slow too.
How should I solve this problem? Or should I try another BaaS, or make a custom API using AWS or AppEngine? I really don't want the hassle of managing load and security features at this point, as this is a non-profit project.