0

We have a use case where users must be able to search content that is only available in Groups that they have access to. The search must be across all groups that they have access to.

Some details: A Group has many Posts, and a user may have access to hundreds of Groups and thousands of Posts within each Group. A search for "Foo" should return all Groups with "Foo" in the name and all Posts, within the Groups that they have access to, and have "Foo" in the content.

The way I thought of dealing with it is to have a list of user_id's associated on each document index and then include the user_id in the query string to verify that the user has access. Once the results are returned we could do an additional check to see that they have access to the content before returning the results.

The document index is something like this:

fields = [
  search.TextField(name="data", value="some searchable stuff"),
  search.AtomField(name="post_id", value="id of post"),
  search.AtomField(name="group_id", value="id of group"),
  search.AtomField(name="user_id", value=user_id_1),
  search.AtomField(name="user_id", value=user_id_2),
  #.... add the thousand other users who have access to the group (done in loop)     
]

#then query run a user 123 would be as follows:
results = index.search("data = Foo AND user_id = 123")

My concern with the above approach: Every new user who subscribes to a group would require the search index to be reindexed to include their user_id on each document.

Is there a better way of handling this use case?

Thanks Rob

Rob Curtis
  • 2,245
  • 24
  • 33
  • Why do you need to include user_id if you already included group_id, and you know which groups a user belongs to? – Andrei Volgin Sep 08 '16 at 17:50
  • @AndreiVolgin so that a user can search site wide, across all groups that they belong to). This could be in the 100s. Are you suggesting passing in the list of group_ids as part of your query? If there are 100, wouldn't 100 OR queries be expensive? – Rob Curtis Sep 08 '16 at 19:13

1 Answers1

0

There is no simple answer to your question. You need to plan for (a) a typical use-case, and (b) extreme cases.

If a typical user belongs to 1-3 groups, searching by group_id maybe the best solution. You will do 1-2 extra searches, but you won't need to re-index every document every time a user joins or exits a group, which is prohibitively expensive.

You can have a separate implementation for extreme cases. If a user belongs to more than X groups, it may be more efficient to retrieve all results matching the keyword, and then filter them by group_id.

An alternative approach is to always retrieve all results regardless of group_id/user_id, and store them in Memcache. Then you can filter them in memory.

Users tend to search using the same keywords - depending on your corpus, 1% of words may account for up to 99% of searches. If you have a lot of users - and a big enough cache - you will get a lot of cache hits. Note that 1GB of cache can fit tens or even hundreds of thousands of query results. An additional advantage of this approach is that it speeds up all queries, especially phrase or multi-keyword searches.

Andrei Volgin
  • 40,755
  • 6
  • 49
  • 58
  • thanks for your response. I see the the cost to index 1GB is $2.00. Do you have any idea how the data use is calculated for content submitted? What I mean is if one submits 1KB of data to be indexed, does the indexed amount increase? Also, do you have any idea whether indexing cost also accounts for cpu cost? Thanks – Rob Curtis Sep 08 '16 at 20:15
  • These are the questions for Google support. I would assume that "indexing" itself is not included in instance costs. I haven't looked at their pricing for Search API for awhile. – Andrei Volgin Sep 08 '16 at 20:27