0

I have a script which send requests to social media site by doing following:

It first scrapes the friends of the account inserted. It then continues to scrape all friends of the accounts found forever (Similar to how search engine crawlers work). Add them to a consumer queue which then adds them as a friend or send them a message. All this in 10-30 threads. I am currently using Queue and it is not checking if the accounts it finds were duplicate of previously found account. That is my problem. Before changing the source code of Queue module. Is there any similar module with duplicate filtering built in.

charles M
  • 75
  • 1
  • 7

1 Answers1

0

Python also includes a data type for sets. A set is an unordered collection with no duplicate elements. Note: to create an empty set you have to use 

set()

There is an ordered set recipe for this which is referred to from the Python 2 Documentation http://code.activestate.com/recipes/576694/

This runs on Py2.6 or later and 3.0 or later without any modifications. The interface is almost exactly the same as a normal set, except that initialisation should be done with a list.

OrderedSet([1, 2, 3])
nivhanin
  • 1,688
  • 3
  • 19
  • 31