I have written some code with python to obtain the list of my followers and following users in Twitter. Once I have this information, I create nodes and relationships in neo4j with py2neo by looping over the list of followers and following users that I obtained.
The code seems to work fine, however not all nodes and relationships are created. I am trying to generate about 170 nodes, however only around 25 are created.
I am wondering if there is any kind of connection limit, of uploading threshold or any other thing that might be creating the problem.
I am using Python 3.6, py2neo 3.1.2 and neo4j Community distribution 3.1.3.
I am not a python expert, so please forgive my code:
import py2neo
from py2neo import Graph
from py2neo import Node, Relationship
from py2neo import authenticate
import tweepy
import time
auth = tweepy.OAuthHandler('...', '...')
auth.set_access_token('...', '...')
api = tweepy.API(auth, wait_on_rate_limit=True, wait_on_rate_limit_notify=True)
user = api.get_user(myUser)
def getFollowers(user_screen_name):
follower_ids=[]
for page in tweepy.Cursor(api.followers, screen_name=user_screen_name).pages():
time.sleep(60)
follower_ids.extend(page)
return follower_ids
def getFollowing(user_screen_name):
following_ids=[]
for page in tweepy.Cursor(api.friends, screen_name=user_screen_name).pages():
time.sleep(60)
following_ids.extend(page)
return following_ids
def createNode(screen_name):
node=Node("User", screen_name=screen_name)
gf.merge(node)
return
def createRelationship(nodeA, nodeB, relationship):
#creates relationship and nodes (if not existant)
nodeA=Node("User", screen_name=nodeA)
nodeB=Node("User", screen_name=nodeB)
gf.merge(nodeA)
gf.merge(nodeB)
gf.merge(Relationship(nodeA, relationship, nodeB))
return
authenticate("localhost:7474",myID,myPass)
gf = Graph()
#First time graph is created
gf.run("CREATE CONSTRAINT ON (u:User) ASSERT u.screen_name IS UNIQUE")
createNode(myUser)
user_followers=getFollowers(myUser)
user_following=getFollowing(myUser)
for followers in user_followers:
createRelationship(followers.screen_name, sc, "FOLLOWS")
i=1
for following in user_following:
createRelationship(sc, following.screen_name, "FOLLOWS")
I cannot think of any reason why did would not work, but I believe it is more a problem related to neo4j than the code itself.
Any help would be very much appreciated,
Thanks in advance