Transferring Twitter Tweets to a txt file

Question

from tweepy import Stream
from tweepy import OAuthHandler
from tweepy.streaming import StreamListener
import json 
from pprint import pprint

data_file = open('twitter.json')  
data = json.load(data_file)
##Json file with all the ckey, csecret, atoken, and asecret
pprint(data)

#consumer key, consumer secret, access token, access secret.
ckey = data["ckey"]
csecret = data["csecret"]
atoken = data["atoken"]
asecret = data["asecret"]

class listener(StreamListener):

def on_data(self, data):
    all_data = json.loads(data)       
    tweet = all_data["text"]        
    username = all_data["user"]["screen_name"]
    print((username,tweet))
    return True

def on_error(self, status):
    print (status)


auth = OAuthHandler(ckey, csecret)
auth.set_access_token(atoken, asecret)

The code above is all standard in accessing the twitter api. However, I need to transfer the tweets obtained from twitter to a .txt file. I tried using the code below twitterStream = Stream(auth, listener())

fid = open("cats based tweets.txt","w")
for tweet in twitterStream.filter(track=[cats]):
        fid.write(tweet)
    fid.close()

I intend on finding all twitter tweets/reposts that include the keyword cats, which it does. However, it is supposed to also write a txt file that includes all the tweets but it doesn't. Can anyone tell me what I need to do it fix it.

EDIT : I used the code that you guys have written but it doesn't return all of the tweets. It prints out like 5 or 6 then the error

RuntimeError: No active exception to reraise

appears and I have no idea why. Why does this occur cause I know it shouldn't.

score 1 · Answer 1 · answered Apr 14 '15 at 18:36

I guess there is a slight indentation error in the snippet you provided, However I will try to fix your error with 2 approaches, the first one is by correcting the indentation and the second one would be to change youron_data method

Approach 1:

fid = open("cats based tweets.txt","w")
for tweet in twitterStream.filter(track=[cats]):
    fid.write(tweet+"\n")
fid.close()

Or you could simply write the above code as :

with open("cats based tweets.txt","w") as fid:
    for tweet in twitterStream.filter(track=[cats]):
        fid.write(tweet+"\n")

Approach 2:

In the second approach we can change the on_data method so that when the program receives a new tweet it opens and file and directly writes to it , but for this we need to open the file in append mode, as opening the file in w writeable mode would overwrite the contents of the file again and again.

def on_data(self, data):
    all_data = json.loads(data)       
    tweet = all_data["text"]        
    username = all_data["user"]["screen_name"]
    print((username,tweet))
    with open("cats based tweets.txt","a") as fid:
        fid.write(tweet+"\n")
    return True

score 1 · Accepted Answer · answered Apr 17 '15 at 17:13

I've done this in a project and my method involves changing the on_data method within the StreamListener object. My code looks like this:

class Listener(StreamListener):
    def __init__(self, api=None, path=None):
        #I don't remember exactly why I defined this.
        self.api = api
        #We'll need this later.
        self.path = path

    def on_data(self, data):
        all_data = json.loads(data)

        tweet = all_data["text"]        
        username = all_data["user"]["screen_name"]
        print((username,tweet))

        #Open, write and close your file.
        savefile = open(file_path, 'ab')
        savefile.write(tweet)
        savefile.close()

        return True

A few things in the actual code, not where you redefined Listener or on_data. In order:

Define the file where you want to save. Let's call that variable the file_path. Don't forget to add the .txt extensions here.

Call the Stream and the Listener:

twitterStream = Stream(authorization, Listener(path=file_path))

Use your filters. Mine are coordinates and I put the filter in a try, except so that my code doesn't stop. Here it is adapted for you:
```
try:
    twitterStream.filter(track=[cats])
except Exception, e:
    print 'Failed filter() with this error:', str(e)
```

Now the text in the tweet should be written in the file whenever a text appears in the stream. Take a look at your file size and you should see it increase. Particularly, if your filter is about cats. Internet loves cats.

except Exception, e: print 'Failed filter() with this error:', str(e) — David Yi, Apr 19 '15 at 00:45
It returns a syntax error that I can't find where the issue is. Sorry for putting 3 comments but my keyboard is having issues. — David Yi, Apr 19 '15 at 00:46
I'm using Python 3.4.2 and at "except Exception, e:" it gives an error saying that there is a syntax error. If I put it as except (Exception, e): It gives that e is not defined — David Yi, Apr 19 '15 at 20:22
Mmm... ok. I'm using 2.7.something. I've seen other ways to write that. Try except Exception as e: — blue_chip, Apr 19 '15 at 23:11

score 0 · Answer 3 · answered Apr 13 '15 at 05:03

0

See the below link then you will know about how to save the tweets to Database as well as to the our local file.

https://github.com/anandstarz/Scrapee/blob/master/tweets

answered Apr 13 '15 at 05:03

Anandhakumar R

371
1
3
17

Transferring Twitter Tweets to a txt file

3 Answers3