Filter Twitter feeds only by language

Question

I am using Tweepy API for extracting Twitter feeds. I want to extract all Twitter feeds of a specific language only. The language filter works only if track filter is provided. The following code returns 406 error:

l = StdOutListener()
auth = OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
stream = Stream(auth, l)
stream.filter(languages=["en"])

How can I extract all the tweets from certain language using Tweepy?

Have you solved this issue. I also want to extract specific language tweet like japanese. — tktktk0711, Sep 22 '17 at 15:03

score 28 · Answer 1 · answered Nov 14 '14 at 00:45

You can't (without special access). Streaming all the tweets (unfiltered) requires a connection to the firehose, which is granted only in specific use cases by Twitter. Honestly, the firehose isn't really necessary--proper use of track can get you more tweets than you know what to do with.

Try using something like this:

stream.filter(languages=["en"], track=["a", "the", "i", "you", "u"]) # etc

Filtering by words like that will get you many, many tweets. If you want real data for the most-used words, check out this article from Time: The 500 Most Frequently Used Words on Twitter. You can use up to 400 keywords, but that will likely approach the 1% limit of tweets at a given time interval. If your track parameter matches 60% of all tweets at a given time, you will still only get 1% (which is a LOT of tweets).

Thanks!! I was typing `stream.filter(lang=["en"], track=["something"])` — Krishh, Apr 05 '16 at 23:15

score 8 · Answer 2 · answered Apr 02 '18 at 15:43

8

Try lang='en' param in Cursor() e.g.

tweepy.Cursor(.. lang='en')

answered Apr 02 '18 at 15:43

Aziz Alto

19,057
5
77
60

Jay Mehta · Answer 3 · 2018-03-24T05:24:44.557

4

Other than getting filtered tweets directly, you can filter it after getting all tweets of different languages by:

tweets = api.search("python")
for tweet in tweets:
   if tweet.lang == "en":
      print(tweet.text)
      #Do the stuff here

Hope it helps.

edited Mar 24 '18 at 05:24

answered Nov 19 '16 at 13:16

Jay Mehta

1,511
15
20

score 2 · Answer 4 · answered Jan 30 '19 at 16:27

You can see the arguments for the track method in the github code https://github.com/tweepy/tweepy/blob/master/tweepy/streaming.py

Put languages in a array of ISO_639-1_codes.

They are:

filter(self, follow=None, track=None, is_async=False, locations=None,
               stall_warnings=False, languages=None, encoding='utf8', filter_level=None):

So to track by languages just put:

class Listener(StreamListener):

    def on_data(self, data):
        j = json.loads(data)
        t = {
          'screenName' : j['user']['screen_name'],
          'text:': j['text']
          }
        print(t)
        return(True)

    def on_status(self, status):
        print(status.text)


auth = OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)

stream = Stream(auth=auth, listener=Listener(),wait_on_rate_limit=True,wait_on_rate_limit_notify=True)

stream.filter(track=['Trump'],languages=["en","fr","es"])

score 1 · Answer 5 · answered Oct 08 '19 at 19:48

1

Tweepy search allows to fetch tweets for specific language. You can use ISO 639-1 code to specify the value for language parameter. Following code will fetch tweets with full text in specified language (English for below example)

    auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
    auth.set_access_token(access_token, access_token_secret)
    api = tweepy.API(auth)
    tweets = api.search(q = keywordtosearch, lang = 'en', count = 100, truncated = False, tweet_mode = 'extended')
    for tweet in tweets:
        print(tweet.full_text)
        #add your code

answered Oct 08 '19 at 19:48

Vishal Kharde

1,553
3
16
34

tried your code, got `AttributeError: 'API' object has no attribute 'search'` – Maged Saeed Oct 12 '21 at 17:05
1

replace search with search_tweets – Bruno Gois Oct 25 '21 at 01:34

score 0 · Answer 6 · answered Feb 03 '19 at 10:30

This worked for me.

auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth)
a=input("Enter Tag: ")
tweets = api.search(a, count=200)
a=[]
for tweet in tweets:
    if tweet.lang == "en":
        a.append(tweet.text)

score 0 · Answer 7 · answered May 25 '20 at 10:05

With the help of GetOldTweets3 (https://pypi.org/project/GetOldTweets3/), you can download tweets (even old ones) by filtering over few criteria, as shown below:

tweetCriteria = got.manager.TweetCriteria().setQuerySearch('Coronavirus')\
                                       .setSince("2020-02-15")\
                                       .setUntil("2020-03-29")\
                                       .setMaxTweets(5)\
                                       .setNear('India')\
                                       .setLang('en')
tweets = got.manager.TweetManager.getTweets(tweetCriteria)
for tweet in tweets:
    print(tweet.text)
    print(tweet.date)
    print(tweet.geo)
    print(tweet.id)
    print(tweet.permalink)
    print(tweet.username)
    print(tweet.retweets)
    print(tweet.favorites)
    print(tweet.mentions)
    print(tweet.hashtags)
    print('*'*50)

Filter Twitter feeds only by language

7 Answers7