0

Following this SO post I have this code where I am using pythons tweepy library to read tweets by a user - user_timeline. I am writing the output to a text file but when I try to read through the text file I am getting json read error. Here is my current code:

import tweepy
import json
import unicodedata

# Consumer keys and access tokens, used for OAuth
access_token = ""
access_token_secret = ""
consumer_key = ""
consumer_secret = ""  

def read_user_timeline():
    # OAuth process, using the keys and tokens
    auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
    auth.set_access_token(access_token, access_token_secret)

    # Creation of the actual interface, using authentication
    api = tweepy.API(auth)

    for status in tweepy.Cursor(api.user_timeline, screen_name='@BillGates').items():

        with open('feed1.txt', 'a',encoding='utf-8') as f:

            print(status._json, file=f)

# A sample function to read the json tweets from the file
def read_tweets(tweets_data_path):

    tweets_file = open(tweets_data_path, "r",encoding='utf-8')
    for line in tweets_file:

        tweet = json.loads(line)
        print(tweet)


if __name__ == '__main__':

    read_user_timeline()
    read_tweets('feed1.txt')

When I run the above code I get error as : json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1) at line tweet = json.loads(line).

What I want to do is write each tweet data as a json format to file and then read through the text file and filter individual components(like text,created_at and so on) for further processing. In my understanding the issue seems to be in the my write-to-file code as it probably is not writing to file in json format but I am unable to find the correct way to do so. What is the error and how can it be corrected?

UPDATE: A sample line from the text file.

{'created_at': 'Sat Dec 23 03:51:46 +0000 2017', 'id': 944415209413754880, 'id_str': '944415209413754880', 'text': 'RT @IanD', 'truncated': False, 'entities': {'hashtags': [], 'symbols': [], 'user_mentions': [{'screen_name': 'Prof_IanD', 'name': 'Ian Davis', 'id': 748348755766702081, 'id_str': '748348755766702081', 'indices': [3, 13]}], 'urls': []}, 'source': '<a href="http://twitter.com/download/iphone" rel="nofollow">Twitter for iPhone</a>', 'in_reply_to_status_id': None, 'in_reply_to_status_id_str': None, 'in_reply_to_user_id': None, 'in_reply_to_user_id_str': None, 'in_reply_to_screen_name': None, 'user': {'id': 4761682514, 'id_str': '4761682514', 'name': '', 'screen_name': '', 'location': 'Salt Lake City, UT', 'description': "", 'url': '', 'entities': {'url': {'urls': [{'url': '', 'expanded_url': '', 'display_url': '', 'indices': [0, 23]}]}, 'description': {'urls': []}}, 'protected': False, 'followers_count': 1379, 'friends_count': 812, 'listed_count': 46, 'created_at': 'Fri Jan 15 06:32:10 +0000 2016', 'favourites_count': 799, 'utc_offset': None, 'time_zone': None, 'geo_enabled': True, 'verified': False, 'statuses_count': 2277, 'lang': 'en', 'contributors_enabled': False, 'is_translator': False, 'is_translation_enabled': False, 'profile_background_color': 'F5F8FA', 'profile_background_image_url': None, 'profile_background_image_url_https': None, 'profile_background_tile': False, 'profile_image_url': '', 'profile_image_url_https': '', 'profile_banner_url': '', 'profile_link_color': '1DA1F2', 'profile_sidebar_border_color': 'C0DEED', 'profile_sidebar_fill_color': 'DDEEF6', 'profile_text_color': '333333', 'profile_use_background_image': True, 'has_extended_profile': False, 'default_profile': True, 'default_profile_image': False, 'following': False, 'follow_request_sent': False, 'notifications': False, 'translator_type': 'none'}, 'geo': None, 'coordinates': None, 'place': None, 'contributors': None, 'retweeted_status': {'created_at': 'Fri Dec 22 20:22:34 +0000 2017', 'id': 944302164544913408, 'id_str': '944302164544913408', 'text': '', 'truncated': True, 'entities': {'hashtags': [], 'symbols': [], 'user_mentions': [], 'urls': [{'url': '', 'expanded_url': '', 'display_url': '', 'indices': [117, 140]}]}, 'source': '<a href="http://twitter.com/#!/download/ipad" rel="nofollow">Twitter for iPad</a>', 'in_reply_to_status_id': None, 'in_reply_to_status_id_str': None, 'in_reply_to_user_id': None, 'in_reply_to_user_id_str': None, 'in_reply_to_screen_name': None, 'user': {'id': 748348755766702081, 'id_str': '748348755766702081', 'name': 'Ian Davis', 'screen_name': 'Prof_IanD', 'location': 'Melbourne, Australia', 'description': '', 'url': None, 'entities': {'description': {'urls': []}}, 'protected': False, 'followers_count': 573, 'friends_count': 264, 'listed_count': 17, 'created_at': 'Thu Jun 30 02:53:41 +0000 2016', 'favourites_count': 1197, 'utc_offset': 39600, 'time_zone': 'Melbourne', 'geo_enabled': True, 'verified': False, 'statuses_count': 1813, 'lang': 'en', 'contributors_enabled': False, 'is_translator': False, 'is_translation_enabled': False, 'profile_background_color': 'F5F8FA', 'profile_background_image_url': None, 'profile_background_image_url_https': None, 'profile_background_tile': False, 'profile_image_url': '', 'profile_image_url_https': '', 'profile_banner_url': 'https://pbs.twimg.com/profile_banners/748348755766702081/1467417750', 'profile_link_color': '1DA1F2', 'profile_sidebar_border_color': 'C0DEED', 'profile_sidebar_fill_color': 'DDEEF6', 'profile_text_color': '333333', 'profile_use_background_image': True, 'has_extended_profile': False, 'default_profile': True, 'default_profile_image': False, 'following': False, 'follow_request_sent': False, 'notifications': False, 'translator_type': 'none'}, 'geo': None, 'coordinates': None, 'place': None, 'contributors': None, 'is_quote_status': True, 'quoted_status_id': 944284512371990528, 'quoted_status_id_str': '944284512371990528', 'quoted_status': {'created_at': 'Fri Dec 22 19:12:26 +0000 2017', 'id': 944284512371990528, 'id_str': '944284512371990528', 'text': '', 'truncated': True, 'entities': {'hashtags': [], 'symbols': [], 'user_mentions': [{'screen_name': '', 'name': '', 'id': 4761682514, 'id_str': '4761682514', 'indices': [53, 65]}], 'urls': [{'url': '', 'expanded_url': 'https://twitter.com/i/web/status/944284512371990528', 'display_url': 'twitter.com/i/web/status/9…', 'indices': [117, 140]}]}, 'source': '<a href="http://twitter.com/download/iphone" rel="nofollow">Twitter for iPhone</a>', 'in_reply_to_status_id': None, 'in_reply_to_status_id_str': None, 'in_reply_to_user_id': None, 'in_reply_to_user_id_str': None, 'in_reply_to_screen_name': None, 'user': {'id': 24660850, 'id_str': '24660850', 'name': '', 'screen_name': '', 'location': 'Los Angeles, CA ', 'description': '', 'url': '', 'entities': {'url': {'urls': [{'url': '', 'expanded_url': '', 'display_url': '', 'indices': [0, 23]}]}, 'description': {'urls': [{'url': '', 'expanded_url': '', 'display_url': '', 'indices': [122, 145]}]}}, 'protected': False, 'followers_count': 2624, 'friends_count': 937, 'listed_count': 89, 'created_at': 'Mon Mar 16 06:46:19 +0000 2009', 'favourites_count': 2741, 'utc_offset': -28800, 'time_zone': 'Pacific Time (US & Canada)', 'geo_enabled': True, 'verified': False, 'statuses_count': 3755, 'lang': 'en', 'contributors_enabled': False, 'is_translator': False, 'is_translation_enabled': False, 'profile_background_color': 'C6E2EE', 'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme2/bg.gif', 'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme2/bg.gif', 'profile_background_tile': False, 'profile_image_url': 'http://pbs.twimg.com/profile_images/98109473/twitter_normal.jpg', 'profile_image_url_https': 'https://pbs.twimg.com/profile_images/98109473/twitter_normal.jpg', 'profile_banner_url': 'https://pbs.twimg.com/profile_banners/24660850/1444348252', 'profile_link_color': '1F98C7', 'profile_sidebar_border_color': 'C6E2EE', 'profile_sidebar_fill_color': 'DAECF4', 'profile_text_color': '663B12', 'profile_use_background_image': True, 'has_extended_profile': False, 'default_profile': False, 'default_profile_image': False, 'following': False, 'follow_request_sent': False, 'notifications': False, 'translator_type': 'none'}, 'geo': None, 'coordinates': None, 'place': {'id': 'ddf8317a51899bd8', 'url': 'https://api.twitter.com/1.1/geo/id/ddf8317a51899bd8.json', 'place_type': 'city', 'name': 'Duarte', 'full_name': 'Duarte, CA', 'country_code': 'US', 'country': 'United States', 'contained_within': [], 'bounding_box': {'type': 'Polygon', 'coordinates': [[[-117.9906137, 34.1182559], [-117.9297178, 34.1182559], [-117.9297178, 34.165263], [-117.9906137, 34.165263]]]}, 'attributes': {}}, 'contributors': None, 'is_quote_status': True, 'quoted_status_id': 944262989942734848, 'quoted_status_id_str': '944262989942734848', 'retweet_count': 11, 'favorite_count': 49, 'favorited': False, 'retweeted': False, 'possibly_sensitive': False, 'lang': 'en'}, 'retweet_count': 6, 'favorite_count': 13, 'favorited': False, 'retweeted': False, 'possibly_sensitive': False, 'lang': 'en'}, 'is_quote_status': True, 'quoted_status_id': 944284512371990528, 'quoted_status_id_str': '944284512371990528', 'retweet_count': 6, 'favorite_count': 0, 'favorited': False, 'retweeted': False, 'lang': 'en'}
user2916886
  • 847
  • 2
  • 16
  • 35
  • Have you looked at the file, or the `line`, to see what's in it? – jonrsharpe Dec 29 '17 at 09:12
  • @jonrsharpe yes I looked at the file and when I print the `line` without passing it through `json.loads()` it prints the line – user2916886 Dec 29 '17 at 09:24
  • Well could you *include* a line that fails to parse? Are you sure it's not, for example, failing on an *empty* line? – jonrsharpe Dec 29 '17 at 09:24
  • As the message says, you need *double* quotes for JSON; you're outputting a python dictionary literal, **not** JSON. Either use `literal_eval` to convert it back, or write actual JSON to the file. – jonrsharpe Dec 29 '17 at 09:39
  • @jonrsharpe I have pasted a line from the file in above post. Its a very long output line so I had to remove few data from it to match SOs rules. No its not failing at empty line as I do not see any output coming from `print(tweet)` line in my code above – user2916886 Dec 29 '17 at 09:39
  • @jonrsharpe How do I write the actual JSON to the file? I assumed the issue is in my write to file code but don't know how to write the data as json as I thought it is writing to json only? – user2916886 Dec 29 '17 at 09:40
  • Use `json.dumps`? Or read the API docs to see if you can get the raw string content. – jonrsharpe Dec 29 '17 at 09:43
  • Here is a working sample i made with your needs : https://pastebin.com/GPEa6Ai5 Just to unstuck you but this doesn't explain your issue. – Jérémy JOKE Dec 29 '17 at 10:42

0 Answers0