1

I want to tokenize data from CSV file. I'm using this code and I'm unable to tokenize the entire column. I am only able to tokenize the first row in the column. The column is known as 'tweet'.

import pandas as pd
import nltk
from nltk import word_tokenize

data=pd.read_csv('/Users/yoshithKotla/Desktop/dingdang/nov19Tweets.csv')

Texts=list(data['tweet'].values)

tokenData = [nltk.word_tokenize(tweet) for tweet in Texts]

print(tokenData)
Yoshith Kotla
  • 135
  • 1
  • 3
  • 13
  • Could you add a sample dataset? Or best of all, could you add the link to the dataset, if there is one? – flaxel Apr 12 '21 at 12:40

1 Answers1

0

Try this code and see what u get:

import csv
from nltk import word_tokenize 
with open('/Users/yoshithKotla/Desktop/dingdang/nov19Tweets.csv', 'r') as csvfile:
   reader = csv.DictReader(csvfile)
   for row in reader:
       tweet = row["tweet"]
       print("Tweet: %s" % tweet)
       tokens = word_tokenize(tweet)
       print(tokens)

To save the output as csv file you can use csv.writer:

writer = csv.writer(open("path_to_output", 'w'))
for row in tokens:
    if counter[row[0]] >= 4:
        writer.writerow(row)
chikabala
  • 653
  • 6
  • 24