13

I have CSV file that looks like the following, 

1994, Category1, Something Happened 1
1994, Category2, Something Happened 2
1995, Category1, Something Happened 3
1996, Category3, Something Happened 4
1998, Category2, Something Happened 5

I want to create two lists, 

Category = [Category1, Category2, Category3]

and

Year = [1994, 1995, 1996, 1998]

I want to omit the duplicates in the column. I am reading the file as following, 

DataCaptured = csv.reader(DataFile, delimiter=',')  
DataCaptured.next()

and Looping through, 

   for Column in DataCaptured:
Gravity M
  • 1,485
  • 5
  • 16
  • 28

3 Answers3

13

You can do:

DataCaptured = csv.reader(DataFile, delimiter=',', skipinitialspace=True) 

Category, Year = [], []
for row in DataCaptured:
    if row[0] not in Year:
        Year.append(row[0])
    if row[1] not in Category:
        Category.append(row[1])    

print Category, Year        
# ['Category1', 'Category2', 'Category3'] ['1994', '1995', '1996', '1998']

As stated in the comments, if order does not matter, using a set would be easier and faster:

Category, Year = set(), set()
for row in DataCaptured:
    Year.add(row[0])
    Category.add(row[1])
dawg
  • 98,345
  • 23
  • 131
  • 206
7

A very concise way to do this is to use pandas, the benefits are: it has a faster CSV pharser; and it works in columns (so it only requires one df.apply(set) to get you there) :

In [244]:
#Suppose the CSV is named temp.csv
df=pd.read_csv('temp.csv',header=None)
df.apply(set)
Out[244]:
0                        set([1994, 1995, 1996, 1998])
1            set([ Category2,  Category3,  Category1])
2    set([ Something Happened 4,  Something Happene...
dtype: object

The downside is that it returns a pandas.Series, and to get access each list, you need to do something like list(df.apply(set)[0]).

Edit

If the order has to be preserved, it can be also done very easily, for example:

for i, item in df.iteritems():
    print item.unique()

item.unique() will return numpy.arrays, instead of lists.

Community
  • 1
  • 1
CT Zhu
  • 52,648
  • 17
  • 120
  • 133
2

dawg pointed out one of the greatest tricks in Python: using set() to remove duplicates from a list. dawg shows how to build the unique list from scratch by adding each item to a set, which is perfect. But here's another equivalent way to do it, generating a list with duplicates and a list without duplicates using a list(set()) approach:

import csv

in_str = [
    'year, category, event',
    '1994, Category1, Something Happened 1',
    '1994, Category2, Something Happened 2',
    '1995, Category1, Something Happened 3',
    '1996, Category3, Something Happened 4',
    '1998, Category2, Something Happened 5'
    ]

cdr = csv.DictReader(in_str, skipinitialspace=True)

col = []
for i in cdr:
    col.append(i['category'])

# all items in the column...
print(col)
# only unique items in the column...
print(list(set(col)))
rob_7cc
  • 797
  • 6
  • 16