get column names from numpy genfromtxt in python

Question

using numpy genfromtxt in python, i want to be able to get column headers as key for a given data. I tried the following, but not able to get the column names for the corresponding data.

column = np.genfromtxt(pathToFile,dtype=str,delimiter=',',usecols=(0))
columnData = np.genfromtxt(pathToFile,dtype=str,delimiter=',')
data = dict(zip(column,columnData.tolist()))

Below is the data file

header0,header1,header2
mydate,3.4,2.0
nextdate,4,6
afterthat,7,8

Currently, it shows data as

{
  "mydate": [
    "mydate",
    "3.4",
    "2.0"
  ],
  "nextdate": [
    "nextdate",
    "4",
    "6"
  ],
  "afterthat": [
    "afterthat",
    "7",
    "8"
  ]
}

I want to get to this format

{
  "mydate": {
    "header1":"3.4",
    "header2":"2.0"
  },
  "nextdate": {
    "header1":"4",
    "header2":"6"
  },
  "afterthat": {
   "header1":"7",
   "header2":  "8"
  }
}

any suggestions?

have you considered [pandas](http://pandas.pydata.org/)? – bananafish Aug 29 '16 at 22:30 — bananafish, Aug 29 '16 at 22:30

MaxU - stand with Ukraine · Answer 1 · 2016-08-29T22:37:41.413

2

Using pandas module:

In [94]: fn = r'D:\temp\.data\z.csv'

read CSV into data frame:

In [95]: df = pd.read_csv(fn)

In [96]: df
Out[96]:
     header0  header1  header2
0     mydate      3.4      2.0
1   nextdate      4.0      6.0
2  afterthat      7.0      8.0

getting desired dict:

In [97]: df.set_index('header0').to_dict('index')
Out[97]:
{'afterthat': {'header1': 7.0, 'header2': 8.0},
 'mydate': {'header1': 3.3999999999999999, 'header2': 2.0},
 'nextdate': {'header1': 4.0, 'header2': 6.0}}

or as a JSON string:

In [107]: df.set_index('header0').to_json(orient='index')
Out[107]: '{"mydate":{"header1":3.4,"header2":2.0},"nextdate":{"header1":4.0,"header2":6.0},"afterthat":{"header1":7.0,"header2":8.0}}'

edited Aug 29 '16 at 22:37

answered Aug 29 '16 at 22:32

MaxU - stand with Ukraine

205,989
36
386
419

Why 3.3999999999999999?? – RAVI Aug 29 '16 at 22:34
@RAVI, it's python/pandas representation of `4.0` ;) – MaxU - stand with Ukraine Aug 29 '16 at 22:38
Why only 3.4 -> 3.3999999999999999 and not any other values like 2.0 -> 1.9999999999999999?? Also in to_json output it got 3.4 correctly. Is there any issue in to_dict?? – RAVI Aug 29 '16 at 22:49
@RAVI, i don't know when python/pandas decides to represent floats differently. But you can see it with the following example `print(0.1 + 0.2)`. BTW: `df.to_dict('index')` shows floats correctly... – MaxU - stand with Ukraine Aug 29 '16 at 22:57

hpaulj · Accepted Answer · 2016-08-29T23:08:08.380

With your sample file and genfromtxt calls I get 2 arrays:

In [89]: column
Out[89]: 
array(['header0', 'mydate', 'nextdate', 'afterthat'], 
      dtype='<U9')
In [90]: columnData
Out[90]: 
array([['header0', 'header1', 'header2'],
       ['mydate', '3.4', '2.0'],
       ['nextdate', '4', '6'],
       ['afterthat', '7', '8']], 
      dtype='<U9')

Pull out the first row of columnData

In [91]: headers=columnData[0,:]
In [92]: headers
Out[92]: 
array(['header0', 'header1', 'header2'], 
      dtype='<U9')

Now construct a dictionary of dictionaries (I don't need the separate column array):

In [94]: {row[0]: {h:v for h,v in zip(headers, row)} for row in columnData[1:]}
Out[94]: 
{'afterthat': {'header0': 'afterthat', 'header1': '7', 'header2': '8'},
 'mydate': {'header0': 'mydate', 'header1': '3.4', 'header2': '2.0'},
 'nextdate': {'header0': 'nextdate', 'header1': '4', 'header2': '6'}}

refine it a bit:

In [95]: {row[0]: {h:v for h,v in zip(headers[1:], row[1:])} for row in columnData[1:]}
Out[95]: 
{'afterthat': {'header1': '7', 'header2': '8'},
 'mydate': {'header1': '3.4', 'header2': '2.0'},
 'nextdate': {'header1': '4', 'header2': '6'}}

I like dictionary comprehensions!

Your dictionary of lists version:

In [100]: {row[0]:row[1:] for row in columnData[1:].tolist()}
Out[100]: {'afterthat': ['7', '8'], 'mydate': ['3.4', '2.0'], 'nextdate': ['4', '6']}

This worked great with numpy. Thank you very much. I cannot use pandas as new installations are very selective in my case unless proper justification. — user2406718, Aug 29 '16 at 23:10

get column names from numpy genfromtxt in python

2 Answers2