0

I have file in which there are 69 columns and I want to plot column 1 vs 0 in python. Column 0 is time stamp and defined as a x in my below code. that one is just read as [nan] all the way. Can anyone help how I can access it as time stamp and convert it into real time parameter.

Output of my original program is as below.

386L, 69L

1.02063350e+01   1.01029230e+01   1.01483550e+01   1.01322510e+01
1.01652060e+01   1.01666750e+01   1.00328040e+01   1.01037690e+01
1.01594840e+01   1.01917720e+01   1.02076490e+01   1.00984500e+01
1.01465400e+01   1.01099130e+01   9.93045400e+00   1.02702020e+01
1.02420290e+01   9.83942200e+00   1.01766670e+01   1.03041800e+01
1.01142430e+01   9.99233500e+00   1.02056930e+01   9.96160800e+00
1.02312710e+01   1.01937070e+01   1.00662410e+01   1.00564220e+01
1.03316840e+01   1.02984290e+01   1.01553350e+01   1.02485920e+01
1.01057070e+01   1.01322900e+01   9.85602100e+00   1.01304120e+01
1.01867200e+01   1.01230980e+01   1.04255890e+01   1.02276980e+01
9.97088100e+00   1.02358880e+01   1.02324460e+01   1.01739110e+01
9.90378000e+00   1.02250190e+01   1.01972960e+01   1.01434230e+01
1.01156340e+01   1.01877680e+01   1.00771640e+01   9.94258300e+00
1.01228480e+01   1.00226400e+01   1.03037610e+01   1.01374190e+01
1.03934040e+01   1.02223120e+01   9.91568800e+00   1.00569950e+01
1.00406810e+01   9.96628000e+00   9.95176400e+00   1.02743280e+01
1.03284240e+01   1.00261550e+01   1.02350400e+01   9.62347500e+00
7.97721500e+00   6.98834500e+00   6.54083700e+00   5.59419300e+00
5.44109600e+00   5.18832000e+00   5.07447900e+00   4.82216500e+00
5.14443900e+00   5.07041500e+00   5.12356600e+00   4.86419400e+00
4.93091800e+00   4.73329300e+00   5.19877800e+00   5.07006600e+00
5.02329300e+00   4.94752100e+00   5.08953600e+00   5.06611700e+00
5.00972200e+00   5.03730200e+00   4.93890900e+00   4.98747800e+00
4.92193600e+00   5.37086000e+00   4.69805300e+00   5.02045900e+00
5.07409300e+00   4.94737800e+00   5.01768700e+00   4.89215900e+00
4.91796700e+00   4.98060300e+00   4.89192100e+00   4.94804300e+00
4.83130500e+00   4.98217200e+00   4.95033600e+00   5.00201600e+00
5.02830800e+00   5.08981000e+00   4.76257000e+00   4.86429500e+00
4.64401200e+00   4.83474300e+00   4.92021900e+00   4.87757000e+00
4.86761000e+00   4.85844700e+00   4.83728900e+00   4.74187300e+00
4.66529000e+00   4.82284800e+00   4.71564600e+00   4.71299600e+00
5.22222600e+00   4.87288500e+00   4.93599900e+00   5.15918100e+00
4.81741600e+00   5.05354700e+00   4.91554200e+00   4.97029600e+00
4.90260000e+00   4.86965600e+00   4.70653400e+00   4.88988400e+00
4.83676100e+00   4.66035100e+00   4.70221100e+00   4.83428200e+00
4.78062500e+00   4.85336800e+00   4.69923700e+00   4.82042900e+00
4.77278600e+00   4.85703000e+00   4.92349300e+00   4.97539500e+00
4.66653900e+00   4.79438100e+00   4.05199500e+00   4.01709300e+00
4.28989800e+00   3.99912900e+00   3.97699200e+00   4.27547500e+00
4.15868000e+00   4.13992000e+00   4.11040000e+00   4.12968500e+00
3.88466100e+00   3.87837800e+00   4.45199600e+00   3.97069900e+00
4.07768100e+00   4.34960200e+00   4.05255100e+00   4.13006600e+00
4.20696700e+00   4.11243100e+00   4.01630000e+00   4.01754900e+00
4.10431500e+00   3.91450600e+00   4.21277800e+00   3.96927900e+00
4.09596500e+00   4.50494600e+00   4.22938300e+00   4.30338000e+00
4.18615500e+00   4.12275400e+00   4.04061600e+00   4.15334000e+00
4.06964500e+00   3.94753000e+00   3.97536300e+00   4.24165000e+00
3.98226700e+00   4.29778300e+00   4.22502600e+00   4.26802800e+00
4.32224600e+00   3.84938100e+00   4.08480200e+00   3.75990800e+00
4.18492200e+00   4.01363700e+00   4.01796300e+00   4.07649600e+00
4.00820700e+00   4.11053300e+00   3.87055100e+00   4.21097700e+00
4.15524400e+00   4.14812500e+00   4.13236500e+00   4.07726200e+00
3.76739800e+00   3.94160800e+00   3.81505400e+00   3.78352000e+00
3.86908200e+00   4.05378300e+00   4.31671500e+00   4.31096900e+00
4.08509900e+00   3.98346500e+00   4.15286100e+00   3.62410400e+00
3.32268500e+00   2.31938000e+00   1.88496600e+00   1.53918800e+00
1.38159400e+00   1.08586400e+00   1.02829900e+00   9.62478000e-01
1.03807300e+00   1.08465700e+00   1.06060300e+00   1.10126200e+00
8.28574000e-01   9.15849000e-01   1.04531400e+00   7.06345000e-01
9.24180000e-01   8.11576000e-01   9.22431000e-01   1.06463300e+00
1.07769300e+00   8.86140000e-01   8.91486000e-01   7.12601000e-01
7.50398000e-01   1.23665800e+00   8.17675000e-01   9.28867000e-01
1.04068000e+00   1.07396100e+00   6.77256000e-01   9.48032000e-01
1.19655800e+00   9.49906000e-01   1.05095600e+00   8.95500000e-01
9.54073000e-01   1.03294700e+00   9.47867000e-01   8.54049000e-01
8.56902000e-01   1.12824500e+00   9.39495000e-01   8.48964000e-01
1.07529300e+00   9.08451000e-01   8.41853000e-01   1.02797300e+00
7.49010000e-01   7.87141000e-01   7.73506000e-01   8.72573000e-01
6.13669000e-01   9.56504000e-01   9.12995000e-01   8.45595000e-01
1.12688400e+00   9.75989000e-01   1.24252300e+00   1.07969800e+00
9.73997000e-01   9.00494000e-01   1.01318800e+00   9.78460000e-01
8.94072000e-01   9.75827000e-01   1.06745400e+00   8.62771000e-01
8.10779000e-01   1.13640000e+00   1.04607500e+00   1.06464800e+00
1.05792800e+00   8.43800000e-01   7.44144000e-01   1.05855100e+00
1.01307500e+00   9.57641000e-01   1.00375700e+00   1.02454600e+00
2.90891000e-01   6.64140000e-02   1.05532000e-01   2.19837000e-01
1.66220000e-02   1.38264000e-01   3.69454000e-01   2.45617000e-01
4.79750000e-02   1.15673000e-01   2.39620000e-01  -6.02350000e-02
2.03631000e-01  -4.06370000e-02   2.96096000e-01   7.09180000e-02
-1.48026000e-01   2.34339000e-01   1.16457000e-01   5.01100000e-02
1.17650000e-01   1.99601000e-01   5.85800000e-03  -6.15620000e-02
-2.64468000e-01   2.79645000e-01   3.86220000e-02   4.73830000e-02
3.71340000e-02   1.15296000e-01   2.40179000e-01   1.63250000e-02
1.51336000e-01   1.13677000e-01   1.42556000e-01   2.41298000e-01
1.30385000e-01   8.48750000e-02   1.59172000e-01   1.71280000e-02
-5.85770000e-02   1.35253000e-01   5.58280000e-02  -4.02310000e-02
7.95880000e-02   6.91060000e-02   3.91030000e-02  -6.27100000e-03
1.84503000e-01   9.53810000e-02   1.53314000e-01   1.08753000e-01
3.14292000e-01   8.03350000e-02   1.11857000e-01   1.48813000e-01
-1.79700000e-03   1.52151000e-01  -5.78250000e-02  -1.23120000e-01
1.84140000e-02   5.37010000e-02   2.08872000e-01   1.80160000e-02
2.40175000e-01   3.48981000e-01   1.06070000e-02   3.37341000e-01
-3.81840000e-02   1.16279000e-01   2.05508000e-01  -9.65380000e-02
1.32069000e-01  -8.47600000e-03   1.84650000e-01   8.12810000e-02
-2.67500000e-02              nan   1.00000000e+01   5.00000000e+00
4.00000000e+00   1.00000000e+00   0.00000000e+00              nan
-7.00000000e-03   9.92500000e-01


10.206335  10.102923  10.148355  10.132251  10.165206  10.166675
10.032804  10.103769  10.159484  10.191772


nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan
nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan
nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan
nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan
nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan
nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan
nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan
nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan
nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan
nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan
nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan
nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan
nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan
nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan
nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan
nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan
nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan
nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan
nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan
nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan
nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan
nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan
nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan
nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan
nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan
nan  nan  nan  nan  nan  nan  nan  nan  nan  nan  nan

2

I am not sure why datetime is shown as nan when I was trying to extract that information. Please help me understand what I did wrong? why I am seeing as nan.

below is the truncated data

9/30/2014 14:13 10.206335

9/30/2014 14:13 10.102923

9/30/2014 14:13 10.148355

9/30/2014 14:13 10.132251

9/30/2014 14:13 10.165206

9/30/2014 14:13 10.166675

9/30/2014 14:13 10.032804

from __future__ import division # avoids problems with integer division
import numpy as np #many numerical routines , like vector, matrix multiplication, FFT
import pylab as p
import scipy as sp
import matplotlib.pyplot as plt
import datetime as dt


data = sp.genfromtxt("C:\\users\mshah\\desktop\\SN 32014-01 manual Stepdown 10 5 4 1 0 Mode 1.TXT", delimiter = "\t")
#print(data[10,1])
print(data.shape)
print(data[:,1])
x = data [:,0]
y = data [:,1]

#z = dt.datetime.strftime(x,"%Y/%m/%d %H:%M")


print(y[:10])
print(x)

nansum = sp.sum(sp.isnan(y))
print nansum
x = x[~sp.isnan(y)]
y = y[~sp.isnan(y)]
plt. scatter(x,y)
plt.title("Step Test Process")
plt.xlabel("time")
plt.ylabel("PPMV")
plt.autoscale(tight=True)
plt.grid()
plt.show()
malay shah
  • 3
  • 4
  • dupe? http://stackoverflow.com/questions/12319969/how-to-use-numpy-genfromtxt-when-first-column-is-string-and-the-remaining-column – Wyrmwood Oct 24 '14 at 22:02
  • 1
    might help if you shared some of the file contents – Padraic Cunningham Oct 24 '14 at 22:27
  • @PadraicCunningham. I added the specifics. the date and time stamp are read as nan when I run print(data[:1]). I dont know how I can share my complete file with you, But I can. – malay shah Oct 24 '14 at 22:48

2 Answers2

0

To plot the first two tab-separated columns where the first column is a date, using pandas:

from datetime import datetime
import matplotlib.pyplot as plt
import pandas as pd

df = pd.read_csv('input.txt',
        sep='\t', parse_dates=[0], header=None, index_col=0, usecols=[0,1],
        date_parser=lambda s: datetime.strptime(s, '%m/%d/%Y %H:%M %S.%f'))

df.plot()
plt.show()
jfs
  • 399,953
  • 195
  • 994
  • 1,670
  • Thanks for your input. However, I am finding below errors. I dont know what is wrong but it seams like it it has issues with parsers.py. – malay shah Oct 27 '14 at 17:01
  • date_parser=lambda s: dt.strptime(s, '%m/%d/%Y %H:%M')) File "C:\Users\mshah\AppData\Local\Continuum\Anaconda\lib\site-packages\pandas\io\parsers.py", line 420, in parser_f return _read(filepath_or_buffer, kwds) File "C:\Users\mshah\AppData\Local\Continuum\Anaconda\lib\site-packages\pandas\io\parsers.py", line 225, in _read return parser.read() – malay shah Oct 27 '14 at 17:02
  • @malayshah: don't put the traceback in the comments, update your question instead (try to create a minimal `input.txt` that still leads to the error and add it verbatim to the question too). I've tested the code using lines like `'9/30/2014 14:13 10.032804\t1\t...'` -- it works. – jfs Oct 27 '14 at 19:26
0

Try adding this after your import statements:

from matplotlib.dates import date2num, MinuteLocator, DateFormatter

def datetime_converter(date_string):
    return date2num(dt.datetime.strptime(date_string, '%m/%d/%Y %H:%M'))

Then modify your call to genfromtxt to use the 'converters' argument,

data = sp.genfromtxt("your_data_file.txt",
                     delimiter = "\t",
                     converters={0: datetime_converter}
                     )

The one issue here is that the data as supplied in this webpage is delimited by spaces, not tabs. As long as the date and time columns are separated by spaces and the data (third column) is separated by a tab (you use the tab delimiter so I assume your data file has tabs in it somewhere) this will work.

If the date and time columns are separated by the same delimiter as your other columns you could parse them separately and combine them after the fact, e.g.:

def date_converter(date_string):
    return date2num(dt.datetime.strptime(date_string, '%m/%d/%Y'))

def time_converter(time_string):
    h,m = time_string.split(':')
    return (int(h) + int(m) / 60.) / 24

data = sp.genfromtxt("txtfile.txt",
                     delimiter = " ",
                     converters={0: date_converter,
                                 1: time_converter}
                     )

x = data[:,0] + data[:,1]
y = data[:,2]

Plotting with matplotlib numeric dates

Before your ax.scatter call, create an axes object that you can work with,

ax = plt.axes()
ax.scatter(x,y)

Then at the end of your script, you can format the xaxis of this axes using the DateLocator and DateFormatter objects (see import dates... statement above),

ax.xaxis.set_major_locator(MinuteLocator())
ax.xaxis.set_major_formatter(DateFormatter('%H:%M'))

Here I have used the MinuteLocator, but dates has several more tickers that can be used.

Hopefully that's more helpful.

farenorth
  • 10,165
  • 2
  • 39
  • 45
  • between date time its a space as a delimiter but between the datetime and reading its a tab. Is there something I can do to use 2 delimiter to read? – malay shah Oct 28 '14 at 16:52
  • If that is the case, then I think the first half of my answer should work for you (I edited because I had named the converter inconsistently). – farenorth Oct 28 '14 at 16:58
  • I found that type of the data is not string its numpy.float64. Is there a way to convert that to data format? – malay shah Oct 28 '14 at 23:17
  • Yes, this is a numpy.float64, I am using [MatPlotLib's numeric date format](http://matplotlib.org/api/dates_api.html#matplotlib.dates.date2num). I assume you mean *date format*? If so, what *date format* do you want? If you are going to plot with MPL, you probably want these numeric dates. I updated my answer to show you how to use date tickers and formatters. – farenorth Oct 28 '14 at 23:49
  • If you want to get back to datetime objects, you could do `x = num2date(x)`. – farenorth Oct 28 '14 at 23:58