0

I'm reading a txt file for calculating the skewT routine in MetPy, however my negative numbers are not read.

According to the tutorial I tried to read a txt file, I modified the original command a little bit

col_names = ['pressure', 'height', 'temperature', 'dewpoint', 'direction', 'speed']
pd.read_fwf('addbna.txt', usecols=[0, 1, 2, 3, 6, 7], names=col_names)

The first rows are red without problem, the issues I have start when there are negative numbers, the minus sign is totally ignored.

These are the files I'm using

https://drive.google.com/open?id=1FOGDNk9fkUooTT2NJCQgpe_12l3sprSl https://drive.google.com/open?id=1aVKeokDOW01Ol8l0UubhTSjqjN6gTCCC

Luis
  • 1
  • 2
  • You need to post the first few lines of your file, `read_fwf` is for reading fixed width files, so is your text file comma separated, tab, fixed width? etc.. – EdChum Jan 08 '19 at 15:55
  • The file is fixed width, however I can't post it adequately – Luis Jan 08 '19 at 20:48
  • I feel like there's a problem with the data file somewhere. Can you upload the file or point to somewhere we can get a sample file? I don't think anyone will be able to help without looking at the data. – DopplerShift Jan 16 '19 at 21:47

2 Answers2

1

The pd.read_fwf() function has the keyword option infer_nrows, which allows you to instruct pd.read to look further down your DataFrame to infer how wide the column widths should be. I had to change mine to 1000 from the default of 100 to get it to find the first instance of negative values in the temperatures on our soundings (recording at 1-sec resolution).

My final read-in instruction became:

df = pd.read_fwf('some_sounding_data.txt', header=0, skiprows=(1,2), infer_nrows=1000)
Shaido
  • 27,497
  • 23
  • 70
  • 73
MadHadders
  • 11
  • 2
0

A little late here, but I am having the exact same issue.

I suspect it's to do with the temperature column not being a fixed width. My dataset begins with numbers like 28.4, 27.6, etc (width of 4), but once it eventually runs below -10.0 (width of 5) the minus sign does not get included in the dataframe temperature column.

I found a hacky way around it by editing the text file directly and adding a "0" in front of the first temperature datum to pad it out into a width of 5 (28.4 became "028.4"). I think if this is the first row read by read_fwf() it assumes the remainder of the column is width 5 and the minus signs won't be ignored.

Hopefully a bright mind out there has found a better solution. I have had no issues plotting skewT's from csv files, so if you can convert your file somehow this could be another way around the issue.

prozac
  • 11
  • 4
  • In my case what messed up everything was the number of rows. The program doesn´t need a large amount of data (I was using more than 200 rows), so, when I reduced to less than 30 everything worked fine – Luis May 18 '19 at 19:32