0

I import a csv file containing numerical data as a DataFrame then convert it to a 2d array of floats [2678x256]. Here is what I did:

date = 20180710                # date of data acquisition [YYYYMMDD]

s = "{}_l2b.csv".format(date)  # to find the csv file corresponding to the date

TWOb = pd.read_csv(s, sep=';', decimal=',')  # csv data imported as DataFrame

for i in range(0,len(TWOb),1):       #convert string to float
    for j in range(1,len(TWOb[0]),1):
        strg = str(TWOb[i,j])
        res = strg.replace('.', '', 1).isdigit() #True if string only contains numerics
    if  res == True : 
        strg1= float(strg)
        TWOb[i,j] = strg1
    else:                      # if string is not numerical, then:
        strg0 = 0.
        TWOb[i,j] = strg0

print(TWOb)

The output is:

array([[-3.31e-06, 0.0, 3.39e-06, ..., nan, nan, 0.0],
   [1.1800000000000001e-07, 0.0, 1.27e-06, ..., 0.0208, 0.0257,
    0.0279],
   [-2.37e-07, 0.0, 1.27e-06, ..., nan, nan, 0.0],
   ...,
   [4.7299999999999996e-07, 0.0, -8.49e-07, ..., 0.0213, 0.0251,
    0.0288],
   [2.37e-07, 0.0, -2.97e-06, ..., 0.0198, 0.0251, 0.0274],
   [0.0, 0.0, 1.27e-06, ..., nan, nan, 0.0]], dtype=object)

But in some parts, like TWOb[32:34,17:20], I get 

array([[-5.489999999999999e-05, -3.570000000000001e-05, '-8,60E-05'],
   [-5.489999999999999e-05, -9.05e-05, '-1,37E-05']], dtype=object)

with strings while it was supposed to convert them all... and I don't know why. I would like to find every str to try to convert them to floats again (or at least figure out why they were not converted in the first place). The original csv file only contains numbers such as-1,56E-05 or 1,27E-06, and blank cases (filled by nan of float type when importing+converting).

Seda S.
  • 21
  • 5
  • Well, your numbers have a "," as a decimal separator and not a "." That is probably causing the problem. – Randrian Mar 05 '20 at 15:37
  • Have you tried using https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.astype.html first ? (+ replace "," with ".") – RobinFrcd Mar 05 '20 at 15:37
  • Randrain, the separator in the csv file is a comma. When correctly converted to float, it become a dot. The problem is: why some are correctly converted and some others are not, despite the fact that they are initially "written" in the same way in the original file? – Seda S. Mar 05 '20 at 15:41
  • https://stackoverflow.com/questions/23636509/python-convert-string-in-scientific-notation-to-float – cyneo Mar 05 '20 at 15:43

0 Answers0