Search nearest value by lat-lon from list data files by Python loop for and saved in array form

Question

I tried to search nearest values for 34 locations with given lat-lon as reference. Set of data files for 30 days (consists of thousands data) arranged in array forms with nearest lat-lon and desired data. How to collocate the lat-lon to find respective desirable data by looping method and saved to text files like below:

Expected output text files:

    Lat     Lon      Date1  Date2   Date3   ………….   Date30
1   6.483   100.267  0       …..    …..       …..   …..
2   6.333   99.733   0       …..    …..      …..    …..
3   6.2     100.4    1.237   …..    …..      …..    …..
4   5.457   100.388  0       …..    …..      …..    …..
5   5.35    100.4    0       …..    …..      …..    …..
6   5.297   100.272  0       …..    …..      …..    …..
7   4.221   100.701  0       …..    …..      …..    …..
8   4.567   101.1    2.003   …..    …..      ...    …..
9   4.467   101.367  9.161   …..    …..      …..    …..
10  …..      …..    …..             
11  …..      …..    …..             
.   …..      …..    …..             
.   …..      …..    …..             
34  …..      …..    ….. 

Index on the left is not necessary in output

The desired data comes from this files under list files "trmm-0.25"
1. Combine.3B42.20161101.Daily.0.25.dat
2. Combine.3B42.20161102.Daily.0.25.dat
3. Combine.3B42.20161103.Daily.0.25.dat
.
.
30. Combine.3B42.20161130.Daily.0.25.dat

Thousands of desired data for each files are arranged like this :
    Lat      Lon        Desired
   -5.000    95.000     9.420
   -5.000    95.250    13.470
   -5.000    95.500    14.790
   -5.000    95.750    12.840
    ................
    9.750   119.750    17.310
    9.750   120.000    19.650
   10.000    95.000    15.480
   10.000    95.250    15.690

The script I used is as follows:

import numpy as np

#MAIN REFERENCE LAT AND LON
data1=np.loadtxt('location-new.dat')
in_lats1=data1[:,0]
in_lons1=data1[:,1]

#SEARCH CLOSE LOCATION AND FIND RESPECTIVE VALUES IN SEVERAL FILES 
files3=np.loadtxt('trmm-0.25',dtype=str)
for x in range(len(files3)):
    file = files3[x]
    data = np.loadtxt(file)

    lats=data[:,0]
    lons=data[:,1]
    trmm=data[:,2]

    ind=[]
    for i in range(len(data1)):
        dist=(lats-in_lats1[i])**2+(lons-in_lons1[i])**2
        ind.append(np.where(dist==np.min(dist))[0][0])

    lat2=lats[ind]
    lon2=lons[ind]
    trmm2=trmm[ind]

    data3=np.array([in_lats1,in_lons1,trmm2])
    data3=np.transpose(data3)

    np.savetxt('Ground-match-trmm-0.25-loop2.dat',data3,fmt='%9.3f')

main reference of 34 locations are given under location-new.dat as follows:

Lat     Lon
6.483   100.267
6.333   99.733
6.2     100.4
5.457   100.388
5.35    100.4
5.297   100.272
.
.
.

Problem: The output saved text appear only for Date1 not at other dates..

Answer 1

You really have 3-dimensions of data here (lat-lon, date, trmm) so as a first pass work with three nested for loops instead of two. You can decide which will be the outer loop based on your memory/speed requirements, but I'll give an example with lat-lon being the outer loop so you don't have to keep the big date files in memory while you're writing your output file.

In this example I've excluded a lot of the heavy lifting to concentrate on the three nested loop iteration you need to get the data from all the dates.

import numpy as np
result = []
latlons=np.loadtxt('location-new.dat')
for latlon in latlons:
    daily_closest_trmms = []

    daily_trmm_files =np.loadtxt('trmm-0.25',dtype=str)
    for daily_trmm_file in daily_trmm_files:
        trmms = np.loadtxt(daily_trmm_file)

        closest_trmm = 0
        for trmm in trmms:
            pass # calculate closest trmm

        daily_closest_trmms.append(closest_trmm)

    result.append(np.concatenate([latlon, daily_closest_trmms]))

np.savetxt('Ground-match-trmm-0.25-loop2.dat',result,fmt='%9.3f')