Need help in the following to reduce the execution time

Question

This is just a part of the code, it takes around 5 min for each iteration ( for a single el) to execute.

The length of fieldvalues is in the order of 1e6. I believe that multiple appends are eating away the time, Is there any alternative way for this ?

                PK2=[]
                E_pl=[]
                E_tot=[]            

                for k in range(103,109):

                    PK2_k=[]
                    E_pl_k=[]
                    E_tot_k=[]

                    b=frame.fieldOutputs['SDV'+str(k)]
                    fieldValues=b.values

                    for v in fieldValues:
                         PK2_k.append(v.data)
                    PK2.append(PK2_k)

                    b=frame.fieldOutputs['SDV'+str(k+6)]
                    fieldValues=b.values

                    for v in fieldValues:
                         E_tot_k.append(v.data)             
                    E_tot.append(E_tot_k)

                    b=frame.fieldOutputs['SDV'+str(k+12)]
                    fieldValues=b.values

                    for v in fieldValues:
                         E_pl_k.append(v.data)                       
                    E_pl.append(E_pl_k)

                t3=time.time()
                print stepName,"All PK,E terms written",t3-at,"Sec"

                PK2=np.array(PK2)   
                PK2=np.transpose(PK2)   

                E_tot=np.array(E_tot)   
                E_tot=np.transpose(E_tot)   

                E_pl=np.array(E_pl) 
                E_pl=np.transpose(E_pl)

Thanks in Advance !

What is `frames`? It would be super-useful if you can show us a full working program that we can run. — John Zwinck, Apr 29 '16 at 03:31
How are you generating this data? If you can change that, you might see a considerable speed improvement. Is the `frames` data structure something you can modify? — aghast, Apr 29 '16 at 03:58
The data is queried from another standard commercial software known as Abaqus. I dont think I can modify the structure. Thanks ! — Mechanician, Apr 29 '16 at 04:03
@ John , even if I put up the full working program you will not be able to run it as you require Abaqus and also my output file. — Mechanician, Apr 29 '16 at 04:04

John Zwinck · Answer 1 · 2016-04-29T04:47:16.603

1

Here's a quick cleanup of your code that might help a little, but probably not a ton. For a bigger speedup we need to see what frames contains, and understand what you're really trying to accomplish with the PK2 and other lists (right now you populate them but never use them, so we could optimize them away entirely, but presumably you are doing something with them later which we don't see).

for frame in frames:
    PK2=[]
    E_pl=[]
    E_tot=[]            

    for k in range(103,109):
        b=frame.fieldOutputs['SDV'+str(k)]
        PK2.append([v.data for v in b.values])

        b=frame.fieldOutputs['SDV'+str(k+6)]
        E_tot.append([v.data for v in b.values])

        b=frame.fieldOutputs['SDV'+str(k+12)]
        E_pl.append([v.data for v in b.values])

    t3=time.time()
    print stepName,"All PK,E terms written",t3-at,"Sec"

Edit: You have now shown us that later on you are doing this:

PK2=np.array(PK2)

It's great that you're using NumPy, but you may as well use it right from the start:

def loadarray(frame, offset):
    keys = range(103+offset, 109+offset)
    valueslist = [frame.fieldOutputs['SDV'+str(key)].values for key in keys]

    count = 0
    for values in valueslist:
        count += len(values)

    arr = np.empty(count)

    count = 0
    for values in valueslist:
        arr[count:count+len(values)] = values
        count += len(values)

    return arr.T

for frame in frames:
    PK2 = loadarray(frame, 0)
    E_tot = loadarray(frame, 6)
    E_pl = loadarray(frame, 12)

Now we've avoided creating the temporary lists, and we have PK2 and the other arrays in their final form, transposed and ready to go.

edited Apr 29 '16 at 04:47

answered Apr 29 '16 at 03:37

John Zwinck

239,568
38
324
436

Thanks John ! let me try this and get back ! This is actually an Abaqus- Python interface script. I best believe that frame is an object in the entire class structure defined in Abaqus. frames = myOdb.steps[stepName].frames numFrames = len(frames) I dont know if this helps! – Mechanician Apr 29 '16 at 03:48
@prithivirajanVeerappan: How many elements are typically in `frames`? – John Zwinck Apr 29 '16 at 03:51
numframes turns out to be 11. As i said , the length of field values is ~ 1e6, so its 6*3*1e6 times ! Yes i am trying to accomplish few things with PK2 but it takes few seconds. – Mechanician Apr 29 '16 at 03:54
It would help if you tell us what you are doing with PK2 afterward. How do you use it? – John Zwinck Apr 29 '16 at 04:02
@prithivirajanVeerappan: Thanks. I've expanded my answer to include a solution where you never create the temporary lists, but go straight to the NumPy array construction, which hopefully will speed things up somewhat. – John Zwinck Apr 29 '16 at 04:49
Thanks a lot John. I shall implement this and let you know the results later today! – Mechanician Apr 29 '16 at 13:51
`arr = np.empty(sum(map(len, values)))` can get rid of one for loop. – Burhan Khalid May 03 '16 at 04:43

score 1 · Answer 2 · answered Apr 29 '16 at 03:54

I think you can squeeze it pretty hard into C:

Sdv_list = [ 'SDV'+str(k) for k in range(103,121)]

for frame in frames:

    PK2=[]
    E_pl=[]
    E_tot=[]

    aggs = [PK2]*6 + [E_tot]*6 + [E_pl]*6
    values = [frame.fieldOutputs[key].values for key in Sdv_list]

    for agg,val in zip(aggs, values):
        agg.append([v.data for v in val])

    t3=time.time()
    print stepName,"All PK,E terms written",t3-at,"Sec"

Need help in the following to reduce the execution time

2 Answers2