Generate benchmark table

Question

I have generated benchmarks for comparing two approaches taken for scaling video files using ffmpeg tool.

The benchmarks are logged in this format :

x.mp4 Output_Resolution : 10 p

Parameter1 : a

Method : A

real    0m5.788s
user    0m16.112s
sys     0m0.313s

Method : B, ParameterB1 : b11

ParameterB2 : b21

real    0m6.637s
user    0m16.618s
sys     0m0.720s

ParameterB2 : b22

real    0m5.486s
user    0m17.570s
sys     0m0.568s

ParameterB2 : b23

real    0m5.232s
user    0m18.212s
sys     0m0.718s

Method : B, ParameterB1 : b12

ParameterB2 : b21

real    0m6.398s
user    0m16.790s
sys     0m0.528s

ParameterB2 : b22

real    0m5.449s
user    0m17.229s
sys     0m0.533s

ParameterB2 : b23

real    0m5.275s
user    0m18.411s
sys     0m0.522s

##################################################################################################################

Parameter1 : b

Method : A

real    0m5.927s
user    0m16.451s
sys     0m0.308s

Method : B, ParameterB1 : b11

ParameterB2 : b21

real    0m6.685s
user    0m17.044s
sys     0m0.597s

ParameterB2 : b22

real    0m5.942s
user    0m18.971s
sys     0m0.804s

ParameterB2 : b23

real    0m6.119s
user    0m20.869s
sys     0m0.792s

.
.
.

There are two methods (A and B). Method A and B share Parameter1 which can take values a,b,c.... Method B has other parameters B1 and B2. ParameterB1 and ParameterB2 take values b11,b12,b13... and b21,b22,b23... respectively. A line separator (which consists of multiple #) is used to separate the measurements for different values of Parameter1.

I would like to view the benchmarks in tabular format.

+--------+---------------------------------------+----------------+----------------+----------------+
| Method |                                       | Parameter1 (a) | Parameter1 (b) | Parameter1 (c) |
+--------+---------------------------------------+----------------+----------------+----------------+
|    A   |                   NA                  | 4.03s          | 3.23s          | 1.4s           |
+--------+-------------------+-------------------+----------------+----------------+----------------+
|    B   | ParameterB1 (b11) | ParameterB2 (b21) | .              |                |                |
|        |                   +-------------------+----------------+----------------+----------------+
|        |                   | ParameterB2 (b22) | .              |                |                |
|        |                   +-------------------+----------------+----------------+----------------+
|        |                   | ParameterB2 (b23) | .              |                |                |
|        +-------------------+-------------------+----------------+----------------+----------------+
|        | ParameterB1 (b12) | ParameterB2 (b21) | .              |                |                |
|        |                   +-------------------+----------------+----------------+----------------+
|        |                   | ParameterB2 (b22) | .              |                |                |
|        |                   +-------------------+----------------+----------------+----------------+
|        |                   | ParameterB2 (b23) | .              |                |                |
|        +-------------------+-------------------+----------------+----------------+----------------+
|        | ParameterB1 (b12) | ParameterB2 (b21) | .              |                |                |
|        |                   +-------------------+----------------+----------------+----------------+
|        |                   | ParameterB2 (b22) | .              |                |                |
|        |                   +-------------------+----------------+----------------+----------------+
|        |                   | ParameterB2 (b23) | .              |                |                |
+--------+-------------------+-------------------+----------------+----------------+----------------+

The cell values consists of the real time values in seconds (real 0m6.119s).

How can I generate such a table using python?

I have written a "not-so-efficient" python script with the help from a previous answer from a similar question I asked a few months ago.

import pprint

def gettime(x):
    m,s = map(float,x[:-1].split('m'))
    return 60 * m + s

with open("log") as fp:
    lines = fp.read().splitlines()

idx = 0
A = {}
B = {}

while idx < len(lines):
    if "Parameter1" in lines[idx]:
        Parameter1 = lines[idx].split(' ')[-1]
        temp1 = {}
        idx += 2
        if "A" in lines[idx]:
            idx += 2
            A[Parameter1] = gettime(lines[idx].split('\t')[-1])
            while idx < len(lines):
                if "B" in lines[idx]:
                    ParameterB1 = lines[idx].split(' ')[-1]
                    temp2 = {}
                    idx += 1
                    while idx < len(lines):
                        if "ParameterB2" in lines[idx]:
                            ParameterB2 = lines[idx].split(' ')[-1]
                            idx += 2
                            temp2[ParameterB2] = gettime(lines[idx].split('\t')[-1])
                        elif "#" in lines[idx] or "B" in lines[idx]:
                            break
                        idx += 1
                    temp1[ParameterB1] = temp2
                elif "#" in lines[idx]:
                    B[Parameter1] = temp1
                    break
                else:
                    idx += 1
    else:
        idx += 1
        
print("A")
print(A)

pp = pprint.PrettyPrinter(sort_dicts = False, depth = 4)
print("B")
pp.pprint(B)

This script parses the log and stores the measurements obtained for respective methods and parameters in a dictionary.

Example output from the script :

A
{'a': 4.03, 'b': 3.23, 'c': 1.4}
B
{'a': {'b21': {'b11': 0.0, 'b12': 0.0, 'b13': 0.0},
       'b22': {'b11': 0.0, 'b12': 0.0, 'b13': 0.0},
       'b23': {'b11': 0.0, 'b12': 0.0, 'b13': 0.0}},
 'b': {'b21': {'b11': 0.0, 'b12': 0.0, 'b13': 0.0},
       'b22': {'b11': 0.0, 'b12': 0.0, 'b13': 0.0},
       'b23': {'b11': 0.0, 'b12': 0.0, 'b13': 0.0}},
 'c': {'b21': {'b11': 0.0, 'b12': 0.0, 'b13': 0.0},
       'b22': {'b11': 0.0, 'b12': 0.0, 'b13': 0.0},
       'b23': {'b11': 0.0, 'b12': 0.0, 'b13': 0.0}}}

How can I print this in tabular format as described above?

Saurabh P Bhandari · Accepted Answer · 2020-07-22T02:14:11.587

Extended the python script (in the question) further to represent the the data stored in the dictionaries in tabular format using pretty table.

import pprint
import io
from prettytable import PrettyTable 
# install PTable package

def gettime(x):
    m,s = map(float,x[:-1].split('m'))
    return 60 * m + s

with open("log") as fp:
    lines = fp.read().splitlines()

idx = 0
A = {}
B = {}
Parameter1_list = []
ParameterB1_list = []
ParameterB2_list = []

while idx < len(lines):
    if "Parameter1" in lines[idx]:
        Parameter1 = lines[idx].split(' ')[-1]
        Parameter1_list.append(Parameter1)
        temp1 = {}
        idx += 2
        if "A" in lines[idx]:
            idx += 2
            A[Parameter1] = gettime(lines[idx].split('\t')[-1])
            while idx < len(lines):
                if "B" in lines[idx]:
                    ParameterB1 = lines[idx].split(' ')[-1]
                    ParameterB1_list.append(ParameterB1)
                    temp2 = {}
                    idx += 1
                    while idx < len(lines):
                        if "ParameterB2" in lines[idx]:
                            ParameterB2 = lines[idx].split(' ')[-1]
                            ParameterB2_list.append(ParameterB2)
                            idx += 2
                            temp2[ParameterB2] = gettime(lines[idx].split('\t')[-1])
                        elif "#" in lines[idx] or "B" in lines[idx]:
                            break
                        idx += 1
                    temp1[ParameterB1] = temp2
                elif "#" in lines[idx]:
                    B[Parameter1] = temp1
                    break
                else:
                    idx += 1
    elif ".mp4" in lines[idx]:
        title = lines[idx]
        idx += 1
    else:
        idx += 1

#print("A")
#print(A)

#pp = pprint.PrettyPrinter(sort_dicts=False,depth=4)
#print("B")
#pp.pprint(B)

Parameter1 = list(dict.fromkeys(Parameter1_list))
ParameterB1 = list(dict.fromkeys(ParameterB1_list))
ParameterB2 = list(dict.fromkeys(ParameterB2_list))

t1 = PrettyTable(['Method','ParameterB1','ParameterB2'])
t2 = PrettyTable(Parameter1)

t1.title = title
t2.title = "Parameter1"

t1.add_row(['A','NA','NA'])
t2.add_row(A.values())

for d in ParameterB1:
    for c in ParameterB2:
        values = []
        for e in Parameter1:
            values.append(B[e][d][c])
        t1.add_row(['B',d,c])
        t2.add_row(values)

o1 = io.StringIO(t1.get_string())
o2 = io.StringIO(t2.get_string())

with open(0,"w") as f1, open('result.txt',"w") as f2:
    for x,y in zip(o1,o2):
        f1.write(x.strip()[:-1] + y.strip() + "\n")
        f2.write(x.strip()[:-1] + y.strip() + "\n")

This writes the table to a file (result.txt) and stdout.

Output :

+------------------------------------+-------------------+
|   x.mp4  Output Resolution : 10p   |     Parameter1    |
+--------+-------------+-------------+------+------+-----+
| Method | ParameterB1 | ParameterB2 |  a   |  b   |  c  |
+--------+-------------+-------------+------+------+-----+
|   A    |      NA     |      NA     | 4.03 | 3.23 | 1.4 |
|   B    |     b11     |     b21     | 0.0  | 0.0  | 0.0 |
|   B    |     b11     |     b22     | 0.0  | 0.0  | 0.0 |
|   B    |     b11     |     b23     | 0.0  | 0.0  | 0.0 |
|   B    |     b12     |     b21     | 0.0  | 0.0  | 0.0 |
|   B    |     b12     |     b22     | 0.0  | 0.0  | 0.0 |
|   B    |     b12     |     b23     | 0.0  | 0.0  | 0.0 |
|   B    |     b13     |     b21     | 0.0  | 0.0  | 0.0 |
|   B    |     b13     |     b22     | 0.0  | 0.0  | 0.0 |
|   B    |     b13     |     b23     | 0.0  | 0.0  | 0.0 |
+--------+-------------+-------------+------+------+-----+

This is the closest I could represent the data in tabular format that I have described in the question.

Generate benchmark table

1 Answers1