Use a captured value as row identifier

Question

I need to parse this raw data in order to process it:

Port        Align-Err     FCS-Err    Xmit-Err     Rcv-Err  UnderSize  OutDiscards
Gi0/1               1           2           3           4          5            6
Gi0/2               11          12          13          14         15           16


Port      Single-Col  Multi-Col   Late-Col  Excess-Col  Carri-Sen      Runts     Giants
Gi0/1              1          2          3           4          5          6          7
Gi0/2              111        122        133         144        155        166        177

To do this, I'm using TextFSM.

I would want this output:

['Gi0/1', '1', '2', '3', '4', '5', '6', '1', '2', '3', '4', '5', '6', '7']
['Gi0/2', '11', '12', '13', '14', '15', '16', '111', '112', '113', '114', '115', '116', '117']

The first template I wrote is this one:

Value PORT (\S+(/\d+)?)
Value ALIGNERR (\d+)
Value FCSERR (\d+)
Value XMITERR (\d+)
Value RCVERR (\d+)
Value UNDERSIZE (\d+)
Value OUTDISCARDS (\d+)
Value SINGLECOL (\d+)
Value MULTICOL (\d+)
Value LATECOL (\d+)
Value EXCESSCOL (\d+)
Value CARRISEN (\d+)
Value RUNTS (\d+)
Value GIANTS (\d+)

Start
  ^Port\s+Align-Err.* -> FIRST
  ^Port\s+Single-Col.* -> SECOND

FIRST
  ^${PORT}\s+${ALIGNERR}\s+${FCSERR}\s+${XMITERR}\s+${RCVERR}\s+${UNDERSIZE}\s+${OUTDISCARDS} -> Continue.Record

SECOND
  ^${PORT}\s+${SINGLECOL}\s+${MULTICOL}\s+${LATECOL}\s+${EXCESSCOL}\s+${CARRISEN}\s+${RUNTS}\s+${GIANTS} -> Record

However, the output is not right:

['Gi0/1', '1', '2', '3', '4', '5', '6', '', '', '', '', '', '', '']
['Gi0/2', '11', '12', '13', '14', '15', '16', '', '', '', '', '', '', '']
['Gi0/1', '1', '2', '3', '4', '5', '6', '', '', '', '', '', '', '']
['Gi0/2', '111', '122', '133', '144', '155', '166', '', '', '', '', '', '', '']

I found a post on the forum giving a solution in pure Regex: TextFSM logic - Avoid capturing same data twice

When I adapt it to my needs, I have a match for what I need: https://regex101.com/r/DY0Meb/6

However, I'm unable to translate it in a TextFSM template, it fails. Here is my template:

Value PORT (\S+(/\d+)?)
Value ALIGNERR (\d+)
Value FCSERR (\d+)
Value XMITERR (\d+)
Value RCVERR (\d+)
Value UNDERSIZE (\d+)
Value OUTDISCARDS (\d+)
Value SINGLECOL (\d+)
Value MULTICOL (\d+)
Value LATECOL (\d+)
Value EXCESSCOL (\d+)
Value CARRISEN (\d+)
Value RUNTS (\d+)
Value GIANTS (\d+)

Start
  ^${PORT}\s+${ALIGNERR}\s+${FCSERR}\s+${XMITERR}\s+${RCVERR}\s+${UNDERSIZE}\s+${OUTDISCARDS}(?=.*\1\s+${SINGLECOL}\s+${MULTICOL}\s+${LATECOL}\s+${EXCESSCOL}\s+${CARRISEN}\s+${RUNTS}\s+${GIANTS}) -> Record

Any clue about why I don't have any matches? I'm a beginner in Regex, and I can't seem to find the solution...

Any help would be very welcome :). Thanks in advance!

Do you have to use `TextFSM`? Or any other `python` solution would suffice? — n1colas.m, May 24 '20 at 00:02
Hi, sorry for the delay ! I posted my solution as an answer, too long for a comment :). — DrySaucisse, May 29 '20 at 11:02

score 2 · Answer 1 · answered Jul 27 '20 at 10:55

There is a bit simpler way exists:

from ttp import ttp
import pprint

data = """
Port        Align-Err     FCS-Err    Xmit-Err     Rcv-Err  UnderSize  OutDiscards
Gi0/1               1           2           3           4          5            6
Gi0/2               11          12          13          14         15           16

Port      Single-Col  Multi-Col   Late-Col  Excess-Col  Carri-Sen      Runts     Giants
Gi0/1              1          2          3           4          5          6          7
Gi0/2              111        122        133         144        155        166        177
"""

template = """
<group name="{{ interface }}*">
{{ interface | exclude("Port") }}   {{ counters | ROW | resub("\s+", ",", 200) | split(",") }}
</group>
"""

parser = ttp(data, template)
parser.parse()
res = parser.result()[0][0]

# res is this now:
# {'Gi0/1': [{'counters': ['1', '2', '3', '4', '5', '6']},
#            {'counters': ['1', '2', '3', '4', '5', '6', '7']}],
#  'Gi0/2': [{'counters': ['11', '12', '13', '14', '15', '16']},
#            {'counters': ['111', '122', '133', '144', '155', '166', '177']}]}

ret = []
for interface, counters in res.items():
    ret.append([interface])
    for i in counters:
        ret[-1] += list(i.values())[0]
    
pprint.pprint(ret, width=100)

# will print:
# [['Gi0/1', '1', '2', '3', '4', '5', '6', '1', '2', '3', '4', '5', '6', '7'],
#  ['Gi0/2', '11', '12', '13', '14', '15', '16', '111', '122', '133', '144', '155', '166', '177']]

TTP - is a library I created to make solutions for above tasks to look simpler.

This looks really really good. Thanks, I'll give it a try ! – DrySaucisse Jul 30 '20 at 11:37 — DrySaucisse, Jul 30 '20 at 11:37

score 0 · Accepted Answer · answered May 29 '20 at 11:11

I finally managed to do what I want.

My team wished to use Ansible for the formatting, so I had to improvise a bit.

I used ntc-ansible for that.

With the help of members of the NTC Slack, I finally got it working. Here's what I came up with:

A functionality that is very poorly documented on the TextFSM repo is that you can, in an index file, combine two templates that share a common "Key" attribute.

So I created two templates:

Value Key PORT (\S+(/\d+)+) 
Value ALIGNERR (\d+) 
Value FCSERR (\d+) 
Value XMITERR (\d+) 
Value RCVERR (\d+) 
Value UNDERSIZE (\d+) 
Value OUTDISCARDS (\d+) 
Start
  ^Port\s+Align-Err.* -> Begin
Begin 
 ^${PORT}\s+${ALIGNERR}\s+${FCSERR}\s+${XMITERR}\s+${RCVERR}\s+${UNDERSIZE}\s+${OUTDISCARDS} -> Record
      ^Port\s+Single-Col.* -> End

And:

Value Key PORT (\S+(/\d+)+) 
Value SINGLECOL (\d+) 
Value MULTICOL (\d+) 
Value LATECOL (\d+) 
Value EXCESSCOL (\d+) 
Value CARRISEN (\d+) 
Value RUNTS (\d+) 
Value GIANTS (\d+)
Start
  ^Port\s+Single-Col.* -> Begin
Begin
^${PORT}\s+${SINGLECOL}\s+${MULTICOL}\s+${LATECOL}\s+${EXCESSCOL}\s+${CARRISEN}\s+${RUNTS}\s+${GIANTS} -> Record

Then, I created an index file containing this:

Template, Hostname, Vendor, Command
show_int_counter_errors1.template:show_int_counter_errors2.template, .*, cisco_ios, sh[[ow]] int[[erfaces]] cou[[nter]] er[[rors]]

You can test it in Python with this little script:

import textfsm
import sys
from textfsm import clitable
# Define Input Data
input_data = sys.stdin.read()
# Initialise CliTable with the content of the 'index' file.
cli_table = clitable.CliTable('index', '.')
# Firstly we will use attributes to match a 'show version' command on a Cisco device.
attributes = {'Command': sys.argv[1], 'Vendor': 'cisco_ios'}
# Parse Data
cli_table.ParseCmd(input_data, attributes)
print(cli_table)

To launch it, just use this command:

python3 test_table.py 'show interface counter errors' < show_int_counter_errors.txt

To use it in Ansible, after installing ntc-ansible, create a 'templates' directory, put the index file and the template files in it, and specify the directory path in the playbook:

- hosts: all
  connection: local
  gather_facts: no

  tasks:
    - name: Formatting the errors
      ntc_show_command:
        connection: ssh
        platform: cisco_ios
        command: 'show interfaces counter errors'
        use_templates: True
        template_dir: 'PATH/TO/TEMPLATES/DIRECTORY'
        host: "{{ ansible_host }}"
        username: "{{ ansible_user }}"
        password: "{{ ansible_password }}"
 register: interface_errors

    - name: Display the registered variable
      debug:
        var: interface_errors

Hope this can help anybody :).

Use a captured value as row identifier

2 Answers2