2

I'm using Pyparsing to parse a log file which has blocks that look like this:

keyName0:                                     foo
keyName1:                                     bar
msgKey [Read]:                                21 FA 00 34
msgKey [Read]:
  MESSAGE 1 of 2
    keyName0:                                 keyValue0
    keyName1:                                 keyValue1
    Flags1:                                   No Flags Set
    Flags1:                                   0
    Flags2:                                   No Flags Set
    Flags2:                                   0
    keyName6:                                 $12AB34CD56EF (123456789)
    keyName7:                                 7
    keyName8:                                 7
    Data [Read]:                              00 01 02 03    04 05 06 07    08 09 10 11    12 13 14 15
                                              20 21 22 23    24 25 26 27    28 29 30 31    32 33 34 35
                                              36 37 38

msgKey [Read]:                                01 02 03 04
msgKey [Read]:
  MESSAGE 2 of 2
    # same structure as message above

keyName3:                                     keyValue3
keyName4 [IN]:                                keyValue4 (123 IN)
keyName4 [OUT]:                               keyValue4 (123 OUT)

I wrote a grammar for the keyName-Value-lines:

key_line = lineEnd + OneOrMore(Word(printables_no_column)).setParseAction(' '.join).setResultsName('keyName') + Suppress(':') \
       + OneOrMore(Word(printables_no_column), stopOn=lineEnd).setParseAction(' '.join).setResultsName('keyValue')

This grammar works fine for the individual lines. Now I tried to use this grammar to describe the grammar of the whole test data:

message = Forward()
key_line = lineEnd + OneOrMore(Word(printables_no_column)).setParseAction(' '.join).setResultsName('keyName') + Suppress(':') \
       + MatchFirst(message, OneOrMore(Word(printables_no_column),stopOn=lineEnd).setParseAction(' '.join).setResultsName('keyValue'))
key_lines = ZeroOrMore(Group(key_line)).setResultsName('keys')
message << Literal('MESSAGE') + number + Literal('of')
           + number.setResultsName('totalMsgs') + key_lines

However, I think that this grammar ends in infinite recursion. I need help to figure out how to use the Forward() recursive grammar properly. Many thanks in advance!

Giuseppe Cianci
  • 407
  • 3
  • 9
  • This isn't really a recursive grammar, just a two-level structure. Don't go down the `Forward` path until you have messages inside messages inside messages. – PaulMcG Jun 07 '18 at 12:29
  • Is keyName1 the number of messages in the block? – PaulMcG Jun 07 '18 at 12:55
  • @PaulMcG The number of messages in the block is given in the fourth line: `MESSAGE number of total_messages`. However, the `number of total_messages` is missing in some blocks of the log, even though there are multiple messages in that block. – Giuseppe Cianci Jun 07 '18 at 16:23

1 Answers1

1

This should move you forward a bit. Still may need to get better structuring overall, but I think the basic bits are here. See embedded comments:

import pyparsing as pp

# your original expression - x.setResultName("x") can now be written just x("x")
# key_line = (lineEnd
#             + OneOrMore(Word(printables_no_column)).setParseAction(' '.join)('keyName')
#             + Suppress(':')
#             + OneOrMore(Word(printables_no_column), stopOn=lineEnd).setParseAction(' '.join)('keyValue'))

# literals in your grammar will be suppressed by default
pp.ParserElement.inlineLiteralsUsing(pp.Suppress)

integer = pp.pyparsing_common.integer
hex_byte = pp.Word(pp.hexnums, exact=2)

# read everything up to ':' -  a little risky to define a Word including spaces, may want to revisit and
# explicitly parse bits, to detect "[IN]" vs "[OUT]", etc.
key_name_expr = pp.Word(pp.printables + " ", excludeChars=':')
key_line = pp.Group(key_name_expr("key_name") + ':'
                    + ~pp.lineEnd()  # make sure key value is on this same line
                    + pp.empty()     # handy trick to advance past white space
                    + pp.restOfLine()('key_value'))

# special key_line to read data bytes
data_body = "Data [Read]:" + pp.OneOrMore(hex_byte)

msg_body = ("msgKey [Read]:" + pp.lineEnd()
            + "MESSAGE" + integer("message_num") + "of" + integer("total_msgs")
            + pp.OneOrMore(pp.Group(key_line)("params*"), stopOn=data_body)
            + data_body("data"))

msg_expr = (pp.OneOrMore(pp.LineStart() + pp.Group(key_line)("params*"), stopOn=msg_body)
            + pp.Optional(pp.Group(msg_body)("body")))

Use searchString to find matching blocks, and dump them out:

for match in msg_expr.searchString(source):
    print(match.dump())
    # some sample code showing how to access parsed data fields
    if match.body:
        print("Msg {message_num}/{total_msgs}".format_map(match.body))
        print(match.body.data)
    print()

Prints (excerpt shown):

[[['keyName1', '2']], [['msgKey [Read]', '21 FA 00 34']], ['\n', 1, 2, [['keyName0', 'keyValue0']], [['keyName1', 'keyValue1']], [['Flags1', 'No Flags Set']], [['Flags1', '0']], [['Flags2', 'No Flags Set']], [['Flags2', '0']], [['keyName6', '$12AB34CD56EF (123456789)']], [['keyName7', '7']], [['keyName8', '7']], '00', '01', '02', '03', '04', '05', '06', '07', '08', '09', '10', '11', '12', '13', '14', '15', '20', '21', '22', '23', '24', '25', '26', '27', '28', '29', '30', '31', '32', '33', '34', '35', '36', '37', '38']]
- body: ['\n', 1, 2, [['keyName0', 'keyValue0']], [['keyName1', 'keyValue1']], [['Flags1', 'No Flags Set']], [['Flags1', '0']], [['Flags2', 'No Flags Set']], [['Flags2', '0']], [['keyName6', '$12AB34CD56EF (123456789)']], [['keyName7', '7']], [['keyName8', '7']], '00', '01', '02', '03', '04', '05', '06', '07', '08', '09', '10', '11', '12', '13', '14', '15', '20', '21', '22', '23', '24', '25', '26', '27', '28', '29', '30', '31', '32', '33', '34', '35', '36', '37', '38']
  - data: ['00', '01', '02', '03', '04', '05', '06', '07', '08', '09', '10', '11', '12', '13', '14', '15', '20', '21', '22', '23', '24', '25', '26', '27', '28', '29', '30', '31', '32', '33', '34', '35', '36', '37', '38']
  - message_num: 1
  - params: [[['keyName0', 'keyValue0']], [['keyName1', 'keyValue1']], [['Flags1', 'No Flags Set']], [['Flags1', '0']], [['Flags2', 'No Flags Set']], [['Flags2', '0']], [['keyName6', '$12AB34CD56EF (123456789)']], [['keyName7', '7']], [['keyName8', '7']]]
    [0]:
      [['keyName0', 'keyValue0']]
      [0]:
        ['keyName0', 'keyValue0']
        - key_name: 'keyName0'
        - key_value: 'keyValue0'
    [1]:
      [['keyName1', 'keyValue1']]
      [0]:
        ['keyName1', 'keyValue1']
        - key_name: 'keyName1'
        - key_value: 'keyValue1'
    [2]:
      [['Flags1', 'No Flags Set']]
      [0]:
        ['Flags1', 'No Flags Set']
        - key_name: 'Flags1'
        - key_value: 'No Flags Set'
    [3]:
      [['Flags1', '0']]
      [0]:
        ['Flags1', '0']
        - key_name: 'Flags1'
        - key_value: '0'
    [4]:
      [['Flags2', 'No Flags Set']]
      [0]:
        ['Flags2', 'No Flags Set']
        - key_name: 'Flags2'
        - key_value: 'No Flags Set'
    [5]:
      [['Flags2', '0']]
      [0]:
        ['Flags2', '0']
        - key_name: 'Flags2'
        - key_value: '0'
    [6]:
      [['keyName6', '$12AB34CD56EF (123456789)']]
      [0]:
        ['keyName6', '$12AB34CD56EF (123456789)']
        - key_name: 'keyName6'
        - key_value: '$12AB34CD56EF (123456789)'
    [7]:
      [['keyName7', '7']]
      [0]:
        ['keyName7', '7']
        - key_name: 'keyName7'
        - key_value: '7'
    [8]:
      [['keyName8', '7']]
      [0]:
        ['keyName8', '7']
        - key_name: 'keyName8'
        - key_value: '7'
  - total_msgs: 2
- params: [[['keyName1', '2']], [['msgKey [Read]', '21 FA 00 34']]]
  [0]:
    [['keyName1', '2']]
    [0]:
      ['keyName1', '2']
      - key_name: 'keyName1'
      - key_value: '2'
  [1]:
    [['msgKey [Read]', '21 FA 00 34']]
    [0]:
      ['msgKey [Read]', '21 FA 00 34']
      - key_name: 'msgKey [Read]'
      - key_value: '21 FA 00 34'
Msg 1/2
['00', '01', '02', '03', '04', '05', '06', '07', '08', '09', '10', '11', '12', '13', '14', '15', '20', '21', '22', '23', '24', '25', '26', '27', '28', '29', '30', '31', '32', '33', '34', '35', '36', '37', '38']
PaulMcG
  • 62,419
  • 16
  • 94
  • 130