Remove semi-unique string in file

Question

I am writing a script to track changes in my routing tables. The script pulls down a snapshot of the routing table every 5 minutes and diffs the table against the version that is 5 minutes old. If there is a deviation in the file, the script generates an email. The problem is each line also has an age counter that represents how old the route is, so this field is always changing. Here is a modified snapshot from a production router:

EDIT Routes with seconds and minutes age have been added for completeness.

B*    0.0.0.0/0 [200/0] via 1.0.10.121, 1d13h
                [200/0] via 1.0.10.111, 1d13h
      10.0.0.0/8 is variably subnetted, 159 subnets, 7 masks
O N1     10.0.0.1/32 [210/21] via 172.18.10.27, 1d13h, Vlan1503
O N1     10.1.0.0/16 [210/21] via 172.18.10.51, 2d17h, Vlan1506
O N1     10.2.0.0/16 [210/21] via 172.18.10.59, 2d17h, Vlan1507
B        10.10.0.0/16 [200/0] via 0.0.0.0, 2d17h, Null0
B        10.186.14.0/24 [200/0] via 1.0.10.124, 1d13h
                        [200/0] via 1.0.10.114, 1d13h
B        10.186.15.0/24 [200/0] via 1.0.10.124, 1d13h
O N2     10.192.32.0/21 [210/20] via 172.18.10.243, 2d17h, Vlan1508
O N2     10.192.40.0/21 [210/20] via 172.18.10.243, 2d17h, Vlan1508
B        10.193.0.0/24 [200/0] via 1.0.10.124, 1d13h
                       [200/0] via 1.0.10.114, 1d13h
C        172.18.10.232/29 is directly connected, Vlan1589
L        172.18.10.233/32 is directly connected, Vlan1589
B        205.1.2.3/27 [200/21] via 1.0.30.5, 2d17h
B     205.3.2.1/24 [200/20] via 1.0.30.5, 2d17h
O N1     2.2.2.2 [110/2] via 192.168.0.2, 00:08:48, Vlan1500
O N1     10.10.83.0/24 [210/21] via 172.18.10.27, 00:00:48, Vlan1503

Note all the "2d17h," "1d13h," etc, these are the age counters that are forever increasing until a device goes down or a link flaps, then they reset. However, the age of the route isn't all that important to me, just that the route is present and the "next hop" (aka via) has not changed. Otherwise I would just "wc -l" the two files and make sure the length was the same.

Is it feasible to strip the age counters from the file and then diff them? If so, what is the best way to remove the counters?

Thanks!

EDIT The desired output would be identical but have the age counters removed, like the following:

B*    0.0.0.0/0 [200/0] via 1.0.10.121,
                [200/0] via 1.0.10.111,
      10.0.0.0/8 is variably subnetted, 159 subnets, 7 masks
O N1     10.0.0.1/32 [210/21] via 172.18.10.27, , Vlan1503
O N1     10.1.0.0/16 [210/21] via 172.18.10.51, , Vlan1506
O N1     10.2.0.0/16 [210/21] via 172.18.10.59, , Vlan1507
B        10.10.0.0/16 [200/0] via 0.0.0.0, , Null0
B        10.186.14.0/24 [200/0] via 1.0.10.124,
                        [200/0] via 1.0.10.114,
B        10.186.15.0/24 [200/0] via 1.0.10.124,
O N2     10.192.32.0/21 [210/20] via 172.18.10.243, , Vlan1508
O N2     10.192.40.0/21 [210/20] via 172.18.10.243, , Vlan1508
B        10.193.0.0/24 [200/0] via 1.0.10.124,
                       [200/0] via 1.0.10.114,
C        172.18.10.232/29 is directly connected, Vlan1589
L        172.18.10.233/32 is directly connected, Vlan1589
B        205.1.2.3/27 [200/21] via 1.0.30.5,
B     205.3.2.1/24 [200/20] via 1.0.30.5,
O N1     2.2.2.2 [110/2] via 192.168.0.2, , Vlan1500
O N1     10.10.83.0/24 [210/21] via 172.18.10.27, , Vlan1503

What I am trying to prevent is two identical tables (same amount of routes, same next hops) failing a diff because one of the routes age counters ticked over in between snapshots of the routing tables. Example:

Script takes snapshot of routing table, all is good, but one route has an age of 00:59:59 (59 minutes and 59 seconds)
5 minutes later script takes a snapshot of the routing table, no routes have been removed or added but the diff between the current table and the 5 minute old table fails because the route that was 00:59:59 is now 1d0h old (I'm assuming the counter would be 1d0h, need to confirm it's not 2d1h)

So the idea is if I strip all the age counters out the diffs wont fail. This might not be best way to accomplish my goal of monitoring routing tables. Unfortunately I don't have a strong background in scripting or programming. However I am open to ideas and all suggestions are welcome. :)

How does it look if the route is less than a day old? Less than an hour old? — Yunnosch, Jul 16 '17 at 07:45
Thanks for the quick reply! Great question, here is output showing a route that is less than an hour old, the format is completely different. Darn, I can't get the output to format well here in a comment, so I am going to edit my post. — FORCED-INDUCTN, Jul 16 '17 at 07:55
Probably a two hour old route has another format, hasn't it? — Yunnosch, Jul 16 '17 at 08:01
Another great question, I apologize for not thinking of this. It appears I made an error, the second route in my edit is 48 seconds old. I believe a route that is an hour old is 01:00:00. I have a lab running to test this but I will have to wait an hour. — FORCED-INDUCTN, Jul 16 '17 at 08:09
Can you show desired output for your sample input? I am not sure that I get exactly what you need. I have a feeling that deleting the ages does not improve the problem. But if that is the desired output, I probably can achieve it. — Yunnosch, Jul 16 '17 at 08:12
The goal is that you add some code of your own to show at least the research effort you made to solve this yourself. — Cyrus, Jul 16 '17 at 08:33
@Yunnosch, I have edited my main post to provide more information. It is very possible I am approaching this the wrong way. Check out my example and let me know what you think, thanks for the help! — FORCED-INDUCTN, Jul 16 '17 at 08:40
@Cyrus, I attempted to better format the data with awk but was not successful, I doubt my failed code would be useful. — FORCED-INDUCTN, Jul 16 '17 at 08:42
@Cyrus I agree with expecting some effort visible in the question. However, OP has spent effort on promptly answering questions in comments here, so I feel the "show effort" is covered. — Yunnosch, Jul 16 '17 at 08:43

Yunnosch · Answer 1 · 2017-07-16T09:06:39.147

0

For deleting the age values from the sample input, you can match the syntax of the whole targeted line instead of the "wildly changing timestamps".
(Nicely phrased summary from your comment, thanks.)

sed -E "s/(^[^.]*(([[:digit:]]+\.){3}[[:digit:]]+\/[[:digit:]]+| +).* via ([[:digit:]]+\.){3}[[:digit:]]+), [[:digit:]:dh]+/\1/"

(GNU sed version 4.2.1, i.e. supporting -E)

-E for convenience, by using extended regular expressions
s/<start><blabla><IPa_or_spaces> via <IPb><, age><blabla><end>/<same_without_age>/,
deletes ages from appropriatly-looking lines

I also deleted the ugly "," coming with an age.
If you want to keep it replace ), -> , ), i.e. para after colon-space instead of before.

I consider this at the upper end of complexity for things to do with sed, it borders on an sed-brain-teaser (which is the actual reason I am doing it, for fun).
Also, any change of syntax might break this.
So have a close look at the answers in awk or perl, which are bount to arrive soon. ;-)
Then use whatever you are most comfortable with, e.g. the language/tool you yourself understand best.

edited Jul 16 '17 at 09:06

answered Jul 16 '17 at 08:42

Yunnosch

26,130
9
42
54

Wow! That is quite a sed line! Very cool, if I get this right you are matching the whole line's syntax instead of just searching for the wildly changing timestamps, this is obviously not something I had thought of. I just tested your solution against a large table of roughly 650 routes and it worked perfectly! Thank you for being so patient and helping me describe my issue better :) – FORCED-INDUCTN Jul 16 '17 at 09:02
There IS a little syntax assumption on the age; it is necessary to not delete the rest of the line. But as you spotted, that syntax assumption is nowhere near precise enough to actually match and delete the age. Nice phrasing of the summary, I used it in the answer now. – Yunnosch Jul 16 '17 at 09:10
You're welcome! Yeah I was beating my head against the wall trying to figure out a way to precisely match that varying time stamp. I like how you matched everything but the time stamp :D that is a very nice reminder to think outside of the box, and some impressive awk wizardry. I'm going to adapt your sed line to another vendors equipment for some practice, thanks again! – FORCED-INDUCTN Jul 16 '17 at 09:25

score 0 · Answer 2 · answered Jul 16 '17 at 13:33

$ sed 's/\(via [0-9.]*, \)[[:alnum:]:]*/\1/' file
B*    0.0.0.0/0 [200/0] via 1.0.10.121,
                [200/0] via 1.0.10.111,
      10.0.0.0/8 is variably subnetted, 159 subnets, 7 masks
O N1     10.0.0.1/32 [210/21] via 172.18.10.27, , Vlan1503
O N1     10.1.0.0/16 [210/21] via 172.18.10.51, , Vlan1506
O N1     10.2.0.0/16 [210/21] via 172.18.10.59, , Vlan1507
B        10.10.0.0/16 [200/0] via 0.0.0.0, , Null0
B        10.186.14.0/24 [200/0] via 1.0.10.124,
                        [200/0] via 1.0.10.114,
B        10.186.15.0/24 [200/0] via 1.0.10.124,
O N2     10.192.32.0/21 [210/20] via 172.18.10.243, , Vlan1508
O N2     10.192.40.0/21 [210/20] via 172.18.10.243, , Vlan1508
B        10.193.0.0/24 [200/0] via 1.0.10.124,
                       [200/0] via 1.0.10.114,
C        172.18.10.232/29 is directly connected, Vlan1589
L        172.18.10.233/32 is directly connected, Vlan1589
B        205.1.2.3/27 [200/21] via 1.0.30.5,
B     205.3.2.1/24 [200/20] via 1.0.30.5,
O N1     2.2.2.2 [110/2] via 192.168.0.2, , Vlan1500
O N1     10.10.83.0/24 [210/21] via 172.18.10.27, , Vlan1503

The above will work in any sed except maybe an old, pre-POSIX one and if that's what you have then just change [:alnum:] to [0-9a-z].

Remove semi-unique string in file

2 Answers2