Need to remove duplicate lines using mawk (specifically)

Question

I have a gawk command that works fine. But I have a machine with mawk installed and when I try to install gawk it complains about broken dependencies. I would like to change this line to mawk syntax.

awk -F '[|]{3}' 'BEGIN {OFS="|||"} !seen[$4]++ {print $4,$7,$3,$5,$6,$8,$9,$10,$11}' $1

Input File: It is a Three Pipe delimited file

A|||B|||C|||D|||E|||F|||G|||H|||I|||J|||K||||L|||M|||N|||O|||P|||Q|||R|||S||||T|||U
1|||2|||3|||4|||5|||6|||7|||8|||9|||10|||11|||12|||13|||14|||15|||16|||17|||18|||19

I receive only the delimiters ||||||||||||||||||||||| without any data in between. — Mallik Kumar, Mar 27 '19 at 06:54
Can you show a real input sample and show us exactly how you ran the command for us to reproduce this behavior? I'm not able to reproduce this — Inian, Mar 27 '19 at 06:55
Ran the command on bash with another file as pipe delimited input. I am not sure if this will help: mawk version 1.3.3 — Mallik Kumar, Mar 27 '19 at 07:02
Wait, is the input file _a pipe delimited_ or _a three pipe delimited file_ (`-F '[|]{3}'`)? — James Brown, Mar 27 '19 at 07:18
Why is your input file mentioned `$1` in the question? `mawk` is a POSIX compliant `awk` which should produce the expected output for your sample. — Inian, Mar 27 '19 at 08:31
Did you check your file for DOS style line endings? What does `file ` return? — Inian, Mar 27 '19 at 08:32
ran dos2unix ..same output. file returns UTF-8 Unicode text, with very long lines — Mallik Kumar, Mar 27 '19 at 08:44
mawk probably has no knowledge yet about `[|]{3}`. Maybe you should write `-F "\\|\\|\\|"` — kvantour, Mar 27 '19 at 08:59
@kvantour Please write it as an answer and I will upvote it!! BTW thanks. — Mallik Kumar, Mar 27 '19 at 09:43

score 1 · Accepted Answer · edited Jun 20 '20 at 09:12

POSIX awk makes use of extended regular expressions which have the possiblility to define character duplication by means of {m,n}

When an ERE matching a single character or an ERE enclosed in parentheses is followed by an interval expression of the format {m}, {m,}, or {m,n}, together with that interval expression it shall match what repeated consecutive occurrences of the ERE would match. The values of m and n are decimal integers in the range 0 <= m<= n<= {RE_DUP_MAX}, where m specifies the exact or minimum number of occurrences and n specifies the maximum number of occurrences. The expression {m} matches exactly m occurrences of the preceding ERE, {m,} matches at least m occurrences, and {m,n} matches any number of occurrences between m and n, inclusive.

_{source: POSIX Regular Expressions}

This method of duplication is unfortunately not supported by mawk as can be read from the manual (Section 3 Regular Expressions).

So instead of defining the field separator FS by means of -F '[|]{3}', you have to make use of -F '[|][|][|]' or -F "\\|\\|\\|"

Need to remove duplicate lines using mawk (specifically)

1 Answers1