Pull out Java error stacks from log files

Question

I have a Java application that, when erroring out, writes an error stack similar to the below for each error.

<Errors>
    <Error ErrorCode="Code" ErrorDescription="Description" ErrorInfo="" ErrorId="ID">
        <Attribute Name="ErrorCode" Value="Code"/>
        <Attribute Name="ErrorDescription" Value="Description"/>
        <Attribute Name="Key" Value="Key"/>
        <Attribute Name="Number" Value="Number"/>
        <Attribute Name="ErrorId" Value="ID"/>
        <Attribute Name="UserId" Value="User"/>
        <Attribute Name="ProgId" Value="Prog"/>
        <Stack>typical Java stack</Stack>
    </Error>
    <Error>
      Similar info to the above
    </Error>
</Errors>

I wrote a Java log parser to go through the log files and gather information about such errors and while it does work, it is slow and inefficient, especially for log files in the hundreds of megabytes. I just basically use string manipulation to detect where the start/end tags are and tally them up.

Is there a way (either via Unix grep, Python, or Java) to efficiently extract the errors and get a count of the number of times each one happens? The entire log file is not XML so I cannot use an XML parser or Xpath. Another problem I am facing is that sometimes the end of an error might roll into another file so the current file might not have the entire stack as above.

EDIT 1:

Here is what I currently have (relevant portions only to save space).

//Parse files
for (File f : allFiles) {
   System.out.println("Parsing: " + f.getAbsolutePath());
   BufferedReader br = new BufferedReader(new FileReader(f));
   String line = "";
   String fullErrorStack = "";
   while ((line = br.readLine()) != null) {     
      if (line.contains("<Errors>")) {
         fullErrorStack = line;
         while (!line.contains("</Errors>")) {
            line = br.readLine();
            try {
               fullErrorStack = fullErrorStack + line.trim() + " ";
            } catch (NullPointerException e) {
               //End of file but end of error stack is in another file.
               fullErrorStack = fullErrorStack + "</Stack></Error></Errors> ";
               break;
            }
         }
         String errorCode = fullErrorStack.substring(fullErrorStack.indexOf("ErrorCode=\"") + "ErrorCode=\"".length(), fullErrorStack.indexOf("\" ", fullErrorStack.indexOf("ErrorCode=\"")));
         String errorDescription = fullErrorStack.substring(fullErrorStack.indexOf("ErrorDescription=\"") + "ErrorDescription=\"".length(), fullErrorStack.indexOf("\" ", fullErrorStack.indexOf("ErrorDescription=\"")));
         String errorStack = fullErrorStack.substring(fullErrorStack.indexOf("<Stack>") + "<Stack>".length(), fullErrorStack.indexOf("</Stack>", fullErrorStack.indexOf("<Stack>")));
         apiErrors.add(f.getAbsolutePath() + splitter + errorCode + ": " + errorDescription + splitter + errorStack.trim());
         fullErrorStack = "";
      }
   }
}


Set<String> uniqueApiErrors = new HashSet<String>(apiErrors);
for (String uniqueApiError : uniqueApiErrors) {
    apiErrorsUnique.add(uniqueApiError + splitter + Collections.frequency(apiErrors, uniqueApiError));
}
Collections.sort(apiErrorsUnique);

EDIT 2:

Sorry for forgetting to mention the desired output. Something like the below would be ideal.

Count, ErrorCode, ErrorDescription, List of files it occurs in (if possible)

Answer 1

Well, it's not technically grep, but if you're open to using other standard UNIX-esque commands, here's a one-liner that could do the job, and it should be fast (would be interested to see results on your dataset, actually):

sed -r -e '/Errors/,/<\/Errors>/!d' *.log -ne 's/.*<Error\s+ErrorCode="([^"]*)"\s+ErrorDescription="([^"]*)".*$/\1: \2/p' | sort | uniq -c | sort -nr

Assuming they're in date order, the *.log glob will also solve the problem of logs rolling (adjust to match your log naming, of course).

Sample output

From my (dubious) test data based on yours:

 10 SomeOtherCode: This extended description
  4 Code: Description
  3 ReallyBadCode: Disaster Description

Brief Explanation

1. Use sed to print only between selected addresses (lines, here)
2. Use sed again to filter these with a regex, replacing the header line with a composed unique-enough error strings (including description), similar to your Java (or at least what we can see of it)
3. Sort and count these unique strings
4. Present in descending order of frequency

Answer 2

Given your updated question:

$ cat tst.awk
BEGIN{ OFS="," }
match($0,/\s+*<Error ErrorCode="([^"]+)" ErrorDescription="([^"]+)".*/,a) {
    code = a[1]
    desc[code] = a[2]
    count[code]++
    files[code][FILENAME]
}
END {
    print "Count", "ErrorCode", "ErrorDescription", "List of files it occurs in"
    for (code in desc) {
        fnames = ""
        for (fname in files[code]) {
            fnames = (fnames ? fnames " " : "") fname
        }
        print count[code], code, desc[code], fnames
    }
}
$
$ awk -f tst.awk file
Count,ErrorCode,ErrorDescription,List of files it occurs in
1,Code,Description,file

It still requires gawk 4.* for the 3rd arg to match() and 2D arrays but again that's easily worked around in any awk.

Per request in the comments here's a non-gawk version:

$ cat tst.awk
BEGIN{ OFS="," }
/[[:space:]]+*<Error / {
    split("",n2v)
    while ( match($0,/[^[:space:]]+="[^"]+/) ) {
        name = value = substr($0,RSTART,RLENGTH)
        sub(/=.*/,"",name)
        sub(/^[^=]+="/,"",value)
        $0 = substr($0,RSTART+RLENGTH)
        n2v[name] = value
    }
    code = n2v["ErrorCode"]
    desc[code] = n2v["ErrorDescription"]
    count[code]++
    if (!seen[code,FILENAME]++) {
        fnames[code] = (code in fnames ? fnames[code] " " : "") FILENAME
    }
}
END {
    print "Count", "ErrorCode", "ErrorDescription", "List of files it occurs in"
    for (code in desc) {
        print count[code], code, desc[code], fnames[code]
    }
}
$
$ awk -f tst.awk file
Count,ErrorCode,ErrorDescription,List of files it occurs in
1,Code,Description,file

There's various ways the above could be done, some briefer, but when input contains name=value pairs I like to create a name2value array (n2v[] is the name I usually give it) so I can access the values by their names. Makes the code easy to understand and modify in future to add fields, etc.

---

Here's my previous answer as there's some things in it you'll find usefule in other situations:

You don't say what you want the output to look like and your posted sample input isn't really adequate to test against and show useful output, but this GNU awk script shows the way to get a count of whatever attribute name/value pairs you like:

$ cat tst.awk         
match($0,/\s+*<Attribute Name="([^"]+)" Value="([^"]+)".*/,a) { count[a[1]][a[2]]++ }
END {
    print "\nIf you just want to see the count of all error codes:"
    name = "ErrorCode"
    for (value in count[name]) {
        print name, value, count[name][value]
    }

    print "\nOr if theres a few specific attributes you care about:"
    split("ErrorId ErrorCode",names,/ /)
    for (i=1; i in names; i++) {
        name = names[i]
        for (value in count[name]) {
            print name, value, count[name][value]
        }
    }

    print "\nOr if you want to see the count of all values for all attributes:"
    for (name in count) {
        for (value in count[name]) {
            print name, value, count[name][value]
        }
    }
}

.

$ gawk -f tst.awk file

If you just want to see the count of all error codes:
ErrorCode Code 1

Or if theres a few specific attributes you care about:
ErrorId ID 1
ErrorCode Code 1

Or if you want to see the count of all values for all attributes:
ErrorId ID 1
ErrorDescription Description 1
ErrorCode Code 1
Number Number 1
ProgId Prog 1
UserId User 1
Key Key 1

If you have data spread across multiple files, the above couldn't care less, just list them all on the command line:

gawk -f tst.awk file1 file2 file3 ...

It uses GNU awk 4.* for true multi-dimensional arrays, but there's trivial workarounds for any other awk if needed.

One way to run an awk command on files found recursively under a directory:

awk -f tst.awk $(find dir -type f -print)

Answer 3

I assume that since you mention Unix grep, you may likely have perl also. Here's a simple perl solution:

#!/usr/bin/perl

my %countForErrorCode;
while (<>) { /<Error ErrorCode="([^"]*)"/ && $countForErrorCode{$1}++ }
foreach my $e (keys %countForErrorCode) { print "$countForErrorCode{$e} $e\n" }

Assuming you are running *nix, save this perl script, make it executable and run with command like...

$ ./grepError.pl *.log

you should get output like...

8 Code1
203 Code2
...

where 'Code1' etc. are the error codes captured between the double quotes in the regex.

I worked this up on Windows with Cygwin. This solution assumes:

1. Location of your perl is /usr/bin/perl. You can verify with $ which perl
2. The regex above, /<Error ErrorCode="([^"]*)"/, is how you want to count.

The code is doing...

1. my %errors declares a map (hash).
2. while (<>) iterates each line of input and assigns current line to built-in variable $_.
3. /<Error ErrorCode="([^"]*)"/ implicitly tries matching against $_.
4. When a match occurs, the parentheses capture the value between the double quotes and assign the captured string to $1.
5. The regex "returns true" on a match only then does the count get incremented && $countForErrorCode{$1}++.
6. For output, iterate the captured error codes with foreach my $e (keys %countForErrorCode) and print the count and code on a line with print "$countForErrorCode{$e} $e\n".

Edit: more detailed output per updated spec

#!/usr/bin/perl

my %dataForError;

while (<>) {
  if (/<Error ErrorCode="([^"]+)"\s*ErrorDescription="([^"]+)"/) {
    if (! $dataForError{$1}) {
      $dataForError{$1} = {}; 
      $dataForError{$1}{'desc'} = $2;
      $dataForError{$1}{'files'} = {};
    }
    $dataForError{$1}{'count'}++;
    $dataForError{$1}{'files'}{$ARGV}++;
  }
}
my @out;
foreach my $e (keys %dataForError) {
  my $files = join("\n\t", keys $dataForError{$e}{'files'});
  my $out = "$dataForError{$e}{'count'}, $e, '$dataForError{$e}{'desc'}'\n\t$files\n";
  push @out, $out;
}
print @out;

And like you posted above, to pick up input files recursively you can run this script like:

$ find . -name "*.log" | xargs grepError.pl

And produce output like:

8, Code2, 'bang'  
    ./today.log  
48, Code4, 'oops'  
    ./2015/jan/yesterday.log  
2, Code1, 'foobar'  
    ./2014/dec/someday.log

Explanation:

1. The script maps each unique error code to a hash that tracks the count, description and unique filenames where the error code is found.
2. Perl auto-magically stores the current input filename into $ARGV.
3. The script counts each unique filename occurance, but does not output those counts.