Effective way to visualize and navigate huge log files

Question

I am in the process of designing a tool to visualize java logs. Specifically, these logs are generated by printing to the console whenever a method is invoked and whenever it is returned. These log statements are injected into the entire Android OS source code using bytecode manipulation. So far we have been able to instrument the Android OS and generate these log statements. The fields contained in these log statements are: process that invoked the method, method signature, argument types, return type and time stamp

The tool that uses these logs will have a detailed view (on zooming in) and a high-level overview (on zooming out). I am looking for efficient ways of visualizing and navigating this huge log file at high-level overview to get valuable information. Each log statement has a hierarchical relationship to another log statement. For Example, a log statement for a method invocation will be the parent of all the log statements of the methods invoked from within the parent method.

My questions are,

1. What would be an effective way to visualize and navigate these hierarchical log statements in a huge log file to get a good high-level overview? Sequence diagrams are useful for a detailed view but aren't suitable for a huge call trace.
2. Are there any existing tools in the market that have similar functionality? I have looked at log visualization tools but none have the high-level overview visualization.
3. As an app developer who is equipped with an instrumented VM that generates log statement for each called method and who can run an app on the said android VM, what information would be useful to you?
4. Any other suggestions?

Thanks in advance.

Edit: I added a few more details about hierarchical nature of the log statements.

Answer 1

I believe the diagram that is closest to what you describe is a Sequence diagram. There used to be a plugin for eclipse that would monitor your source code and build up a sequence diagram of all the call/return/timing/etc.

Here's a description of how the diagrams work and how to create them by hand:

http://agilemodeling.com/artifacts/sequenceDiagram.htm

They are a great way to model how the code interacts at runtime at a very high level.

Just with a quick query--this is in the eclipse marketplace:

https://marketplace.eclipse.org/content/objectaid-uml-explorer#group-metrics-tab

I haven't used this one, just what came up in the query, however I do remember creating an awesome diagram with code I was analyzing--covered an entire wall of my cube and was really helpful.

Answer 2

From your question, the data your using are server logs; and the structure of log files are very complex task and processing different log files for visualization is also possible under a certain limitation, you will need to create or develop your own standard tool to classify and aggregates the log files for processing then for visualization. However, some tools and platforms are already existed such as:

One of the effective ways is to process data in pipelines for big data information, since these log statements are server logs using Apache sparks, splunk, or Cisco Mars with integration in SIEM's solutions to process logs in real-time is one of the most effective ways used for processing server log files.

HPE Security ArcSight Data Platform also can deliver a high-performance, cost-effective solution that unifies big data collection, reporting and analysis across enterprise machine data.

Clock View and PeekKernelFlows use NetFloW log to monitor large IP spaces over long periods. In addition, a tool called ELVIS extensible log visualization tool, can correlate the process of server logs and provide a representation summary view of the important data.

I highly suggest you use a simple log structure ex. FTP or IIS server log files of selected fields, and visualize the output by using D3.js modules for static and interactive visualization specifically with Parallel Coordinates.