Stack Overflow
Stack Overflow

Questions tagged [log-analysis]

Log-analysis is the process of parsing log file, created from any logging service or application.

126 questions
1
vote
1 answer

goaccess log parsing of very simple log not working

My first question in this community, hope it's gonna be awesome around here! To the topic: I have this very simple log file constructed in the following manner 192.168.178.21 [21/07/2015] "GET /path/to/somewhere HTTP/1.1" "/path/to/somewhere"…
whatTheHack
  • 33
  • 5
1
vote
2 answers

How to decrease the time of log analysis for large files in java

I have to analyze different log files which include retrieving time-stamp, URL, etc. I am using multithreading for this. Each thread is accessing different log file and doing the task. Program for doing it : public class…
SachinSarawgi
  • 2,632
  • 20
  • 28
1
vote
1 answer

What is the fastest way to search for patterns through 20-30 GB of multiple logfiles

I am performing log analysis, which I want to automate so that it runs daily and reports findings. The analysis runs on standard workstations, 8 cores, up to 32 GB of free RAM. The prototyping is based on GNU Grep (--mmap), Sqlite (on a RAM disk)…
wishi
  • 7,188
  • 17
  • 64
  • 103
1
vote
1 answer

How to get data from regularly appended log file in Apache Spark?

I have one Apache access log file which has some data and it is continuously increasing. I want to analyze that data using Apache Spark Streaming API. And Spark is new for me and i created one program in which ,i use jssc.textFileStream(directory)…
Kaushal
  • 3,237
  • 3
  • 29
  • 48
1
vote
1 answer

Ideal database for grouping data by timestamp

I'm in the process of testing some noSql solutions for handling some basic log analytics. I'm looking for something that is optimized for reads. The data has a timestamp and some other columns that I want to count and sum. I need the ability to…
Daniel
  • 34,125
  • 17
  • 102
  • 150
1
vote
2 answers

How to find most frequent user agent in nginx access.log

In order to counter a botnet attack, I am trying to analyze a nginx access.log file to find which user agents are the most frequent, so that I can find the culprits and deny them. How can I do that?
qliq
  • 11,695
  • 15
  • 54
  • 66
1
vote
1 answer

Python script analyzing log files

I am creating a python script to analyze a log file(example: http://www.monitorware.com/en/logsamples/apache.php) and needs suggestions of methods that I can use to achieve this. reading the log file with open method. OK. counting elapsed time of…
JustinBieber
  • 1,127
  • 3
  • 12
  • 16
1
vote
1 answer

goaccess analystis nginx log - parseing url

I am using goaccess statistics my ngnix log. But the problem is that the same url have difference parameter. 115.*.*.115 - - [01/Nov/2013:06:15:29 +0000] "GET /this/is/example/test.html?ver=53&q=aaaaaa HTTP/1.1" 200 64 "-" "-" 115.*.*.115 - -…
leiyonglin
  • 6,474
  • 12
  • 36
  • 41
1
vote
1 answer

Usage log analysis

I'm developing a RESTful API with java and want to have usage statistics. so i've set a filter that log all authenticated request in a defined format, it currently simply writes the data to a day rolling file. i want to have an administration panel…
Yoav A
  • 555
  • 1
  • 4
  • 16
1
vote
2 answers

starfish or splunk

hiall My goal is to analyze log files of Hadoop and there are two tools starfish(open source) and splunk(commercial product). Does anyone know the pros and cons as to which one to choose. I really appreciate your answer. Thanks
user1687035
  • 253
  • 1
  • 4
  • 13
1
vote
1 answer

Which nosql database for heterogeneous records?

I'm looking at different options for storing log entries for easier querying/reporting. Currently I write scripts that parse and find the data, but the data is becoming more and more in demand, so it's becoming worth it to put the log data in a…
Niklas
  • 5,736
  • 7
  • 35
  • 42
0
votes
0 answers

output of the most important and frequently encountered logs

I have rsyslog + loganalyzer, it simply displays events that are sent by devices on the network. Is it possible to make the most popular events for the day displayed and not disappear?
user21402031
  • 31
  • 5
0
votes
0 answers

Highlight different keywords with different colours in vim

Using Vim for Log analysis with the inbuilt VIM search functionality and hlsearch, I'd like to assign a different colour for each search result I get. For e.g entries of my log would be. CHECKPOINT 1 CHECKPOINT 2 CHECKPOINT 3 CHECKPOINT…
raving_sage
  • 13
  • 1
  • 5
0
votes
0 answers

Anomaly log file type detection and predict future log error

I am working on anomaly log analysis methods such as drain approach to use to structure log data into structure data. We have lots of log file and we have no idea this specific log file belong to which software? like android, HDF and etc. How we…
Khawar Islam
  • 2,556
  • 2
  • 34
  • 56
0
votes
1 answer

is it possible run logdna-agent in USS unix systems

I am trying to feed my server and jar logs into LogDNA log analysis service. My services are all running in mainframe z/OS environment, I have written a simple node js code which could use the LogDNA log ingestion service, that feeds log with REST…
Jeeva nandham
  • 27
  • 4
1 2 3
…
8 9