Stack Overflow
Stack Overflow

Questions tagged [logfile-analysis]

The act of analyzing `logfiles` using methods such as, but not limited to: `parsing` and `deciphering`, etc.

The act of analyzing logfiles using methods such as, but not limited to: parsing and deciphering, etc.

79 questions
4
votes
1 answer

Exception Log Analysis Tools or Techniques

I'm logging Exceptions, and it would be nice if there were some tools or technique to perform some analysis of the Exception logs, can anyone recommend any? I'm imagining something with some kind of heuristic analysis, but I could be…
Myster
  • 17,704
  • 13
  • 64
  • 93
3
votes
1 answer

Extract occurrence of text between brackets from a text file Python

Log file: INFO:werkzeug:127.0.0.1 - - [20/Sep/2018 19:40:00] "GET /socket.io/?polling HTTP/1.1" 200 - INFO:engineio: Received packet MESSAGE, ["key",{"data":{"tag1":12,"tag2":13,"tag3": 14"...}}] I'm interested in extracting only the text from with…
spinState010
  • 35
  • 6
3
votes
1 answer

Expanding timeseries events with Pandas

Problem I'm looking for suggestions on how to make this more pythonic and improve the efficiency. I have a dataframe with events, each having at minimum a start and end timestamp. I am expanding the number of records so that the new table has one…
Knut Flage Henriksen
  • 135
  • 8
3
votes
1 answer

Process the lines in reverse order

How do I process a log file (in my case nginx access.log) in reverse order? Background I am developing a log file analyser script and I am just not able to get my head around on how to process huge log files from the end so I can sort out the time…
elhombre
  • 2,839
  • 7
  • 28
  • 28
3
votes
4 answers

Count the most common occurrences of a unknown strings in a file

I have a large file full of lines like this... 19:54:05 10.10.8.5 [SERVER] Response sent: www.example.com. type A by 192.168.4.5 19:55:10 10.10.8.5 [SERVER] Response sent: ns1.example.com. type A by 192.168.4.5 19:55:23 10.10.8.5 [SERVER] Response…
notAduck
  • 190
  • 1
  • 3
  • 13
2
votes
2 answers

Matching any character and or undefined newlines with regex in python

I have to parse a log txt file with regex in python. This is an example of a txt (named file): 20/01/18, 08:11 - Peter: Good morning How are you? Peter 20/01/18, 09:00 - Caroline: I am fine thanks. You? 20/01/18, 09:01 - Peter: Good I had some…
LUm-1
  • 41
  • 1
  • 5
2
votes
1 answer

Open a .log extension file in Python

I'm trying to open a .log extension file in Python but I keep encountering an IOError. I'm wondering if this has to do with the extension because clearly, the only way to get into that loop was if 'some.log' existed in the directory. location =…
robinhood91
  • 1,641
  • 3
  • 20
  • 36
2
votes
0 answers

Improve logfile parser with multi-line matching criteria

Given a somewhat peculiar logfile, represented by the following snippet: FILE (insert): file=Templates\xyz_EN_0615.pdf key=KEY_EN_AP_PAID FILE (insert): file=Templates\xyz_DE_0615.pdf key=KEY_DE_STD_PAID FILE (insert):…
ikaerom
  • 538
  • 5
  • 27
2
votes
5 answers

Best way to search for number of hours in the file

I have a log file containing timestamp and data (delimited with ','). I would like to have a Python script to parse the log file to count the number of data occurring for each hour. Here's the example of the log file content: 2015-06-18…
mike
  • 91
  • 1
  • 8
2
votes
2 answers

R split() function size increase issue

I have the following data set > head(data) X UserID NPS V3 V4 V5 Event V7 Element ElementValue 1 1 254727216 10 0 19 10 nps.agent.14b.no other attempt was…
anonR
  • 849
  • 7
  • 26
2
votes
1 answer

Parse log file with spaces

I am trying to parse a logfile (text file saved as a *.log file) into Visual Basic 2013 Express' DataGridView. The log file uses spaces as its delimiter, but the message part of the log has spaces in it. Here is the layout of the log file: Date…
CampSoup1988
  • 139
  • 6
  • 21
2
votes
3 answers

best way to parsing Large files by regex python

I have to parse a large log file (2GB) using reg ex in python. In the log file regular expression matches line which I am interested in. Log file can also have unwanted data. Here is a sample from the file: "#DEBUG:: BFM [L4] 5.4401e+08ps MSG DIR:TX…
user2109788
  • 1,266
  • 2
  • 12
  • 29
2
votes
2 answers

Parsing gzipped log file line by line inside a tar gzipped file

I have the following structure for log files: /var/log/2013-12-24.tar.gz This file has : 2013-12-24/{00,01,02...23}.log.gz files I need to parse these files, using Python, without extracting any files on disk OR memory Any suggestions ?
Ashish Disawal
  • 45
  • 1
  • 9
2
votes
2 answers

dataset of intrusion detection for apache log

FOUND ON : HoneyNet Project. in scan31 and scan34 PHPIDS have a filter rules to detect attack pattern in web access log file. I want to ask how do I know that PHPIDS can detect the attack (xss, sqli, any attack they told that they can detect) in…
Aoffiizr Kasidit
  • 21
  • 4
1
vote
0 answers

Log file length

I need to read a log file that is currently in use in order to monitor it for changes in length. I’d like to find the length of the log file when I click a button then when the length increases I’d like to know this and scan each line after the…
Jack
  • 41
  • 4
1
2
3 4 5 6