Questions tagged [log-files]

A log file is a file, often in clear text, which is generated by a software process to record some specific details about the workings of this process.

A log file is a file which is generated by a software process to record some specific details (events) about the workings of this process. Most logfiles are plain, human-readable text files, some however are using a binary format to conserve space and need specific treatment for post-processing. (Example: Unix login history in utmp/wtmp format)

A log file can be categorized into several types:

  1. Access log which records information related to the usage of the service provided by this process.
  2. Error log which records error conditions.
  3. Info/Debug log which records more detailed information as compared with other logs.
  4. Transaction log to document a chain of events, this often is required to be immutable to avoid later tampering.
  5. redo logs, containing the complete information to replay a chain of actions, often used where incomplete data transactions would lead to inconsistent data, for example in file systems or databases.

The main two goals of creating and maintaining log files are:

  1. Finding and debugging problems.
  2. Gathering usage information.

Depending on the frequence of events and the level of detail, logfiles may be growing very quickly, and usually require a strategy for managing their growth.

Depending on special reqirements, log-files may go to specialized devices to meet criteria of speed or immutability.

1103 questions
13
votes
6 answers

Open source tomcat log viewer

I'm managing several tomcat instances (aprox 30-35) and looking for some console to centralize the logs. I tried php-syslog-ng before (no with tomcat), and found it useful. I would like that these software would have some of these features: -Syntax…
Ari
13
votes
4 answers

Proper way to rotate Nginx logs

I would like to achieve rotation of nginx logs that: would work without any extra software (i.e. - best if without "logrotate") would create rotated files with names based on date Best approach is something like PostgreSQL has - i.e. in its…
user13185
13
votes
7 answers

How did you implement log management on your servers?

I'm trying to figure out how other people implement their log management systems. I have 20-30 Linux servers and a few Windows boxes (most of them virtualized). We utilize a lot of Perl and Bash scripts to do most of our automated jobs and I'm…
Edward
  • 449
  • 3
  • 12
13
votes
2 answers

Rotate a file that's open and being written at all times

I have an linux application that continually writes logging information into a log file, eg. /var/log/application.log. As the application does not rotate the file automatically, this log file can reach a size of gigabytes in some weeks, so I want to…
Bruno Polaco
  • 345
  • 2
  • 9
13
votes
5 answers

How to Enable IPtables TRACE Target on Debian Squeeze (6)

I am trying to use the TRACE target of IPtables but I can't seem to get any trace information logged. I want to use what is described here: Debugger for Iptables. From the iptables man for TRACE: This target marks packes so that the kernel will…
bernie
  • 385
  • 1
  • 3
  • 12
13
votes
9 answers

Anyone know of a tool to detect and report on repeating patterns in a log file?

I need to monitor some large noisy log files (500m/day) from a Java application (log4j). Right now I manually look at the files, grep for "ERROR" and so on. However it should be possible for a tool to spot repeating patterns in the file, count them…
David Tinker
  • 589
  • 1
  • 8
  • 18
13
votes
4 answers

Is there a way to remove "Last message repeated x times" from logs?

I have a script that updates me on any changes to the kernel log on my (Debian) servers. The problem is that occasionally the log contains lines like: last message repeated 4 times I realize that this is a feature to shorten the size of the logs -…
Brent
  • 22,857
  • 19
  • 70
  • 102
13
votes
5 answers

What is a good SQUID Logs Analyzer?

Can anyone recommend me a good tool for analyzing squid logs ? I'm currently using SARG. Is there something better? I need to easily see what sites were visited/denied, when and by whom. There is a list of squid analyzers. Some of them seem to be…
Jindrich
  • 4,968
  • 8
  • 30
  • 42
13
votes
2 answers

How to enable error log in lighttpd properly?

I have a Centos 5 system with Lighttpd and fastcgi enabled. It does log access but does not log errors. I have Internal Server Error 500 and no info in log and when I try to open not -existing file also - no info in error log. How to enable it…
Tom Smykowski
  • 1,125
  • 5
  • 19
  • 27
13
votes
5 answers

Fast extraction of a time range from syslog logfile?

I've got a logfile in the standard syslog format. It looks like this, except with hundreds of lines per second: Jan 11 07:48:46 blahblahblah... Jan 11 07:49:00 blahblahblah... Jan 11 07:50:13 blahblahblah... Jan 11 07:51:22 blahblahblah... Jan 11…
mike
  • 3,963
  • 11
  • 30
  • 27
12
votes
3 answers

Good Free Tomcat Log Analyser?

Could someone recommend a good and free program for analysing Apache Tomcat logs. I don't need real-time analysis or anything like that. I have a log and I just want a quick and easy way to get some usage statistics from it. The stats don't even…
Bart B
  • 3,457
  • 6
  • 31
  • 42
12
votes
3 answers

Fail2ban log filled with entries saying "fail2ban.filter : WARNING Determined IP using DNS Lookup:.."

My fail2ban log at /var/log/fail2ban.log is completely filled with entries saying: fail2ban.filter : WARNING Determined IP using DNS Lookup: [IP address] I think this may have begun after I changed my ssh port... Any idea what the cause of this is…
Dirk Calloway
  • 252
  • 1
  • 2
  • 9
12
votes
3 answers

nginx access logs ignore certain requests

I'm running nginx behind haproxy (running on the same server). I've configured haproxy to use a simple html file on nginx to verify the service is up, since I don't have/want a valid "/" URL on this host. Nginx doesn't support the OPTIONS request…
Michael Marano
  • 151
  • 1
  • 1
  • 4
12
votes
6 answers

Is anybody using Splunk in a large-scale production environment?

I've been watching the videos at splunk.com and really it's hard to believe that one can get all those features for free, there's still that "where's the catch?" in the back of my head. So it'd be great if anybody that is actually using it Splunk on…
Nano Taboada
  • 145
  • 2
  • 9
12
votes
6 answers

nginx error log was huge, so I deleted and created a new one, now nginx won't start

I deleted the /var/log/nginx/error.log file, and then created a new one using: sudo nano error.log Doing ls -la shows that the error.log and access.log have the same permissions. When I try and start nginx I get the error: alert: could not open…
Blankman
  • 2,891
  • 10
  • 39
  • 68