Questions tagged [log-files]

A log file is a file, often in clear text, which is generated by a software process to record some specific details about the workings of this process.

A log file is a file which is generated by a software process to record some specific details (events) about the workings of this process. Most logfiles are plain, human-readable text files, some however are using a binary format to conserve space and need specific treatment for post-processing. (Example: Unix login history in utmp/wtmp format)

A log file can be categorized into several types:

  1. Access log which records information related to the usage of the service provided by this process.
  2. Error log which records error conditions.
  3. Info/Debug log which records more detailed information as compared with other logs.
  4. Transaction log to document a chain of events, this often is required to be immutable to avoid later tampering.
  5. redo logs, containing the complete information to replay a chain of actions, often used where incomplete data transactions would lead to inconsistent data, for example in file systems or databases.

The main two goals of creating and maintaining log files are:

  1. Finding and debugging problems.
  2. Gathering usage information.

Depending on the frequence of events and the level of detail, logfiles may be growing very quickly, and usually require a strategy for managing their growth.

Depending on special reqirements, log-files may go to specialized devices to meet criteria of speed or immutability.

1103 questions
8
votes
6 answers

Send email alert on log file entry?

Under Linux, what's a simple way to automatically watch a logfile, and email me if a certain string appears? I have an application that will log certain failures to a logfile, but has no built-in way of sending alerts or executing scripts on…
davr
  • 1,729
  • 3
  • 14
  • 25
8
votes
4 answers

Can expire_logs_days be less than 1 day in MySQL?

So... yesterday I received an "after the fact email" about a campaign that has started for one of the services that I run. Now the DB server is getting hammered, hard, to the tune of about 300mb/min in binary logging for the replicate. As you could…
blacktip
  • 301
  • 1
  • 2
  • 5
7
votes
1 answer

Possible to log X-FORWARDED-FOR to nginx error_log?

Nginx allows custom log formats for access logs -- right now I'm logging http_x_forwarded_for and it's working fine. The problem is my error log only shows the LB IP. After Googling and reading through their documentation, it seems error_log…
skrewler
  • 308
  • 1
  • 3
  • 13
7
votes
2 answers

Cloud mapping experiment. Contact research@pdrlabs.net in the access logs

54.204.131.75 - - [09/Jul/2014:17:53:43 +0100] "HEAD / HTTP/1.1" 200 283 "-" "Cloud mapping experiment. Contact research@pdrlabs.net" A new line in my access log the other day. As far as i can tell this is most likely a phishing scam. Does anyone…
John
  • 887
  • 4
  • 15
  • 25
7
votes
3 answers

W3C log analyzer

I'm looking for a W3C log analyzer. I know there are plenty around, but it's hard to choose a good one. I was wondering if anyone could make a few recommendations?
lomaxx
  • 235
  • 4
  • 6
7
votes
3 answers

Avoid logging of certain missing files into the Apache2 error log

I'd like to avoid logging some missing files (that gives a 404) into the Apache2 error log. I want to do this on an Eclipse update site for my plugin. Problem is that the Eclipse P2 code tries to access its metadata files like content.xml,…
Peter Štibraný
  • 329
  • 4
  • 11
7
votes
4 answers

get text between keywords in log file

If I have a log file and want to dump only the text between 1234 and 9876 in to another file, how can i do this easily? If I have a text file like this: idsfsvcvs sdf sdf e e sd vs d s g sg s vc d slkdfnls 1234 keep me text 9876 das a g w eg dsf…
Darkmage
  • 323
  • 3
  • 12
7
votes
1 answer

OSX /var/log/system.log -- where do i define pruning interval?

I have a script that periodically checks /var/log/system.log and I've noticed across our network that some machines keep logfiles for a month through reboots and other activities, while some purge the file each night or each reboot. Does anyone…
David Houde
  • 3,200
  • 1
  • 16
  • 19
7
votes
5 answers

Linux - martian source in /var/log/messages

I keep getting these messages in /var/log/messages : Mar 8 23:17:25 saas1 kernel: martian source 169.254.1.1 from 169.254.95.118, on dev usb0 Mar 8 23:17:25 saas1 kernel: ll header: ff:ff:ff:ff:ff:ff:00:21:5e:de:1b:be:08:06 Over and over every 5…
edumike
  • 199
  • 1
  • 2
  • 7
7
votes
3 answers

How to enable non-error Apache log on MAMP

How can I enable non-error server logging on MAMP (or, am I totally blind, and is it already enabled by default?) So far, I can only find the error logs for Apache, MySQL and PHP in /Applications/MAMP/logs. But I'd like to access the server log (ie,…
yc209
7
votes
2 answers

Logformat for catching asked hostname in a *.domain.com scenario?

I have an Apache 2.2 VirtualHost with a *.example.com ServerName. This is required for my scenario, all subdomains are handled with the same site. Now, in the access log, I am trying to figure out a LogFormat variable (or way) that will let me log…
Dhiraj Gupta
  • 173
  • 1
  • 1
  • 5
7
votes
3 answers

What is the best tool to aggregate traffic stats from multiple nginx servers?

The setup: 2 or more nginx machines each machine has the same virtual hosts traffic is load balanced via DNS to each machine I need to figure out what are the best tools to use to get some traffic stats, mostly interested in amount of hits and…
gekkz
  • 4,229
  • 2
  • 20
  • 19
7
votes
2 answers

How can I view log files in Linux and apply custom filers while viewing?

(I asked this question on stackoverflow but it might be better off here...) I need to read through some gigantic log files on a Linux system. There's a lot of clutter in the logs. At the moment I'm doing something like this: cat logfile.txt | grep…
Dan
  • 171
  • 3
7
votes
1 answer

Apache2 access.log has multiple lines that say "combine"

I'm running an Apache2 webserver on Ubuntu 20.04.2 LTS. I noticed that my access.log has several lines that simply say combine on them, with no information about the request or anything else. There are other lines in the access.log that do not say…
Noah Broyles
  • 179
  • 6
6
votes
1 answer

How to create a GDPR compliant HTTP server access log with focus on remote IP anonymization and nginx?

EU's General Data Protection Regulation (GDPR), and the German DSGVO implementation, are very strict when it comes to individual-related data (such as IP addresses). However this question is not about the GDPR, but how to implement the regulation…
burnersk
  • 2,056
  • 5
  • 27
  • 39