75

I have few log files around 100MBs each. Personally I find it cumbersome to deal with such big files. I know that log lines that are interesting to me are only between 200 to 400 lines or so.

What would be a good way to extract relavant log lines from these files ie I just want to pipe the range of line numbers to another file.

For example, the inputs are:

filename: MyHugeLogFile.log
Starting line number: 38438
Ending line number:   39276

Is there a command that I can run in cygwin to cat out only that range in that file? I know that if I can somehow display that range in stdout then I can also pipe to an output file.

Note: Adding Linux tag for more visibility, but I need a solution that might work in cygwin. (Usually linux commands do work in cygwin).

iankit
  • 8,806
  • 10
  • 50
  • 56
bits
  • 8,110
  • 8
  • 46
  • 55

7 Answers7

152

Sounds like a job for sed:

sed -n '8,12p' yourfile

...will send lines 8 through 12 of yourfile to standard out.

If you want to prepend the line number, you may wish to use cat -n first:

cat -n yourfile | sed -n '8,12p'
johnsyweb
  • 136,902
  • 23
  • 188
  • 247
  • @bits: Happy to help. I appended the `cat` section of my answer while you typed that. Perhaps that'll be useful too. – johnsyweb Apr 15 '11 at 23:59
  • 2
    I think the first solution that doesn't involve `cat` is most suitable for me. Simple and concise. – bits Apr 16 '11 at 04:08
19

You can use wc -l to figure out the total # of lines.

You can then combine head and tail to get at the range you want. Let's assume the log is 40,000 lines, you want the last 1562 lines, then of those you want the first 838. So:

tail -1562 MyHugeLogFile.log | head -838 | ....

Or there's probably an easier way using sed or awk.

David
  • 2,226
  • 32
  • 39
14

I saw this thread when I was trying to split a file in files with 100 000 lines. A better solution than sed for that is:

split -l 100000 database.sql database-

It will give files like:

database-aaa
database-aab
database-aac
...
Dorian
  • 22,759
  • 8
  • 120
  • 116
7

And if you simply want to cut part of a file - say from line 26 to 142 - and input it to a newfile : cat file-to-cut.txt | sed -n '26,142p' >> new-file.txt

Marc Perrin-Pelletier
  • 12,696
  • 8
  • 28
  • 36
  • 1
    What you are saying is same as the accepted answer: http://stackoverflow.com/a/5683408 – bits Feb 13 '15 at 19:40
  • After doing this I'm not sure if 'cut' is the right word given that on my GNU/Linux system the command does a 'copy' rather than what one would normally attribute to a cut. – René Alejandro Santiago Nov 15 '17 at 15:38
6

How about this:

$ seq 1 100000 | tail -n +10000 | head -n 10
10000
10001
10002
10003
10004
10005
10006
10007
10008
10009

It uses tail to output from the 10,000th line and onwards and then head to only keep 10 lines.

The same (almost) result with sed:

$ seq 1 100000 | sed -n '10000,10010p'
10000
10001
10002
10003
10004
10005
10006
10007
10008
10009
10010

This one has the advantage of allowing you to input the line range directly.

thkala
  • 84,049
  • 23
  • 157
  • 201
  • Its a possible solution, but that requires me to calculate `39276-38438=838`. Because I will have to use `838` as input to head. I am looking for a solution in which the input parameters are strictly the beginning and ending line numbers, i.e. `38438` and `39276`. – bits Apr 15 '11 at 23:50
1

If you are interested only in the last X lines, you can use the "tail" command like this.

$ tail -n XXXXX yourlogfile.log >> mycroppedfile.txt

This will save the last XXXXX lines of your log file to a new file called "mycroppedfile.txt"

Jose Antonio Escobar Garcia
  • 309
  • 3
  • 6
-2

This is an old thread but I was surprised nobody mentioned grep. The -A option allows specifying a number of lines to print after a search match and the -B option includes lines before a match. The following command would output 10 lines before and 10 lines after occurrences of "my search string" in the file "mylogfile.log":

grep -A 10 -B 10 "my search string" mylogfile.log

If there are multiple matches within a large file the output can rapidly get unwieldy. Two helpful options are -n which tells grep to include line numbers and --color which highlights the matched text in the output.

If there is more than file to be searched grep allows multiple files to be listed separated by spaces. Wildcards can also be used. Putting it all together:

grep -A 10 -B 10 -n --color "my search string" *.log someOtherFile.txt

hbolingbroke
  • 1