4

I have files which are very large (> 5G), and I want to remove some lines by the line numbers without moving (copy and paste) files.

I know this command works for a small size file. (my sed command do not recognize -i option)

sed "${line}d" file.txt > file.tmp && mv file.tmp file.txt

This command takes relatively long time because of the size. I just need to remove the first line and the last line, but also want to know how to remove line number n, for example.

Jiho Noh
  • 479
  • 1
  • 5
  • 11

3 Answers3

0

Because of the way files are stored on standard filesystems (NTFS, EXTFS, ...), you cannot remove parts of a file in-place.

The only thing you can do in-place is

  • append to the end of a file (append mode)
  • modify data in a file (read-write mode)

Other operations must use a temporary file, or temporary memory to read the file fully and write it back modified.

EDIT: you can also "shrink" a file as read here using a C program (Linux or Windows would work) so that means that you could remove the last line (but still not the first line or any line in between)

Community
  • 1
  • 1
Jean-François Fabre
  • 137,073
  • 23
  • 153
  • 219
  • I see, but what if the line you want to delete is either the first line or the last one? Is there a way to shrink the file on the filesystem by moving the pointers (or equivalent meta information) of the begin or end of the file? – Jiho Noh Nov 04 '16 at 21:11
  • see my edit. Maybe you could remove the start of the file by moving pointers but you would have to edit the disk blocks (superuser + super-dangerous :)) – Jean-François Fabre Nov 04 '16 at 21:15
0

you can use the ed command which is quite similar to sed

ed -s file.text

you can use the d command, $d will delete the last line while 1d will delete the first one, and wq will write and exit.

The following command will do everything (delete first and last line, write, and exit)

echo -e '1d\n$d\nwq' | ed -s test.txt

using sed you can use the same commands sed '1d;$d' test.txt

Adam
  • 17,838
  • 32
  • 54
-1

If you are using a recent Linux, you can remove chunks of the file in any position: https://lwn.net/Articles/415889/

There's a command to remove any part of the file: fallocate

See: https://manpages.ubuntu.com/manpages/xenial/man1/fallocate.1.html

For example: fallocate -p -o 10G -l 1G qqq

Javier
  • 2,752
  • 15
  • 30