A process that removes part of a file and then let you append more data is nearly never available on any system, even though it is possible, it's just not something one does. It could be done at kernel level and be really efficient, but I've never see that either. (i.e. the kernel would simply unlink inodes from the start of the file and have an offset in the first inode of the file for byte capability--opposed to page ability.)
On a Unix system, you can use mmap() and unmap() for that purpose. So when your app. determines that the file size went over a certain amount, it would have to read from the start of the file, determine the location of, for example, the 10,000th line of log, and then memmove() the rest over to the start. Finally, it would truncate the file and reopen it in append mode. This last step is a very important step...
// WARNING: code without any error checking
// if multiple processes may run in parallel, make sure to use a lock as well
int fd = open("/var/log/mylog.log", O_RDWR);
ssize_t size = lseek(fd.get(), 0, SEEK_END);
lseek(fd.get(), 0, SEEK_SET);
char * start = (char *)mmap(nullptr,
size,
PROT_READ | PROT_WRITE, MAP_SHARED,
fd,
0);
char * end = start + size;
char * l10000 = start; // search line 10,000
for(int line(0); line < 10000; ++line)
{
for(; l10000 < end && *l10000 != '\n'; ++l10000);
if(*l10000 == '\n')
{
++l10000; // skip the '\n'
}
}
ssize_t new_size = end - l10000;
memmove(start, l10000, new_size);
truncate(fd, new_size);
close(fd);
(example found in sendmail::dequeue() on GitHub which includes all the error checking not found here.)
IMPORTANT: the memmove() call is going to be slow, especially on a rather large log file.
Note that most of the time, when a process opens a log file, it keeps it opened and that means changing the file under its feet won't do much good. Actually, in the mmap() example here, you would create a gap with many zeroes (\0 characters) between the moved data and the next write if you don't make sure to close and reopen the log (not shown in the code).
So, it's doable in code (here in C++, you could easily get that to compile in C, though.) However, if you just want to use bash, logrotate is certainly your best bet. However, by default, at least on Ubuntu, logrotate runs only once per day. You can change that specifically for the users who use your application or system wide.
At least you can run it hourly by moving or copying the logrotate script like so:
sudo cp /etc/cron.daily/logrotate /etc/cron.hourly/logrotate
You can also setup a per minute CRON file which runs that script. To edit the root crontab file:
sudo crontab -u root -e
Then add one line:
* * * * * root /etc/cron.daily/logrotate
Make sure to test and see that it works as expected. If you add such, you could also remove the /etc/cron.daily/logrotate script from there so it does not try to run it twice (once on the 'daily' run and once on the per minute run.)
Just be aware that there is a lingering bug in CRON as shown in my bug report to Ubuntu. It can cause memory problems when using CRON too much (like once a minute).
Also, as mentioned previously with the code sample above, you must reopen the log file. Just rotating won't do you any good unless the application either reopens the log file each time it wants to write to it, or it is told to rotate (i.e. close the old file and open the new one.) Without that rotation kick, the application will continue to append data to the old file, it does not matter what it is named. Unix remembers because it uses the inode once the file was opened and not the filename. (Under MS-Windows, you won't be able to rename without first closing all accesses to the file... that's very annoying!)
In many cases, you either restart the whole app. (because it's too dumb to know how to reopen the log), you send the app. a signal so it reopens the log file, or the app. is aware that the file changed, somehow...
If the app. is not capable or knowing, restarting will be your only option. That may be weird for a user if it has a UI.
