Doesn't boost::log respect keywords::max_size when custom collector is set?

Question

I am using Boost 1.61 and what I ultimately want is:

1. Rotate current log file if its size is reaching FILE_ROTATION_SIZE
2. Handle the rotated file with a custom collector for further compression (is this the right way for compressing rotated files?)
3. Remove some of oldest (desirably compressed) files if the total size of files reaches FILES_MAX_SIZE

Now, without any collector I can achieve points 1 and 3 just with the following snippet

sink = logging::add_file_log(
      keywords::file_name = fileName,
      keywords::rotation_size = FILE_ROTATION_SIZE,
      keywords::scan_method = sinks::file::scan_method::scan_matching,
      keywords::target = logDir.native(),
      keywords::max_size = FILES_MAX_SIZE,
      keywords::format =
      (
         expr::stream
            << expr::format_date_time< boost::posix_time::ptime >("TimeStamp", "%Y-%m-%d %H:%M:%S.%f")
            << " " << expr::attr< boost::log::attributes::current_thread_id::value_type >("ThreadID") << " "
            << "<" << expr::attr< Logger::SeverityLevel >("Severity") << "> "
            << expr::message
            << expr::attr< std::wstring >("Suffix")
      ),
      keywords::auto_flush = true

   );

However, when setting a collector with sink->locked_backend()->set_file_collector(_fileCollector); inherited from

boost::log::sinks::file::collector which for now does basically nothing the files still continue rotate, but older files don't get removed.

Here is how the collector looks like:

class FileCollector : public boost::log::sinks::file::collector
{
   virtual void store_file(boost::filesystem::path const& src_path) override;

   virtual uintmax_t scan_for_files(boost::log::sinks::file::scan_method method,
                                       boost::filesystem::path const& pattern = boost::filesystem::path(),
                                       unsigned int* counter = 0) override;
};

void FileCollector::store_file(boost::filesystem::path const& src_path)
{
   LOG << "src_path: " << src_path;
}

uintmax_t FileCollector::scan_for_files(boost::log::sinks::file::scan_method method,
                                                boost::filesystem::path const& pattern,
                                                unsigned int* counter)
{
   return 1;
}

scan_for_files() is not being called at all.

From the answer to this question, I can see author says max_size is a collector parameter, so I assume there should be some kind of way of setting it in my FileCollector class. Calling sinks::file::make_collector() instead of inheriting custom collector seems is not an option, because it doesn't have means for providing the desired store_file() callback, where I plan to put the compression logic.

Does this mean that I should keep tracking the total size and handle the deletion when needed by myself?

Thanks!

Answer 1

is this the right way for compressing rotated files?

Yes, if you have to do this within your application. Note that unless you're using asynchronous logging, log file rotation is done synchronously. This means that some random log statements in your application will take a rather long time to complete while the log file is being compressed.

A better solution could be using a separate service such as logrotate that will handle log rotation and compression without interfering with your application performance.

Does this mean that I should keep tracking the total size and handle the deletion when needed by myself?

Yes, the file collector is fully responsible for managing the rotated files. Whenever store_file is called, your file collector has to perform the necessary steps to compress and store the file in the target storage and possibly remove any older files.

The parameters like target and max_size are interpreted by the file collector implemented in Boost.Log, which is created by sinks::file::make_collector. If you implement your own collector, you have to initialize it in your own way - possibly by passing these parameters to the collector constructor.