Split blf-files in Python into smaller files

Question

I'm wondering if it is possible to split BLF-files in Python? I know that there is a library (can) that supports BLF-files, but find no documentation on how to split/save. I can read a BLF-file with:

 import can
 log = can.BLFReader("logfile.blf")

Would appreciate any help if anybody has knowledge in how I would split this file, and save it into smaller blf-files.

score 1 · Accepted Answer · answered Apr 22 '22 at 21:00

1

You can read your logfile and write all messages to new files, changing file every N messages (in my example N=100, which is very low). I find it very useful to look at the doc, otherwise I wouldn't know how to resume iteration over log_in (in this case as a generator) or how to easily get object_count:

import can

OUTPUT_MAX_SIZE = 100

with open("blf_file.blf", 'rb') as f_in:
    log_in = can.io.BLFReader(f_in)
    log_in_iter = log_in.__iter__()
    object_count = log_in.object_count
    i = 0
    while i*OUTPUT_MAX_SIZE < object_count:
        i += 1
        with open(f"smaller_file{i}.blf", 'wb') as f_out:
            log_out = can.io.BLFWriter(f_out)
            j = 0
            while j < OUTPUT_MAX_SIZE:
                log_out.on_message_received(log_in_iter.__next__())
                j += 1
            log_out.stop()

answered Apr 22 '22 at 21:00

Tranbi

11,407
6
16
33

Thanks for the reply. How did you determine from the doc that object_count gives the number of messages? It was not obvious for me. Another question: Is there a faster way than this? It is rather slow for files larger than 100 MB. – fejz1234 Apr 28 '22 at 06:19
I just read the `__init__` method and the name was kinda explicit... Regarding speed, since blf are binary data, I can't think of any other way than what `BLFReader` already does. 100MB is huge so it takes time. If performance is an issue you could try with another language, [C++ for instance](https://github.com/Technica-Engineering/vector_blf) – Tranbi Apr 28 '22 at 07:15
I'm trying to learn C++, but have no idea how one would split the BLF. I've installed the library you included, but have no clue which class attribute that is available to split a BLF file. – fejz1234 May 10 '22 at 07:42
I haven't tried to use this library myself but I would start by looking at [File.cpp](https://github.com/Technica-Engineering/vector_blf/blob/master/src/Vector/BLF/File.cpp). If you encounter specific issues, do not hesitate to post a new question on SO. You could even contact the author directly. – Tranbi May 10 '22 at 07:59
Ok thanks. Do you know the contact information to the author? – fejz1234 May 10 '22 at 08:43
It's at the top of the file I linked ;-) – Tranbi May 10 '22 at 09:22
I've managed to write some C++ code to split it into smaller files. But one thing I noticed is that there is no performance difference, and the code uses the same amount of memory as the code in Python does. Might be because the can-library in Python is based on the same C++ library (Toby Lorenz)? – fejz1234 May 11 '22 at 12:50
You're right. I didn't notice that. So it seems your chances of speeding it up are slim... Someone had [tried](https://stackoverflow.com/questions/56630277/optimize-blf-reader-for-python-can-performance) 3 years ago but didn't get any useful answer either. I would suggest [converting all your .blf files to .asc](https://stackoverflow.com/questions/70757009/how-do-i-convert-blf-data-of-can-to-asc-using-python/70762284#70762284) overnight and work with them... it might be faster – Tranbi May 11 '22 at 12:59
My goal is to speed up my graphical interface (GUI) that I created for my colleagues, it splits a large BLF-file into smaller chunks of BLF-files (written in Python). It is very slow for larger files. Would the .asc way still make sense for me to try out? You wrote overnight, so maybe that would not be the solution then? – fejz1234 May 11 '22 at 13:18

Split blf-files in Python into smaller files

1 Answers1