0

I'm running apache MiNiFi c++, The flow starts with a GetFile processor.

The input directory includes some large files, and when I run MiNiFi the files above ~1.5 GB fail and do not get queued.

The log file states:

[org::apache::nifi::minifi::processors::GetFile] [Warning] failed to stat large_file_path_here

The other smaller files are queued as expected.

Does anyone have a clue what can be wrong? Why can't the processor manage the larger files?

Thanks in advance.

Adi Fuchs
  • 205
  • 2
  • 9

1 Answers1

1

What you found seems like a bug that is present even in the current MiNiFi implementation even up to today. The issue is that file sizes you mentioned, a narrowing exception happens here when trying to determine the length of the file to be written into the content repository.

We will try to fix this issue asap.

Adam Hunyadi
  • 1,890
  • 16
  • 32
  • Why then am I getting the exception "failed to stat" and not the exception written in line 78 in the link you've sent? – Adi Fuchs Feb 23 '21 at 14:59
  • 1
    It seems like, the root cause of this issue is that we use _stat on windows, but we should be using _stat64 to query files larger than 2GB. This is a minor issue compared to what I described. MiNiFi should have been tested with large files: I will make sure to increase the coverage for the upcoming releases. – Adam Hunyadi Feb 24 '21 at 13:29