I figured out what is going on.
I used the SysInternals Process Monitor to see what processes access the files. Turned out that the files are accessed by two processes: Windows Defender and (obviously) ExamDiff.
What happens is that once git copies the files to the temp folder, Windows Defender starts scanning them in the background.
Now and antivirus shouldn't modify the files that it scans, but apparently it does, by changing the files' extended attributes:
MsSense.exe,6168,SetEAFile,C:\Temp\git-difftool.a35220\right\Dir1\Dir2\Dir3\File.cpp,SUCCESS,
This should not affect the comparison in any way, but apparently it triggers the directory change notification
ExamDiff.exe,22236,NotifyChangeDirectory,C:\Temp\git-difftool.a35220\right,SUCCESS,"Filter: FILE_NOTIFY_CHANGE_FILE_NAME, FILE_NOTIFY_CHANGE_DIR_NAME, FILE_NOTIFY_CHANGE_ATTRIBUTES, FILE_NOTIFY_CHANGE_SIZE, FILE_NOTIFY_CHANGE_LAST_WRITE, FILE_NOTIFY_CHANGE_CREATION, FILE_NOTIFY_CHANGE_SECURITY"
And there we have it: ExamDiff is listening for directory changes (either file contents or metadata) in order to alert the user that the directory has changed. It doesn't care about the NTFS extended attributes changing, doesn't (or cannot) filter it out.
I wonder if there is a way in Win32 to check whether a change didn't affect anything other than the extended attributes (without saving the state of the folder and comparing), but that's a different question.
Edit: found a link:
https://social.technet.microsoft.com/Forums/en-US/c900b28d-4281-4a98-b6ca-418cf84f3cab/microsoft-defender-atp-mssenseexe-is-creating-extended-attributes