0

I've been searching for a solution to get all created/modified and deleted files by a specific process from an event trace (ETW) session (I will process data from an etl file not from a real-time session).

Apparently the simplest solution to get this done was to get the FileCreate and FileDelete events from FileIo_Name class and map them to the corresponding DiskIo_TypeGroup1 events. However, this solution isn't working for me since I don't receive any DiskIo_TypeGroup1 events for the corresponding FileDelete events, so I can not get the process ID. Also not all FileCreate events have an associated DiskIo_TypeGroup1 event (I think this happens for the empty created files or only for the opened files).

Note: I need DiskIo_TypeGroup1 mapping because FileIo_Name events don't have the ThreadId and ProcessId members populated - they are set to (ULONG)-1. Also, I can not decide which files where just opened or modified without knowing the "file write size". DiskIo_TypeGroup1 also don't have the ThreadId and ProcessId (in event header, on newer OS's) members populated, but it has the IssuingThreadId structure member from which I can obtain the ProcessId mapping to Thread_TypeGroup1 class events.

So I investigated how the FileIo_Create class can help me, and remarked that I can get the CreateOptions member which can have the following flags: (FILE_SUPERSEDE, FILE_CREATE, FILE_OPEN, FILE_OPEN_IF, FILE_OVERWRITE, FILE_OVERWRITE_IF). But the initial problem still persists. How can I check if a file was created from scratch instead of being just opened (e.g. in case of FILE_SUPERSEDE)?

Maybe I can use the FileIo_ReadWrite class to get Write event. Like using the DiskIo_TypeGroup1 class. So, if something was written to a file, then can I suppose that the file was either created or modified?

To find the deleted files I think that the FileIo_Info class and Delete event are the solution. Guess that I can receive Delete events and map them to FileIo_Name to get the file names.

Note: The FileIo_Create, FileIo_Info, FileIo_ReadWrite contain information about process id.

Are my suppositions right? What will be the best solution for my problem?

tux2nicolae
  • 21
  • 7
  • which language do you use? For .net you can use TraceEvent (https://www.nuget.org/packages/Microsoft.Diagnostics.Tracing.TraceEvent) to parse the ETL, here you have Events fro all FileIO and DiskIO events in the KernelTraceEventParser class. – magicandre1981 Jun 30 '16 at 15:48
  • Hi @magicandre1981, I am using C++, but already looked at the [KernelTraceEventParser](http://bcl.codeplex.com/SourceControl/latest#Tools/ETW/traceEvent/KernelTraceEventParser.cs) implementation, unfortunately this hasn't solved my problem because *TraceEvent* only parses and formats events in a nicer way, not having any high level processing, like my case of deducing created/modified/deleted files from a specific process. Thanks for your comment. – tux2nicolae Jul 01 '16 at 07:29
  • what do you mean with high level? any events has a processid, processname property – magicandre1981 Jul 01 '16 at 14:32
  • Yes you're right, almost all events has a processID except FileIo_Name (or at least i can't get them) and some other events like [TcpIp](https://msdn.microsoft.com/en-us/library/windows/desktop/aa364128(v=vs.85).aspx) (see Remarks on the bottom). Another related question can be found here : http://stackoverflow.com/questions/26440639/etw-system-calls-tracing Also i extracted the ETL with [Tracerpt](https://technet.microsoft.com/en-us/library/cc732700(v=ws.11).aspx) to see the results [IMG](http://i.imgur.com/pexkJVt.png). – tux2nicolae Jul 01 '16 at 15:32
  • `using (_source = new ETWTraceEventSource(dataFileName)) { _kernelTraceEventParser = new KernelTraceEventParser(_source); _kernelTraceEventParser.FileIODelete += data => { var filename = data.FileName; }; _source.Process(); }` – magicandre1981 Jul 02 '16 at 06:48
  • I can can get a filename at a FileIO event. I have no idea which issue you have – magicandre1981 Jul 02 '16 at 06:50
  • Hi @magicandre1981, I will take a deeper look at the KernelTraceEventParser implementation to see what is the difference(since i use directly native API's) or what i have missed, and will come back with details, thank you. – tux2nicolae Jul 02 '16 at 07:49
  • have you made some progress? – magicandre1981 Jul 06 '16 at 15:05
  • See also https://stackoverflow.com/a/51722043/403671 – Simon Mourier Jul 06 '22 at 10:13

1 Answers1

2

I will share my implemented solution as follow :

  1. Created Files:

    • I have stored all FileIo_Create events as a pending create operation and waited to receive associated FileIo_OpEnd to decide if the file was opened, created, overwritten, or superseded from the ExtraInfo structure member.

  2. Modified Files:

    • I marked files as dirty for every Write event from FileIo_ReadWrite and every SetInfo event with InfoClass->FileEndOfFileInformation and InfoClass->FileValidDataLengthInformation from FileIo_Info. Finally on Cleanup event from FileIo_SimpleOp verify if the file was marked as dirty and store as modified.

  3. Deleted files:

    • I marked the files as deleted if was opened with the CreateOptions->FILE_DELETE_ON_CLOSE flag from FileIo_Create or if a Delete event from FileIo_Info appears. Finally on Cleanup event from FileIo_SimpleOp stored the file as deleted.

Also the process id and file name was obtained from the FileIo_Create events, more precisely from OpenPath structure member and ProcessId event header member.

tux2nicolae
  • 21
  • 7
  • Nice, but how did you figure out what does each ExtraInfo value mean? – MoeKav Oct 23 '18 at 21:18
  • I'm sorry but I don't remember where I got this information from. But as my code tells, the ExtraInfo value of *FileIo_OpEnd* associated with *FileIo_Create* contains one of the following values : `#define FILE_SUPERSEDED 0x00000000 #define FILE_OPENED 0x00000001 #define FILE_CREATED 0x00000002 #define FILE_OVERWRITTEN 0x00000003 #define FILE_EXISTS 0x00000004 #define FILE_DOES_NOT_EXIST 0x00000005` – tux2nicolae Nov 06 '18 at 15:58
  • @MoeKav please check the above message and see if that helps you. – tux2nicolae Nov 06 '18 at 16:07
  • @T.Nicolae FILE_DELETE_ON_CLOSE of CreateOptions was re-named to FILE_ATTRIBUTE_OFFLINE – Ahmed Osama Sep 19 '21 at 04:40
  • @T.Nicolae How can you match a FileIOCreate event with FileIOOperationEnd event? FileIOOpEndTraceData does not have FileName or FileKey properties – Ahmed Osama Sep 19 '21 at 05:20
  • @T.Nicolae I found out that IrpPtr property can be used to match the events "FileIo events are logged at the beginning of the operation. OpEnd events can be enabled separately to indicate the end of those operations. Irp can be used to correlate the begin and end events." – Ahmed Osama Sep 19 '21 at 06:42
  • 1
    @AhmedOsama indeed I matched the events by IrpPtr – tux2nicolae Sep 20 '21 at 07:14