0

My goal is to read dataBuffer from A and write the data to device B repeatedly using MSDN APIs. I am using ReadFileEx and WriteFileEx to request overlapped I/O reading and writing, so that the original time-consuming for-loop using ReadFile and WriteFile can now go background and the UI main thread can be responsive all the time.

I first create structure MY_OVERLAPPED which inherits from OVERLAPPED and two more elements are added, event is a pointer to MyClass which can help access the variables in that class, handles, dataBuffer, etc., and count is for resetting Offset value each time before ReadFileEx is called. The reading position is count*524288, since I read 524288 bytes each time and store in dataBuffer.

My question is: 1.how do I not use the inherited OVERLAPPED structure's Offset element to change reading position, such as using reference count?

2.The following code does not work the way I think. The ReadCompletionRoutine can only run one time and end up with receiving extremely long offset number. For now I presume that's the issue and focus on it first, so please feel free to correct any of my code structure.

My code flow is: 1. set reading point to 0 2. call overlapped ReadFileEx 3. ReadCompletion will be called, the data read will be written using overlapped WriteFileEx 4. WriteCompletion will be called, set pointer to next 524288 bytes and read 5.(and so on) two completion routines will call each other till all data is transfered

MyClass.h

struct MY_OVERLAPPED: OVERLAPPED {
    MyClass *event;
    unsigned long long count;
};

MyClass.cpp - main

MY_OVERLAPPED overlap;
memset(&overlap, 0,sizeof(overlap));

//point to this class (MyClass), so all variables can later be accessed
overlap.event = this; 

//set read position******how do I not use Offset??
overlap.Offset = 0;
overlap.OffsetHigh = 0;
overlap.count = 0;

//start the first read io request, read 524288 bytes, which 524288 bytes will be written in ReadCompletionRoutine
ReadFileEx(overlap.event->hSource, overlap.event->data, 524288, &overlap, ReadCompletionRoutine);

SleepEx(0,TRUE);

MyClass.cpp - CALLBACKs

void CALLBACK ReadCompletionRoutine(DWORD errorCode, DWORD bytestransfered, LPOVERLAPPED lpOverlapped)
{
    //type cast to MY_OVERLAPPED
    MY_OVERLAPPED *overlap = static_cast<MY_OVERLAPPED*>(lpOverlapped);

    //write 524288 bytes and continue to read next 524288 bytes in WriteCompletionRoutine
    WriteFileEx(overlap->event->hDevice, overlap->event->data, 524288, overlap, WriteCompletionRoutine);
}

void CALLBACK WriteCompletionRoutine(DWORD errorCode, DWORD bytestransfered, LPOVERLAPPED lpOverlapped)
{
    MY_OVERLAPPED *overlap = static_cast<MY_OVERLAPPED*>(lpOverlapped);

    //set new offset to 524288*i, i = overlap->count for next block reading
    overlap->count = (overlap->count)+1;
    LARGE_INTEGER location;
    location.QuadPart = 524288*(overlap->count);
    overlap->Offset = location.LowPart;
    overlap->OffsetHigh = location.HighPart;

    ReadFileEx(overlap->event->hSource, overlap->event->data, 524288, overlap, ReadCompletionRoutine);
}
ctheadn8954
  • 129
  • 1
  • 8
  • **1** - yes - every i/o request must have unique `OVERLAPPED`. **2** use apc or iocp completion instead `hEvent` and you will be know which operation completed. `512` bytes - too small block much better use say 64kb buffer. better use cyclic buffer and `ReadFileScatter` / `WriteFileGather` – RbMm Jul 03 '18 at 13:28
  • Thanks for replying! So I will have to define 2 OVERLAPPED struct, one for read and one for write, for each loop, is it correct? And to identify the 2 overlap processes in every loop, I'll have to use apc or iocp completion, is it correct? – ctheadn8954 Jul 03 '18 at 13:44
  • you not need any loop at all. do all tasks in callbacks. possible use 64kb lock-free cyclic buffer and read/write by 4kb (page_size) chunks. – RbMm Jul 03 '18 at 13:48
  • If a freezing thread is the problem then overlapped I/O is not very likely to be helpful. It is just going to freeze on WFMO instead. Tends to be a UI thread problem, MsgWaitForMultipleObjects() is a solution but that is deep surgery. Focus first on not needing to overlap WriteFile, it is not common for serial communications, make the buffer large enough or drip-feed the device with a timer. WaitForCommEvent() on a worker thread is a decent way to find out that ReadFile() is not going to block, PostMessage to the UI thread to let it know. – Hans Passant Jul 03 '18 at 14:05
  • @RbMm I think this is beyond my know-how, do you have any reference so I can have some study on? thanks – ctheadn8954 Jul 03 '18 at 14:25
  • @HansPassant What does it mean by freeze on "WFMO", what's that? As your suggestion, I will need to create another thread? – ctheadn8954 Jul 03 '18 at 14:28
  • WFMO = WaitForMultipleObjects. Overlapped I/O is nice in a dedicated server-style app that handles many clients, a web server for example. That is not what a typical serial comm apps look like, they have other stuff to do. Like whatever that "freezing thread" does. Hard to help you btw when you don't talk about that important detail. – Hans Passant Jul 03 '18 at 14:33
  • @HansPassant Thanks! Got it. Can you explain more details on how to make the buffer large enough or drip-feed the device with a timer? Important detailed information is appreciated! – ctheadn8954 Jul 03 '18 at 14:54
  • @RbMm I am afraid that I have to use loop, since I would like to show the current progress like a progress bar. Can you provide an example on how to make WriteFileEx execute only after ReadFileEx is returned using callback(APC) in a for-loop? Thanks! – ctheadn8954 Jul 03 '18 at 17:29
  • you not need any loops 100%. you need work from callbacks. say when read callback called - you just call writefile with returned data and so on – RbMm Jul 03 '18 at 19:51
  • @RbMm Regarding not to use loop, I am now putting one ReadFileEx in the main function. Once the overlapped io is requested and is completed, the CALLBACK function I defined will be called. Inside the CALLBACK function, I put one WriteFileEx to write the data that has been read. Besides, in order to keep reading the rest data from the file, I put another ReadFileEx in the CALLBACK function after the WriteFileEx. I am now confusing if the method is correct? Second, according to your first answer, are you suggesting that using ReadFileScatter/WriteFileGather instead of ReadFileEx/WriteFIileEx? – ctheadn8954 Jul 04 '18 at 05:37
  • yes, this method is correct. another way can use iocp instead apc completion. call `BindIoCompletionCallback` on files and use `ReadFile`|`WriteFile` (without `Ex`) about `ReadFileScatter`/`WriteFileGather` advantage here that possible use two(or more) not contiguous buffers for read and write. this can be effective used with cyclic buffer. but this better when we work with 2 disk files copy. i not note at begin that you use serial as on file. not sure are this will be good for serial. – RbMm Jul 04 '18 at 07:19
  • @RbMm now I see the difference. Continuing my current method, I am faced with a problem of callback function: how a callback function access global class variables, such as handles, buffers, class functions? – ctheadn8954 Jul 04 '18 at 12:12
  • @Nick_Chang - you must not use any **global** data. callback function got pointer to `OVERLAPPED` passed to io. you need create own data structure inherited from overlapped. and you got pointer to this data back in callback. in this data structure must be pointer to class object where handle, buffers, etc – RbMm Jul 04 '18 at 12:22
  • @RbMm Do you mean try to use the given overlapped structure that is passed to the callback function only and try to get my own data structure from the overlapped structure? Not sure how to implement that, since all my variables are in a class now. Besides, I think I am keep using the same overlapped structure in the callback function throughout each ReadFileEx and WriteFileEx to keep track of the overlap->Offset position. Is it proper to do this? – ctheadn8954 Jul 04 '18 at 16:59
  • yes, you need use only pointer to overlapped.simply inherit you structure`struct MY_IRP : OVERLAPPED` and in callback do `static_cast(lpOverlapped)`. in `MY_IRP` define pointer to you class. mandatory use reference count on your object. not use any `OVERLAPPED` internal fields like `Offset` - it used by `ReadFile` – RbMm Jul 04 '18 at 18:00
  • @RbMm Now my new instance is created in another class. It looks like: Myclass *instance = new MyClass(); How to let MY_IRP struct to get this pointer(instance), since MY_IRP is in another class. – ctheadn8954 Jul 05 '18 at 05:47
  • 2. if I cant use the Offset inherited from OVERLAPPED struct, how ReadFileEx know which location to point to? I mean even if I define an Offset in MY_IRP, no link is made between MY_IRP and ReadFileEx – ctheadn8954 Jul 05 '18 at 05:51
  • how i say - you can not use any members of `OVERLAPPED`. you can not use offset, because it used by readfile[exe]. your first question i dont understand. are this not obvious ? `struct MY_IRP : OVERLAPPED { MyClass* _pObj; ...}` simply save pointer to `MyClass` in `MY_IRP`. and `MyClass` mandatory must have reference counting life. call `AddRef` when you assign pointer to it in `MY_IRP` and `Release` when destroy `MY_IRP` – RbMm Jul 05 '18 at 07:05
  • so you need have 2 classes - `MyClass` encapsulate your object state - file handle, read/write offset, pointer to another class(file) to where you write|read data. may be window handle to where you send progress messages. etc. and use reference counting on this object. another class`MY_IRP` - encapsulate single io operation context. it must inherit (or containing) `OVERLAPPED`, have pointer to `MyClass`, pointer to buffers which you use in I/O request, operation code may be , if you do several type I/O operation on object (read,write,ioctl,etc). in callback you got pointer to `OVERLAPPED` – RbMm Jul 05 '18 at 07:12
  • form this `OVERLAPPED` pointer you get back pointer to `MY_IRP` via `static_cast(lpOverlap)`. and here you will be have full context – RbMm Jul 05 '18 at 07:13
  • @RbMm Got it, I reedit my question. Detail is added, please help me figure out the correct way to do asynchronous I/O, thanks! – ctheadn8954 Jul 05 '18 at 12:26

0 Answers0