-1

I'm trying to make kernel32's ReadFile work since native c# alternatives are slow as...

The following code was working a couple of times, then suddenly only gives "Attempted to read or write protected memory. This is often an indication that other memory is corrupt."

I figured a restart of the server would make it work a couple times before getting the same error, but it seems I was wrong there, it happens no matter what I do now..

Here's my code:

using System.Runtime.InteropServices;



private WinFileIO.WinFileIO _winFile;

_winFile = new WinFileIO.WinFileIO(new byte[bufferSize]);
_winFile.OpenForReading(_fileInfo.FullName);


public WinFileIO(Array Buffer)
{
    PinBuffer(Buffer);
    pHandleRead = IntPtr.Zero;
    pHandleWrite = IntPtr.Zero;
    bufferSize = Buffer.GetLength(0);
}


private void PinBuffer(Array Buffer)
{
    UnpinBuffer();
    gchBuf = GCHandle.Alloc(Buffer, GCHandleType.Pinned);
    IntPtr pAddr = Marshal.UnsafeAddrOfPinnedArrayElement(Buffer, 0);
    // pBuffer is the pointer used for all of the I/O functions in this class.
    pBuffer = (void*)pAddr.ToPointer();
}


public void UnpinBuffer()
{
    if (gchBuf.IsAllocated)
        gchBuf.Free();
}


[System.Runtime.InteropServices.DllImport("kernel32", SetLastError = true)]
static extern unsafe System.IntPtr CreateFile
(
     string FileName,          // file name
     uint DesiredAccess,       // access mode
     uint ShareMode,           // share mode
     uint SecurityAttributes,  // Security Attributes
     uint CreationDisposition, // how to create
     uint FlagsAndAttributes,  // file attributes
     int hTemplateFile         // handle to template file
);


[System.Runtime.InteropServices.DllImport("kernel32", SetLastError = true)]
private unsafe static extern bool ReadFile
(
    int hFile, 
    byte[] arraypBuffer,
    int NumberOfBytesToRead,
    ref int lpNumberOfBytesToRead,
    int* ptr
);


[System.Runtime.InteropServices.DllImport("kernel32", SetLastError = true)]
static extern unsafe bool SetFilePointer
(
    System.IntPtr hObject,
    int lDistanceToMove,
    ref int lpDistanceToMoveHigh,
    EMoveMethod dwMoveMethod
);


public void OpenForReading(string FileName)
{
    Close(true, false);
    pHandleRead = CreateFile(FileName, 3, 3, 0, OPEN_EXISTING, 0, 0);
    if (pHandleRead == System.IntPtr.Zero)
    {
        Win32Exception WE = new Win32Exception();
        ApplicationException AE = new ApplicationException("WinFileIO:OpenForReading - Could not open file " +
          FileName + " - " + WE.Message);
        throw AE;
    }
}


public void MoveOffset(int moveDistance, EMoveMethod moveMethod)
{
    if (pHandleRead != System.IntPtr.Zero)
    {
        int moveDistanceHigh = 0;
        SetFilePointer(pHandleRead, moveDistance, ref moveDistanceHigh, moveMethod);
    }
}


public unsafe Tuple<int, byte[]> ReadChunk(int bufferSize, int offset)
{
    int bytesRead = 0;
    byte[] bufBytes = new byte[bufferSize];
    MoveOffset(offset, EMoveMethod.Begin);
    int pointer = (int)Marshal.PtrToStructure(pHandleRead, typeof(int));

    if (ReadFile(pointer, bufBytes, bufferSize, ref bytesRead, null))
    {
        byte[] outBytes = new byte[bytesRead];
        Array.Copy(bufBytes, outBytes, bytesRead);
        return new Tuple<int, byte[]>(bytesRead, outBytes);
    }
    return null;
}

I think that's all the relevant code.

I'm guessing it has something to do with the CreateFile and ReadFile-signatures not being fully compatible (IntPtr vs. int), the Marshal.PtrToStructure for the IntPtr not pointing where it should, or memory not being freed or something? The fact that it didn't work for a couple tries after a reboot confuses me though.

Anyone spot anything obvious or have any suggestions I can look into?

Thanks

Edit: As you might notice, this is kind of a mishmash of different approaches, I'm not using the pinned buffer anymore as I struggled to get the contents of the buffer read like I would want to.

Edit2: The stacktrace says this is the problem:

at System.Runtime.InteropServices.Marshal.PtrToStructure(IntPtr ptr, Type structureType)
CeeRo
  • 1,142
  • 3
  • 12
  • 21
  • Please provide a compilable sample – AcidJunkie Oct 07 '13 at 13:01
  • That's gonna take a lot of work... How would you go about debugging it then? – CeeRo Oct 07 '13 at 13:11
  • it's difficult just do guess the source of the problem when having a code snipped that is not executable or has a lot of dependencies to other libraries. If you want people to help you, then you also need to provide a compilable code snippet. Just make a separate C# project with just the necessary stuff inside – AcidJunkie Oct 07 '13 at 13:20
  • The RemObjects.SDK.Types.Binary is just a byte array more or less... The rest is all there in the code I think... Maybe I could have thrown in `using System.Runtime.InteropServices;` I'll edit – CeeRo Oct 07 '13 at 13:23
  • What makes you think this will go any faster than the managed alternative? Managed does not mean its slow. You just have to use it properly. The underlying FileStream implementation does more or less the same as all this (Seek -> SetFilePointer, Read -> ReadFile, etc.) – Simon Mourier Oct 07 '13 at 13:27
  • What I've read, and my own couple of runs before it didn't want to cooperate anymore. There was a very noticeable difference. The main issue is FileStream.Seek when it's making big jumps, it's sloooow – CeeRo Oct 07 '13 at 13:29
  • Got some more info, problem appears to be on this line as I suspected: `int pointer = (int)Marshal.PtrToStructure(pHandleRead, typeof(int));` – CeeRo Oct 07 '13 at 14:37

2 Answers2

3

Your pinvoke declarations are bad, get good ones from pinvoke.net

The 1st argument of ReadFile() is a handle, the one you got from CreateFile(). It is IntPtr. You dug yourself a hole by trying to convert the IntPtr to int, the Marshal.PtrToStructure() call is not correct. And will almost always bomb with an AVE, a handle is not a pointer.

Fix the [DllImport] declaration and use the IntPtr you got from CreateFile() directly. And don't forget that System.IO.FileStream is the .NET wrapper for these winapi functions, you only ever need to pinvoke CreateFile() if you need to open a handle to a device instead of a file. You then still use the FileStream constructor that takes a handle and use its Read() method to call ReadFile().

Hans Passant
  • 922,412
  • 146
  • 1,693
  • 2,536
  • I got the declarations from pinvoke.net... I just realized the Marshal.PtrToStructure call is incorrect, and turns out that seems to be the problem. Weird that it was working a couple times though. Anyways, using those same declarations and pointer = pHandleRead.ToInt64(); / ToInt32() seems to have solved it... So far at least. I'll run some more testing to see how much faster this solution is, if any. – CeeRo Oct 07 '13 at 15:04
  • Hmm, pinvoke.net isn't always perfect but it most certainly doesn't have any declarations on the page [for ReadFile](http://pinvoke.net/default.aspx/kernel32/ReadFile.html) that uses *int* for the handle argument. Using ToInt64 or ToInt32 is completely wrong as well. If you are doing this to increase perf then you are wasting your time. FileStream does the same thing you do. Well, correctly. – Hans Passant Oct 07 '13 at 15:09
  • Look at C# signature #4... What's an AVE btw? And I'm not convinced FileStream is as quick as this will be once I get it ironed out.. http://designingefficientsoftware.wordpress.com/2011/03/03/efficient-file-io-from-csharp/ – CeeRo Oct 07 '13 at 15:14
  • I looked. AVE = AccessViolationException. The only accurate statement I see in that blog post is "it is very difficult to completely trust the results". – Hans Passant Oct 07 '13 at 15:24
  • Well... From our 3-run-testing before it stopped working earlier, we could lower the buffering time on the client to half when making big jumps in a videofile, compared to FileStream. I will test it more though now that it's working again. And did you look at sig #4? :p – CeeRo Oct 07 '13 at 15:27
  • How much testing have you done on this yourself since you seem so grumpy about it? No offense – CeeRo Oct 07 '13 at 15:29
  • I've been reading up a bit on handles and IntPtr though, and why you were saying ToInt32/64 is completely wrong escapes me... IntPtr is a platformindependent int, meaning it can hold 64-bit (and 32 bit ofcourse) values. When it holds a handle that then obviously is an int, how can it be wrong to cast it to a proper 64 or 32 bit int depending on OS? And testing still shows this being faster than FileStream. – CeeRo Oct 08 '13 at 10:43
  • @CeeRo It's wrong because it's not needed. The parameter to ReadFile is the same type as that returned by CreateFile. It's IntPtr. Compiles on both 32 and 64 bit. Your native code may be faster than your managed code. That probably just means your managed code is inefficiently written. – David Heffernan Oct 08 '13 at 11:12
1

This code

int pointer = (int)Marshal.PtrToStructure(pHandleRead, typeof(int));

does not do what you think it does. This code attempts to treat pHandleRead as a pointer to int. But that's not what you want at all. The code that does what you want is:

int iHandleRead = pHandleRead.ToInt32();

In fact you only find yourself writing that code because your p/invoke declarations are poor. Your ReadFile declaration should receive the file handle as IntPtr and not as int. If it did so then you would not need the code above at all and would pass pHandleRead.

More general points:

  • I do not understand why you have chosen to use unsafe code here. Nothing at all here requires that.
  • I also think it unlikely that this code, if you make it work, will perform better than the managed equivalent.
David Heffernan
  • 601,492
  • 42
  • 1,072
  • 1,490
  • Thanks for the answer. I've already realized the IntPtr isn't really a pointer like I thought and changed it to ToInt32/64 (depending on System.Environment). The reason I picked that ReadFile declaration is because it seemed like the easiest one to work with for doing what I wanted. I am in uncharted territory here (for me) though, and more or less learning by trial and error. Is it a problem in practice to cast the IntPtr to int32/64? From what I've read an IntPtr is really an int... Also, about unsafe, are you saying I can remove all unsafe-declarations or? As I said, this is new to me – CeeRo Oct 08 '13 at 10:41
  • My code is based on the code found in this blog post: http://designingefficientsoftware.wordpress.com/2011/03/03/efficient-file-io-from-csharp/ (look near the end, under Download and installation, for dl-link) and then modified for my needs. That's where all the unsafe's are from. And I can also mention that the code is working now and testing so far shows it to be a lot faster when it comes to "big offset-jumps" (this isn't for linear file reading). Anyways, any and all tips on improving the code would be great – CeeRo Oct 08 '13 at 10:49
  • Sure you can remove all of the unsafe. You'll need to replace pointers with IntPtr. The code will only be faster because the managed code is sub-optimal. Your managed code could be fixed. At least that's my guess. Can't be 100% sure since can't see all the code. – David Heffernan Oct 08 '13 at 10:57
  • Using ToInt32 or ToInt64 is wrong. Don't do that. Changed the ReadFile declaration like I said. Your ReadFile does not work on x64. – David Heffernan Oct 08 '13 at 10:58
  • Aha, yeah that's a good reason for changing it. I would post my managed code, but I'm not sure where to do it. Maybe I should edit it into the op – CeeRo Oct 08 '13 at 11:10
  • I don't think so. I think this question has been answered. If you want to ask why your managed code is slow, that would be a new question. – David Heffernan Oct 08 '13 at 11:10
  • Right. Well, here it is, maybe you can spot anything obvious, I can't: if (_startFrame == 0) { _fileStream.Position = 0; byte[] headerBuffer = new byte[_frameOffsetList.ElementAt(1)]; readBytesCount = _fileStream.Read(headerBuffer, 0, headerBuffer.Length); toSend.Write(headerBuffer, 0, readBytesCount); } else { long seekOffset = startOffset - _fileStream.Position; _fileStream.Seek(seekOffset, SeekOrigin.Current); readBytesCount = _fileStream.Read(_buffer, 0, _buffer.Length); toSend.Write(_buffer, 0, readBytesCount); } – CeeRo Oct 08 '13 at 11:31
  • 1
    Sorry, not in comments to this question. You've asked a question here. You've neither voted nor accepted. Why should we keep on helping you. If you want to ask a new question, do so. But please have the decency to acknowledge the help you have had here. Hans pointed at the error, and I gave perhaps a little more explanation. Feel free to upvote and accept Hans answer. – David Heffernan Oct 08 '13 at 11:33
  • Yeah, you're right. The formatting was terrible too, I thought it would work better. Now that I've read up more about it Hans's answer doesn't seem so bad, but yesterday he came off as really grumpy/aggressive and short/uninformative. I'll accept his answer. Thanks – CeeRo Oct 08 '13 at 11:38
  • @Hans can be quite direct (absolutely nothing wrong with that in my view) but he is the resident expert on this topic. When you get Hans answering your question, you listen to him! – David Heffernan Oct 08 '13 at 11:40