f# unsafe memory mapped file access

Question

I am playing around with some different methods for accessing a binary file, and one of the methods I am trying to play around with is adapted from the following question:

How can I quickly read bytes from a memory mapped file in .NET?

My translation to F# is as follows:

let memMapRead2 (fileName: String, offset: int64, size: int) =
    use mmf = MemoryMappedFile.CreateFromFile(fileName, FileMode.Open, "mappedFile", 0L, MemoryMappedFileAccess.Read) 
    use view= mmf.CreateViewAccessor(offset, 0L, MemoryMappedFileAccess.Read)

    let buf = Array.zeroCreate<byte> size

    let mutable pptr: byte = 0uy
    let ptr: nativeint = NativePtr.toNativeInt<byte> &&pptr
    let bptr: nativeptr<byte> = ptr |> NativePtr.ofNativeInt<byte>
    view.SafeMemoryMappedViewHandle.AcquirePointer(ref bptr)
    Marshal.Copy(IntPtr.Add(ptr, 0), buf, 0, size)
    view.SafeMemoryMappedViewHandle.ReleasePointer()

    buf

I am ultimately trying to read in all of the bytes from the file, but I want to see if I can get any faster than a FileStream or a MemoryMappedStream (both of which seem to benchmark similarly). However, the above code throws the following exception:

Unhandled Exception: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
   at System.Runtime.InteropServices.Marshal.CopyToManaged(IntPtr source, Object destination, Int32 startIndex, Int32 length)
   at FSI_0007.memMapRead2(String fileName) in C:\Users\******************************\Reader.fs:line 30
   at <StartupCode$FSI_0008>.$FSI_0008.main@()
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.RuntimeType.InvokeMember(String name, BindingFlags bindingFlags, Binder binder, Object target, Object[] providedArgs, ParameterModifier[] modifiers, CultureInfo culture, String[] namedParams)
   at Microsoft.FSharp.Compiler.AbstractIL.ILRuntimeWriter.execEntryPtFun@2123.Invoke(Tuple`2 tupledArg, Unit unitVar2)
   at Microsoft.FSharp.Compiler.Interactive.Shell.clo@1088-286.Invoke(FSharpFunc`2 exec)
   at Microsoft.FSharp.Primitives.Basics.List.iter[T](FSharpFunc`2 f, FSharpList`1 x)
   at Microsoft.FSharp.Compiler.Interactive.Shell.FsiDynamicCompiler.ProcessInputs(CompilationThreadToken ctok, ErrorLogger errorLogger, FsiDynamicCompilerState istate, FSharpList`1 inputs, Boolean showTypes, Boolean isIncrementalFragment, Boolean isInteractiveItExpr, FSharpList`1 prefixPath)
   at Microsoft.FSharp.Compiler.Interactive.Shell.FsiDynamicCompiler.EvalParsedDefinitions(CompilationThreadToken ctok, ErrorLogger errorLogger, FsiDynamicCompilerState istate, Boolean showTypes, Boolean isInteractiveItExpr, FSharpList`1 defs)
   at Microsoft.FSharp.Compiler.Interactive.Shell.clo@1899-294.Invoke(FsiDynamicCompilerState istate)
   at Microsoft.FSharp.Compiler.Interactive.Shell.FsiInteractionProcessor.InteractiveCatch[b](ErrorLogger errorLogger, FSharpFunc`2 f, b istate)
   at Microsoft.FSharp.Compiler.Interactive.Shell.FsiInteractionProcessor.execParsedInteractions(CompilationThreadToken ctok, TcConfig tcConfig, FsiDynamicCompilerState istate, FSharpOption`1 action, ErrorLogger errorLogger, FSharpOption`1 lastResult)
   at Microsoft.FSharp.Compiler.Interactive.Shell.clo@2078-297.Invoke(CompilationThreadToken ctok, TcConfig tcConfig, FsiDynamicCompilerState istate)
   at Microsoft.FSharp.Compiler.Interactive.Shell.FsiInteractionProcessor.mainThreadProcessAction[a,b](a ctok, FSharpFunc`2 action, b istate)
   at Microsoft.FSharp.Compiler.Interactive.Shell.res@2141-18.Invoke(ErrorLogger errorLogger, Tuple`2 tupledArg)
   at Microsoft.FSharp.Compiler.Interactive.Shell.clo@1831-292.Invoke(Unit unitVar0)
   at Sample.Microsoft.FSharp.Compiler.Interactive.Main.Microsoft-FSharp-Compiler-Interactive-Shell-Settings-IEventLoop-Invoke@104.Invoke()
   at System.Windows.Forms.Control.InvokeMarshaledCallbackHelper(Object obj)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Windows.Forms.Control.InvokeMarshaledCallback(ThreadMethodEntry tme)
   at System.Windows.Forms.Control.InvokeMarshaledCallbacks()
   at System.Windows.Forms.Control.WndProc(Message& m)
   at System.Windows.Forms.Form.WndProc(Message& m)
   at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
   at System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG& msg)
   at System.Windows.Forms.Application.ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(IntPtr dwComponentID, Int32 reason, Int32 pvLoopData)
   at System.Windows.Forms.Application.ThreadContext.RunMessageLoopInner(Int32 reason, ApplicationContext context)
   at System.Windows.Forms.Application.ThreadContext.RunMessageLoop(Int32 reason, ApplicationContext context)
   at Sample.Microsoft.FSharp.Compiler.Interactive.Main.WinFormsEventLoop.Microsoft-FSharp-Compiler-Interactive-Shell-Settings-IEventLoop-Run()
   at Microsoft.FSharp.Compiler.Interactive.Shell.runLoop@2383(FsiConsoleOutput fsiConsoleOutput, FsiEvaluationSessionHostConfig fsi, Unit unitVar0)
   at Microsoft.FSharp.Compiler.Interactive.Shell.FsiEvaluationSession.Run()
   at Sample.Microsoft.FSharp.Compiler.Interactive.Main.evaluateSession(String[] argv)
   at Sample.Microsoft.FSharp.Compiler.Interactive.Main.MainMain(String[] argv)
   at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
   at System.AppDomain.ExecuteAssemblyByName(AssemblyName assemblyName, String[] args)
   at System.AppDomain.ExecuteAssemblyByName(AssemblyName assemblyName, String[] args)
   at Sample.Microsoft.FSharp.Compiler.Interactive.Main.MainMain(String[] argv)
Session termination detected. Press Enter to restart.

FSI has never been so angry with me before - what exactly happened? Should I be using some sort of SecurityCritical attribute with this block of code? I know F# has the fixed keyword for working with pointers, but I don't think that needs to be used here (could be wrong, though).

score 3 · Accepted Answer · answered Oct 29 '18 at 09:47

3

I tested the following, and it works:

let memMapReadFile fname offset size =
    use mmf =
        MemoryMappedFile.CreateFromFile (fname, FileMode.Open,
                                         "mappedFile", 0L,
                                         MemoryMappedFileAccess.Read)
    use view = mmf.CreateViewAccessor (offset, 0L,
                                       MemoryMappedFileAccess.Read)
    let buf = Array.zeroCreate<byte> size
    let ptr =
        let p = ref (NativePtr.ofNativeInt 0n)
        view.SafeMemoryMappedViewHandle.AcquirePointer p
        !p
    Marshal.Copy (NativePtr.toNativeInt ptr, buf, 0, size)
    view.SafeMemoryMappedViewHandle.ReleasePointer ()
    buf

answered Oct 29 '18 at 09:47

dumetrulo

1,993
9
11

So presumably it was ptr that was the problem? – normal chemist Oct 29 '18 at 15:03
I would say your ``ptr`` was the beginning of the problem: it points to ``pptr`` which is a single byte, and trying to read more than 1 byte from that location will probably result in illegal memory access. Your ``ref bptr`` creates a reference on the heap to which ``AcquirePointer`` writes the pointer to the memory to be read but since you did not save the result of ``ref bptr``, you are unable to access the pointer; it is essentially thrown away. – dumetrulo Oct 29 '18 at 15:20
I see. I’m more used to the c# pointer syntax, I pretty much was just trying to get the types right since there aren’t a whole lot of f# pointer manipulation examples. Thanks for pointing me in the right direction! I’ll try it out and mark it. – normal chemist Oct 29 '18 at 15:22
You can avoid the extra allocation of a reference cell by using byrefs. It's arguably a bug (that may be resolved) that there's an equivalence of `ref` and `byref` in some circumstances. – Phillip Carter Oct 30 '18 at 18:47

f# unsafe memory mapped file access

1 Answers1