2

I have a very verbose dictionary that stores 350 keys, each of which is a byte array of 3 bytes. This dictionary is to interpret incoming data from home-brewed hardware.

In stress-testing an application, I've hit a strange error where it is telling me that there is a KeyNotFoundException. The strange thing is that the key is present in the dictionary, and I am staring right at it. The dictionary uses a special comparator which could be the reason for the issue:

private static Dictionary<byte[ ], string> PlayerMap =
    new Dictionary<byte[ ], string>( new ByteArrayComparator( ) );

public class ByteArrayComparator : IEqualityComparer<byte[ ]> {
    public bool Equals( byte[ ] left, byte[ ] right ) {
        if ( left == null || right == null )
            return left == right;
        return left.SequenceEqual( right );
    }

    public int GetHashCode( byte[ ] Key ) {
        if ( Key == null )
            throw new ArgumentNullException( "Key" );
        return Key.Sum( B => B );
    }
}

This only happens when I simulate users pressing buttons at insane speeds, which means that the dictionary is being queried for the key many, many times.

Why is this happening? Is there something wrong with the comparator?

EDIT

For a bit of clarity the controller runs asynchronously (as it needs to in order to be able to handle the incoming data from multiple various sources). I do not want to post all of the source code for the controller as there is quite a bit, and some of it is sensitive, but I will post these two methods which handle the control initialization and method where it listens for incoming data :

private static void Initialize(
    string DevicePath,
    R3Controller.HIDP_CAPS Capabilities,
    R3Controller.HIDD_ATTRIBUTES Attributes ) {

    R3Controller.hRead = R3Controller.OpenDevice( DevicePath );
    R3Controller.hWrite = R3Controller.OpenDevice( DevicePath );

    R3Controller.fsRead = new FileStream( hRead, FileAccess.ReadWrite, Capabilities.OutputReportByteLength, false );
    R3Controller.fsWrite = new FileStream( hWrite, FileAccess.ReadWrite, Capabilities.OutputReportByteLength, false );

    if ( R3Controller.fsRead.CanRead ) {
        R3Controller.barData = new byte[R3Controller.devCapabilities.Value.InputReportByteLength];
        if ( R3Controller.fsRead.CanRead )
            R3Controller.fsRead.BeginRead( R3Controller.barData, 0, R3Controller.barData.Length,
                new AsyncCallback( R3Controller.Listen ), R3Controller.barData );
        else
            throw new Exception( "R3 Controller Can't Read Incoming Data" );
    }
}

private static void Listen( IAsyncResult IAR ) {
    R3Controller.fsRead.EndRead( IAR );
    if ( R3Controller.fsRead.CanRead )
        R3Controller.fsRead.BeginRead( R3Controller.barData, 0, R3Controller.barData.Length,
            new AsyncCallback( R3Controller.Listen ), R3Controller.barData );
    else
        throw new Exception( "R3 Controller Can't Read Incoming Data" );
    R3InputEventArgs Args = new R3InputEventArgs( R3Controller.barData );
    if ( R3Controller.Locked || R3Controller.LockedControllers.Contains( Args.Controller ) ) {
        //Respond to locked presses if necessary...
        if ( R3Controller._LockedFeedback != null )
            R3Controller._LockedFeedback( null, Args );
        /*GetInvocationList( ).ToList( ).ForEach( E => (
            E.Clone( ) as EventHandler<R3InputEventArgs> ).BeginInvoke( null, Args, R3Controller.Heard, null ) );*/
    } else if ( R3Controller._ButtonPressed != null )
        R3Controller._ButtonPressed(null, Args);/*.GetInvocationList( ).ToList( ).ForEach(
            E => ( E.Clone( ) as EventHandler<R3InputEventArgs> ).BeginInvoke( null, Args, R3Controller.Heard, null ) );*/
}
Will
  • 3,413
  • 7
  • 50
  • 107
  • 2
    Since you mention this only happening when you perform operations very quickly, it could be a concurrency issue. Have you looked into `ConcurrentDictionary`? – cost May 03 '15 at 20:43
  • 1
    If your key is 3 bytes, you may want to generate the hash code differently: `(((int)key[0] << 16) | key[1] << 8 | key[2]).GetHashCode();` But as @cost indicated, this does not seem to be a problem related to the comperator. More likely a threading race condition. – Alex May 03 '15 at 20:46
  • Are you only reading or are you simultaneously writing too? If so, you should put that in the question. The code in there now doesn't enable us to reproduce the issue. – Patrick Hofman May 03 '15 at 20:54
  • ... okay; i'm confused. I thought this was a legitimate question, why all the down votes? – Will May 03 '15 at 20:59
  • 1
    I'm not a downvoter, but maybe people think there is a lack of information. Is multi-threading involved? Is the Dictionary being updated at the same time as it is being accessed? – RenniePet May 03 '15 at 21:01
  • Some people didn't read what the down votes are for :( I voted up. I wonder why you must use array as the key? cause the key itself will be the "pointer" to the array, not the array itself. – Ziv Weissman May 03 '15 at 21:02
  • 1
    The reason for an array is because the home-brewed hardware is... shoddy. When it was created eons ago, the person who created it didn't know binary and so the settings are all out of whack. I could have used basic math to interpret incoming data, but because the incoming data is all screwy, it's impossible and must be hard coded, and it makes me hate my life. – Will May 03 '15 at 21:04
  • @Will have you tried a different key data type as a test? If you do the same with a value type (like int, or long), does it behave the same? If certainly should help isolate the issue. – psubsee2003 May 03 '15 at 21:05
  • If multi-threading is involved then it might help if you put "volatile" on the declaration of the Dictionary. – RenniePet May 03 '15 at 21:09
  • @psubsee2003 : I can do that for the first two sets (10 entries), which are the only values against which I am querying anyway. Will let you know in a second but as people have indicated, this is likely a threading issue... – Will May 03 '15 at 21:23
  • @psubsee2003 : As you suggested, I changed the incoming data from a byte[] to an Int16 via the BitConverter.ToIn16(foo[], 1) method - The stress test resulted in the same error. – Will May 03 '15 at 21:33
  • @Will that seems to suggest the threading is the issue. I only suggested it to eliminate anything else. – psubsee2003 May 03 '15 at 21:35
  • @RenniePet : I tried adding the volatile keyword : No love. – Will May 03 '15 at 21:35
  • @psubsee2003 Thanks; it did eliminate the possibility of the issue being with the dictionary itself. – Will May 03 '15 at 21:36
  • Is it possible that the 3-byte key that you get in C# can be corrupted at a lower level, like in a driver written in C++ or something like that? So in a stress situation you get a 3-byte key that is actually one byte from one key and two bytes from the previous key? – RenniePet May 03 '15 at 21:51
  • @RenniePet : Maybe? I don't know, but if that is the case, it's on microsoft since all the low level stuff is handled by P/Invoke via the HID.DLL, SetupAPI.dll, and Kernel32.DLLs. I WISH we had our own software for this hardware. However; I have looked at the incoming byte[] that is making this explode, and I can confirm that the value is [0x00, 0x00, 0x63], which is present as the 9th element in the dictionary. – Will May 03 '15 at 21:58
  • OK, so it's an HID device on a USB connection? But that does not rule out the possibility that a homebrew driver is involved. Some HID devices use the generic Windows HID driver, but many install their own HID driver. EDIT: Just saw your edit to your comment - forget this. – RenniePet May 03 '15 at 22:01
  • Ours uses the Window HID driver. There was an archaic driver used back in the days our company dealt primarily in VB6 (may they die in a fire), however the source code was never released to us for that driver... although it may be the case that what you are saying is true, and if that is the case, are we just totally hosed when dealing with this scenario? Also : If what you are saying were the case, why would I be seeing the incoming data as [0x00, 0x00, 0x63] and not something garbled? Is it possible for 0x63 != 0x63? – Will May 03 '15 at 22:04
  • Sorry, I'm probably wasting your time. Is it possible that those 3 bytes are being modified by P/Invoke and Marshalling code at the same time as they are being used by Dictionary as a key? When Dictionary throws that exception, are the 3 bytes still the same as they were on their way into the Dictionary code? (If you can see what I mean ...) – RenniePet May 03 '15 at 22:13
  • @RenniePet : I don't think that's possible considering all byte[]s within the dictionary are their own separate instances of a byte[]... or I don't get what you mean... – Will May 03 '15 at 22:15
  • No, I mean the 3-byte key that you're using to do the lookup. – RenniePet May 03 '15 at 22:16
  • @RenniePet : I do not think so. The bit of code responsible for reading the controller data into a byte array is only ever called once per fire, and then that array is used to build the custom EventArgs. – Will May 03 '15 at 22:28
  • Sorry I'm going on and on about this ... "and then that array is used to build the custom EventArgs". You realize that the array as such is not copied to the R3InputEventArgs object? It contains a _reference_ to R3Controller.barData. So if R3Controller.barData is getting overwritten asynchronously, one byte at a time, while the Dictionary code is using it as a key, bad things could happen. – RenniePet May 03 '15 at 23:20
  • Okay - so to be absolutely certain that this is prevented, I could call arr.CloneValue() or something on it... I will put that into place and test it. – Will May 04 '15 at 00:27
  • @Will How did your testing go? Did you get this problem worked out? – cost May 05 '15 at 22:08
  • @cost Tests proved nothing good. I tried what was suggested about copying an array of the incoming data and sending that and it did not work. Tried with a concurrentdictionary, didn't work. I will just have to implement an error-catch that sends data indicating something did not work as it should have when this particular occurrence transpires. I have also been informed it may be a limitation of the hardware. – Will May 06 '15 at 22:48
  • @Will Maybe this is a terrible idea, but could you try converting the array into a standard four byte integer? Then you could have a dictionary that uses integers as keys – cost May 09 '15 at 00:34
  • @cost Wouldn't help : The problem has nothing to do with the key; it's a threading issue... – Will May 09 '15 at 02:03

1 Answers1

1

If you're performing a read and a write operation at the same/close time, the Dictionary class by itself isn't thread safe. So if lots of threads are accessing it very quickly, you can run into all sorts of interesting issues. Since you mention that this only happens when you perform operations very quickly, there's a good chance that could be the problem. To my knowledge, performing only reads with no writes shouldn't cause any trouble with Dictionary.

If you're using .NET 4+, there's a thread safe version of Dictionary called ConcurrentDictionary, which is made just for situations like these.

cost
  • 4,420
  • 8
  • 48
  • 80
  • Reading a dictionary is thread safe as long there are no write operations. – Patrick Hofman May 03 '15 at 20:53
  • @PatrickHofman You are correct. The asker didn't specify if they were adding new records or not, though I will edit my answer to be more specific – cost May 03 '15 at 20:56
  • The dictionary is only read from; never written to except during initialization... although the dictionary itself is stored as a static value within some EventArgs that are being created with rapid succession. That is likely the problem (and in which case a very stupid and amateurish mistake on my part). I will try with a regular dictionary outside of the EventArgs class I created. – Will May 03 '15 at 21:02
  • If the events aren't synchronous, there isn't an issue. There is if they are asynchronous... – Patrick Hofman May 03 '15 at 21:07
  • They are most certainly Asynchronous. – Will May 03 '15 at 21:36
  • @cost : I have tested the scenario with the concurrent dictionary : No love, it still exploded. – Will May 03 '15 at 22:07
  • @Will When the bug happens, is it a different key or the same key every time? – cost May 03 '15 at 22:32
  • @cost Same key : [0x00 0x00 0x63]. Though; I would imagine if I had 6 hands and enough stamina I might be able to provoke the bug on another controller... let me test with a different setting... EDIT : Quick text revealed that the issue presents consistently with different devices. – Will May 03 '15 at 22:43