Error writing to MongoDB using Parallelism

Question

I have a collection in mongo that has subdocuments, then read xml files and they'll record in MongoDB. Each xml file is a document in mongo.

My classes

public class Header
{
    public Header()
    {
        Operation= new List<Operation>();
    }

    public ObjectId Id { get; set; }
    public Int64 Code1 {get; set;}
    public Int64 Code2 {get; set;}
    public string Name { get; set; }
    public List<Operation> Operations { get; set; }
}

public class Operation
{
    public Operation()
    {
        Itens = new List<Item>();
    }

    public string Value { get; set; }
    public List<Item> Item { get; set; }
}

public class Item
{
    public string Value { get; set; }
}

Headline in the class, and Codigo2 Code1 used to create the index of the document in MongoDB. Code1 and Codigo2 compose the XML file name, and since they are all in one folder there is no possibility of being duplicated.

Recording in MongoDB

To record in mongo am using the following code:

var po = new ParallelOptions() { MaxDegreeOfParallelism = Environment.ProcessorCount > 1 ? Environment.ProcessorCount / 2 : 1 };

Parallel.ForEach(arquivos, po, (arquivo, state) =>
{

    MongoCollection<Header> collection = MongoConnect.GetHeader();

            try
            {
                var Header = new Header();
                Header.Name = @"Valor 1";
                Header.Code1 = arquivo.Name.Split('-').Count() > 1 ? Int64.Parse((arquivo.Name.Split('-')[1]).Replace(".", "")) : 0;
                Header.Code2 = arquivo.Name.Split('-').Count() > 1 ? Int64.Parse((arquivo.Name.Split('-')[2]).Replace(".siag", "")) : 0;

                var body = record.SelectSingleNode("body");
                if (body != null)
                {
                    string[] linhas = body.InnerText.Split(new String[] { "\r\n" }, StringSplitOptions.RemoveEmptyEntries);
                    foreach (var linha in linhas)
                    {
                        string conteudo = linha;
                        var operation = new Operation();
                        if (!conteudo.Contains("\t"))
                        {
                            string tipo = conteudo.Substring(0, conteudo.IndexOf(' ')).Trim();
                            string tabela = conteudo.Substring(0, conteudo.IndexOf("Quando:", System.StringComparison.Ordinal)).Trim();
                            operation.value = tabela;
                            conteudo = conteudo.Remove(0, (tabela + " Quando:").Length);

                            Header.Operations.Add(operation);
                        }
                        else
                        {
                            var item = new Item();
                            string[] campos = conteudo.Split('\t');
                            item.Value = campos[0];

                            Header.Operations.Last().Itens.Add(item);
                        }
                    }

                    try
                    {
                        collection.Save(Header);
                    }
                    catch (Exception ex)
                    {
                        //Duplicate Key error show here
                    }
                }
            }
            catch (Exception ex)
            {
                //Log Error Here
            }

});

Note: Please, no consider read of the file, put there just to illustrate.

Full Error

Erro: WriteConcern detected an error ''. (Response was { "ok" : 1, "code" : 11000, "err" :
 "insertDocument :: caused by :: 11000 E11000 duplicate key error index:
 DB.Collection.$Codigo1_1_Codigo2_1  dup key: { : 359922397, : 1217185957 }", 
"n" : NumberLong(0) })

Do you let the driver create ObjectIds automatically for you? — i3arnon, Sep 04 '14 at 11:54

MarkKGreenway · Accepted Answer · 2014-09-04T20:15:27.463

2

The reason is this : source code

shows the objectId gets created :

static ObjectId()
    {
        __staticMachine = (GetMachineHash() + AppDomain.CurrentDomain.Id) & 0x00ffffff; // add AppDomain Id to ensure uniqueness across AppDomains
        __staticIncrement = (new Random()).Next();

        try
        {
            __staticPid = (short)GetCurrentProcessId(); // use low order two bytes only
        }
        catch (SecurityException)
        {
            __staticPid = 0;
        }
    }

But if you run :

  var po = new ParallelOptions() { MaxDegreeOfParallelism = Environment.ProcessorCount > 1 ? Environment.ProcessorCount / 2 : 1 };

        var items = new List<string>()
        {
            "Foo",
            "Bar"
        };
        Parallel.ForEach(items, po, (arquivo, state) =>
        {
            Console.WriteLine((new Random()).Next());
        });

You get :

  1259271181
  1259271181

Because the random does not do well paralellized. You have to either define an Id that isn't using ObjectId. Or Make it threadsafe

From our comments I would create the Header Class Like :

public class Header
{
   public Header()
   {
      Operation= new List<Operation>();
   }
   [BsonId]
   public Codes Id {get; set;}
   public Int64 Code2 {get; set;}
   public string Name { get; set; }
   public List<Operation> Operations { get; set; }
}
public class Codes {
   public Int64 Code1 {get; set;}
   public Int64 Code2 {get; set;}
}

edited Sep 04 '14 at 20:15

answered Sep 04 '14 at 13:16

MarkKGreenway

8,494
5
34
53

How do I fix this? How can I create the Id without using the ObjectId type? I Already tried use threadsafe, but not resolve my problem. – Renatto Machado Sep 04 '14 at 18:25
I do one test, remove the index of the my collection and run with parallel, after I go in mongodb and the ObjectId of the my collection is not duplicate. – Renatto Machado Sep 04 '14 at 19:26
So Random only gives the same one in parallel sometimes because the rand uses the current time to generate. I would use meaningful key, is Name unique? – MarkKGreenway Sep 04 '14 at 20:07
So I created a composite key for you when I updated this Answer. – MarkKGreenway Sep 04 '14 at 20:15
So, but I tried remove the index and run with parallelism for see if was duplicate key, but key was not duplicate. So, it seems not be the that you say. – Renatto Machado Sep 05 '14 at 10:12
Checked as solotion, because say: "Or Make it threadsafe". **ThreadSave was the solution!** – Renatto Machado Sep 16 '14 at 18:20

Error writing to MongoDB using Parallelism

My classes

Recording in MongoDB

Full Error

1 Answers1