Big Data Xml file (file size over 20GB) convert to Json File

Question

I have an XML file. I want to convert it to JSON with C#. However, the XML file is over 20 GB.

I have tried to read XML with XmlReader, then append every node to a JSON file. I wrote the following code:

var path = @"c:\result.json";
TextWriter tw = new StreamWriter(path, true, Encoding.UTF8);
tw.Write("{\"A\":");

using (XmlTextReader xmlTextReader = new XmlTextReader("c:\\muslum.xml"))
{
    while (xmlTextReader.Read())
    {
        if (xmlTextReader.Name == "A")
        {
            var xmlDoc = new XmlDocument();
            var v = xmlTextReader.ReadInnerXml();

            string json = Newtonsoft.Json.JsonConvert.SerializeXmlNode(xmlDoc, Newtonsoft.Json.Formatting.None, true);

            tw.Write(json);
        }
    }
}

tw.Write("}");
tw.Close();

This code not working. I am getting error while converting json. Is there any best way to perform the conversion?

Have you looked at XDocument which you can query xml with linq. It will make your task easier — Murdock, Jan 27 '14 at 15:10
What's the error? Try to process a smaller file (let's say two nodes) to see what's wrong with that JSON (for example: if your loop produces an array...where is it open/closed?) — Adriano Repetti, Jan 27 '14 at 15:10
Have you try the solution of this question : http://stackoverflow.com/questions/7711818/storing-large-xml-in-mongodb (kind of double content) — aloisdg, Jan 27 '14 at 15:11
Here's one error `@"c:\\result.json"` you mean either `@"c:\result.json"` or `"c:\\result.json"` — weston, Jan 27 '14 at 15:11
mongo has a max size of 16MB for a document. are you sure you want to convert 20GB to json? — i3arnon, Jan 27 '14 at 15:23
@I3arnon, I want to use I free and fast database. So I decided to use mongodb. What do you prefer to me. — Müslüm ÖZTÜRK, Jan 28 '14 at 06:17
@vittore Yes I have. Before Importing to db, I is validating with xsd schema. — Müslüm ÖZTÜRK, Jan 28 '14 at 06:34

vittore · Answer 1 · 2014-01-28T12:10:48.990

I would do it the following way

generate classes out of xsd schema using xsd.exe
open file and read top level ( i e your document level ) tags one by one (with XmlTextReader or XmlReader)
serialize each tag into object using generated classes
deserialize resulting object to json and save to whatever
consider saving in batches of 1000-2000 tags
you are right about serialize/deserialize being slow. still doing work in several threads, preferably using TPL will give you good speed. Also consider using json.net serializer, it is really a lot faster than standard ones ( it is standard for web.api though)

I can put some code snippets in the morning if you need them. We are processing big ( 1-10gigs) files this manner in order to save data to sql server database.

I thought this. But serialize and deserialize processes take long time. So Importing xml to db get very long time. I asked the question find the fastest way. I will be happy if you share code snippets. Thanks — Müslüm ÖZTÜRK, Jan 28 '14 at 06:51

Big Data Xml file (file size over 20GB) convert to Json File

1 Answers1