TextWriter not writing all files

Question

I've got written a service that has a separate thread running that reads roughly 400 records from a database and serializes them into xml files. It runs fine, there are no errors and it reports all files have been exported correctly, yet only a handful of xml files appear afterwards, and its always a different number each time. I've checked to see if it's a certain record causing problems, but they all read out fine, and seem to write fin, but don't...

After playing around and putting a delay in of 250ms between each write they are all exported properly, so I assume it must have something to do with writing so many files in such a quick succession, but I have no idea why, I would have thought it would report some kind of error if they didn't write properly, yet there's nothing.

Here is the code for anyone who wants to try it:

static void Main(string[] args)
{
    ExportTestData();
}

public static void ExportTestData()
{
    List<TestObject> testObjs = GetData();

    foreach (TestObject obj in testObjs)
    {
        ExportObj(obj);
        //Thread.Sleep(10);
    }
}

public static List<TestObject> GetData()
{
    List<TestObject> result = new List<TestObject>();

    for (int i = 0; i < 500; i++)
    {
        result.Add(new TestObject()
        {
            Date = DateTime.Now.AddDays(-1),
            AnotherDate = DateTime.Now.AddDays(-2),
            AnotherAnotherDate = DateTime.Now,
            DoubleOne = 1.0,
            DoubleTwo = 2.0,
            DoubleThree = 3.0,
            Number = 345,
            SomeCode = "blah",
            SomeId = "wobble wobble"
        });
    }

    return result;
}

public static void ExportObj(TestObject obj)
{
    try
    {
        string path = Path.Combine(@"C:\temp\exports", String.Format("{0}-{1}{2}", DateTime.Now.ToString("yyyyMMdd"), String.Format("{0:HHmmssfff}", DateTime.Now), ".xml"));
        SerializeTo(obj, path);
    }
    catch (Exception ex)
    {

    }
}

public static bool SerializeTo<T>(T obj, string path)
{
    XmlSerializer xs = new XmlSerializer(obj.GetType());
    using (TextWriter writer = new StreamWriter(path, false))
    {
        xs.Serialize(writer, obj);
    }
    return true;
}

Try commenting\uncommenting the Thread.Sleep(10) to see the problem

Does anybody have any idea why it does this? And can suggest how I can avoid this problem?

Thanks

EDIT: Solved. The time based filename wasn't unique enough and was overwriting previously written files. Should've spotted it earlier, thanks for your help

@Lasse Path's passed in with a unique filepath for each file — Iain Ward, Jan 25 '11 at 10:59

score 1 · Answer 1 · answered Jan 25 '11 at 10:57

1

Perhaps try putting the writer in a using block for immediate disposal? Something like

XmlSerializer xs = new XmlSerializer(obj.GetType());
using(TextWriter writer = new StreamWriter(path, false)) 
{
    xs.Serialize(writer, obj);
}

answered Jan 25 '11 at 10:57

Tom

3,354
1
22
25

You don't need the Close call if you're calling Dispose (via the using statement) – Jon Skeet Jan 25 '11 at 10:58
What is explaining this behavior? I don't understand why not disposing the writer would cause his problem. Can you explain please ? – Shimrod Jan 25 '11 at 10:59
The disposal of the text writer is letting the OS know that the resource (file) is no longer needed. Holding onto the file without disposing (especially if you're doing loads on a thread) can prevent the file from being written properly and freeing it for actual writing. – Tom Jan 25 '11 at 11:03
@Tom: But unless there was an exception, the StreamWriter would have been closed anyway. It's a good idea to use a using statement, but I don't believe this is the problem. – Jon Skeet Jan 25 '11 at 11:57
@Jon yes you're right, it was suspicious at best, at least it's all sorted now :) – Tom Jan 25 '11 at 13:19

score 1 · Accepted Answer · answered Jan 25 '11 at 11:55

1

Ok I've found the problem, I was using a time based filename that I thought would be unique enough for each file, turns out in a loop that tight they're coming out with the same filenames and are over-writing each other.

If I change it to use actually unique filenames it works! Thanks for your help

answered Jan 25 '11 at 11:55

Iain Ward

9,850
5
34
41

Found same 1 minute ago. Next time create Unit Tests before implementing a feature. :) – Vasyl Boroviak Jan 25 '11 at 12:04

score 0 · Answer 3 · answered Jan 25 '11 at 10:57

0

Dispose the writer

public static bool SerializeTo<T>(T obj, string path)
{
    XmlSerializer xs = new XmlSerializer(obj.GetType());
    using(TextWriter writer = new StreamWriter(path, false)) {
        xs.Serialize(writer, obj);
        writer.Close();
    }
    return true;
}

answered Jan 25 '11 at 10:57

djeeg

6,685
3
25
28

Why not write.Dispose();// after writer.Close(); – Pratik Jan 25 '11 at 11:01

score 0 · Answer 4 · answered Jan 25 '11 at 11:00

0

If you're not getting any exceptions, then the using statements proposed by other answers won't help - although you should change to use them anyway. At that point, you don't need the close call any more:

XmlSerializer xs = new XmlSerializer(obj.GetType());
using(TextWriter writer = new StreamWriter(path, false))
{
    xs.Serialize(writer, obj);
}

I don't think the problem lies in this code, however. I suspect it's something like the "capturing a loop variable in a lambda expression" problem which crops up so often. If you can come up with a short but complete program which demonstrates the problem, it will be a lot easier to diagnose.

I suggest you create a simple console application which tries to create (say) 5000 files serializing some simple object. See if you can get that to fail in the same way.

answered Jan 25 '11 at 11:00

Jon Skeet

1,421,763
867
9,128
9,194

Thanks Jon, I've taken your advice and recreated the problem in a test app. I've updated my question to include a link to download it – Iain Ward Jan 25 '11 at 11:50
@w69rdy: It's a pain to download it from somewhere else... how big is it? I'd expect it to be easily under 100 lines - you should include it in the question. – Jon Skeet Jan 25 '11 at 11:56
Good point, so I've included it in the question now. When I stripped it down it became a bit clearer why it wasn't working, turns out my way of generating 'unique' filenames really isn't that clever.. – Iain Ward Jan 25 '11 at 12:03

score 0 · Answer 5 · answered Jan 25 '11 at 11:06

0

Multi-threading may cause that problem. The 250ms delay is an evidence of that.

Do you have multiple threads doing that?

answered Jan 25 '11 at 11:06

Vasyl Boroviak

5,959
5
51
70

I do have other threads running at the same time, but I've tried it without and it doesn't make a difference. I've included a test app in my questions now if you'd like to see the issue in action – Iain Ward Jan 25 '11 at 11:51

TextWriter not writing all files

5 Answers5