Loading large amount of text files from a share drive to search for a world. C#

Question

Im trying to perform a search in a share drive that contains at least 90,000 files.

The code below is what I have:

  private void button2_Click(object sender, EventArgs e)
  {

        if (string.IsNullOrEmpty(textBox1.Text))
        {
            MessageBox.Show("No folder: Not listed", "Error", MessageBoxButtons.OK, MessageBoxIcon.Warning);
        }
        else
        {

            List<string> allFiles = new List<string>();
            AddFilesnames(sourceFolder, allFiles);

            foreach (string fileName in allFiles)
            {
                string contents = File.ReadAllText(fileName);
                if (contents.Contains(searchword))
                {
                    listBox1.BeginUpdate();
                    listBox1.Items.Add(fileName);
                    listBox1.EndUpdate();
                    label4.Text = (Convert.ToInt32(label4.Text) + 1).ToString();
                }

            }

            if (listBox1.Items.Count == 0)
            {
                MessageBox.Show("no files");
            }
        }
    }

    public void listboxtofile()
    {
        DialogResult resDialog = dlgSaveFile.ShowDialog();
        if (resDialog.ToString() == "OK")
        {
            FileInfo fi = new FileInfo(dlgSaveFile.FileName);
            StreamWriter sw = fi.CreateText();
            foreach (string sItem in listBox1.Items)
            {
                sw.WriteLine(sItem);
            }
            sw.Close();
        }
    }

    public  void AddFilesnames(string sourceDir, List<string> allFiles)
    {
        string[] fileEntries = Directory.GetFiles(sourceDir);
        foreach (string fileName in fileEntries)
        { 
            DateTime from1 = dateTimePicker1.Value.Date;
            DateTime to1 = dateTimePicker2.Value.Date;
            DateTime creationTime = File.GetCreationTime(fileName);

            if (creationTime >= from1 && creationTime <= to1)
            {
                allFiles.Add(fileName);
            }               

        }

        //Recursion   
        string[] subdirectoryEntries = Directory.GetDirectories(sourceDir);
        foreach (string item in subdirectoryEntries)
        {
            // Avoid "reparse points"
            if ((File.GetAttributes(item) & FileAttributes.ReparsePoint) != FileAttributes.ReparsePoint)
            {
                AddFileNamesToList(item, allFiles);
                label4.Text = (Convert.ToInt32(label4.Text) + 1).ToString();
            }
        }
    }

    private void button3_Click(object sender, EventArgs e)
    {
        listBox1.Items.Clear();

        textBox1.Clear();
    }

    private void button5_Click(object sender, EventArgs e)
    {
        listboxtofile();
    }
  }
}

So far this is working with 3,000 files but I need to have access to a shared drive that contains 90,000 files and when I try to search in the share drive the windows form get frozen. I will really apreciate any help from you all.

Look at using the BackgroundWorker class (http://msdn.microsoft.com/en-us/library/system.componentmodel.backgroundworker.aspx). — Plymouth223, Aug 07 '13 at 03:03
The issue is classic UI responsiveness and has nothing to do with data-access. You are doing all your work on the UI thread, and thus the UI can't update. — Aron, Aug 07 '13 at 03:06

score 0 · Answer 1 · answered Aug 07 '13 at 03:05

0

My suggestion would be to do any file system work in a separate thread. If you do the file searching in the same thread as your form, Windows has no chance to update the form. You won't be able to update any of your form controls (like listBox1) directly from the new thread, but you can look into delegates and C# threading in general for ways to work around this.

Also, if you just need to search files for a specific word, have you looked into existing tools like grep?

answered Aug 07 '13 at 03:05

nhellwig

106
3

I agree that this project sounds to be reinventing the wheel. However it is also a good exercise in good UI design. There is a lot of work that can be done to produce a good asynchronous data flow pattern. I would suggest that you look at writing this in TPL-Dataflow or Reactive Extensions as an exercise in UI design, but actually use an existing tool like grep. – Aron Aug 07 '13 at 03:27

Loading large amount of text files from a share drive to search for a world. C#

1 Answers1