-6

Ok here i am going to list objects of my software. Currently memory usage is increasing by the time passes, though it should not increase because i am not keeping any resources. Using only database.

c# 4.0 , visual studio 2010

Lets start with objects. On these objects should i ever call dispose or use "using"

string variable;
int variable;
dataset variable;
HtmlDocument variable;
List<string> variable;
HtmlNode variable;
Uri variable;
DateTime variable;
HtmlWeb variable;
Regex variable;
MatchCollection variable;
bool variable;

piece of code (anything required after File.WriteAllText ? method)

                File.WriteAllText("hatalar/" + UtcTimeNow.Ticks.ToString() + GenerateRandomValue.GenerateRandomValueDefault(10000000) + ".txt", srCrawledUrl + " unknown error page id " + srPageId);

piece of code

                if (irFinishedLast > -1)
            {
                var newTask = Task.Factory.StartNew(() =>
                {
                    fcStartSubPageCrawl(srMainSiteURL, srMainSiteId, irWhichMainTask);
                });
                lock (lockerMainSitesArray)
                {
                    if (MainSitesTaskList[irWhichMainTask, irFinishedLast] != null)
                        MainSitesTaskList[irWhichMainTask, irFinishedLast].Dispose();
                    MainSitesTaskList[irWhichMainTask, irFinishedLast] = newTask;
                }
            }

Alright now classes and functions. Public static function which is being called by many threads at the same time. It is inside public static class.

public static string srInserIntoPagesCommand = "insert into myTable (PageUrl,MainSiteId,CrawlDateInt,CrawlDateChar,CrawlDepth,ExtractedPageId,CrawlStatus) values " +
            "(@PageUrl,@MainSiteId,@CrawlDateInt,@CrawlDateChar,@CrawlDepth,@ExtractedPageId,@CrawlStatus)";

        public static bool InsertIntoPages(string PageUrl, string MainSiteId, string CrawlDateInt, string CrawlDateChar, string CrawlDepth, string ExtractedPageId, string CrawlStatus)
        {
            string srPageUrl = PageUrl;
            string srMainSiteId = MainSiteId;
            string srCrawlDateInt = CrawlDateInt;
            string srCrawlDateChar = CrawlDateChar;
            string srCrawlDepth = CrawlDepth;
            string srExtractedPageId = ExtractedPageId;
            string srCrawlStatus = CrawlStatus;

            if (srCrawlDateInt.Length < 1)
                srCrawlDateInt = "0";
            if (srCrawlDateChar.Length < 1)
                srCrawlDateChar = "null";
            if (srCrawlStatus.Length < 1)
                srCrawlStatus = "0";

            using (SqlConnection connection = new SqlConnection(DbConnection.srConnectionString))
            {
                using (SqlCommand cmd = new SqlCommand(srInserIntoPagesCommand, connection))
                {
                    cmd.CommandType = CommandType.Text;
                    cmd.Parameters.AddWithValue("@PageUrl", srPageUrl);
                    cmd.Parameters.AddWithValue("@MainSiteId", srMainSiteId);
                    cmd.Parameters.AddWithValue("@CrawlDateInt", srCrawlDateInt);
                    cmd.Parameters.AddWithValue("@CrawlDateChar", srCrawlDateChar);
                    cmd.Parameters.AddWithValue("@CrawlDepth", srCrawlDepth);
                    cmd.Parameters.AddWithValue("@ExtractedPageId", srExtractedPageId);
                    cmd.Parameters.AddWithValue("@CrawlStatus", srCrawlStatus);
                    try
                    {
                        connection.Open();
                        cmd.ExecuteNonQuery();
                    }
                    catch (Exception E)
                    {
                        DateTime UtcTimeNow = DateTime.UtcNow;
                        File.WriteAllText("pageshatalar/" + UtcTimeNow.Ticks.ToString() + GenerateRandomValue.GenerateRandomValueDefault(1000000) + ".txt", "InsertIntoPages \r\n\r\n" + E.Message.ToString() + "\r\n\r\n" + srPageUrl);
                        return false;
                    }
                }
                connection.Close();
            }
            return true;
        }

public static database connection for select queries mostly inside public static class

public static string srConnectionString = "server=localhost;database=mydb;uid=sa;pwd=mypw; Max Pool Size=20000; Pooling=True;";

public static DataSet db_Select_Query(string strQuery)
{
    DataSet dSet = new DataSet();
    try
    {
        using (SqlConnection connection = new SqlConnection(srConnectionString))
        {
            connection.Open();
            using (SqlDataAdapter DA = new SqlDataAdapter(strQuery, connection))
            {
                DA.Fill(dSet);
            }
            connection.Close();
        }
        return dSet;
    }
    catch
    {
        DateTime UtcTimeNow = DateTime.UtcNow;
        File.WriteAllText("sqlhatalar/" + UtcTimeNow.Ticks.ToString() + GenerateRandomValue.GenerateRandomValueDefault(1000000) + ".txt", strQuery);
        return null;
    }
}
Furkan Gözükara
  • 22,964
  • 77
  • 205
  • 342
  • Your db_Select_Query() is the most likely culprit of memory usage, as it could return some very large datasets. From where do you call that method, and approximately how often? What do you do with those datasets once you get them? – drharris Oct 22 '11 at 02:20
  • actually i am only selecting top 1 row. so it is not the cause :) – Furkan Gözükara Oct 22 '11 at 02:43

3 Answers3

2

You don't randomly call dispose on any particular thing. The using statement and the .Dispose method are for objects that implement the interface IDisposable

 public interface IDisposable
 {
       void Dispose();
 }

A cursory inspection of your top variable list indicates that none of these variables implement IDisposable. You cannot invoke Dispose() on these, nor can you wrap them in the using statement.

Further, invoking Dispose() on actual disposable objects is not intended as a form of memory management, it is intended to release unmanaged resources. The garbage collector is not connected to this. If you have a problem with memory management, it might reveal that you are keeping too many objects alive for too long. You need to explore scoping and lifetimes, collection sizes, etc., as those are what will be adding to your garbage.

Your memory problem is not being revealed in your displayed code, except to say that you could be processing and returning large DataSets that your callers could further hold onto longer than necessary. Or it could be something else entirely unrelated to any of the code displayed. If you have a memory problem, run a memory profiler to identify the problem areas. To that end, you can get free trials of profilers from Red Gate or JetBrains.

Anthony Pegram
  • 123,721
  • 27
  • 225
  • 246
  • can you look the variable list and tell me which objects should i cover with "using" – Furkan Gözükara Oct 22 '11 at 02:09
  • 1
    As far as I can tell, `DataSet` is the only type that implements `IDisposable`, although it seems to be a degenerate call in most instances. But you should explore the documentation for the types that you are unsure about. Start at MSDN. – Anthony Pegram Oct 22 '11 at 02:16
  • But again, `Dispose` isn't going to solve your memory problem. This code doesn't point to any memory problem. But you undoubtedly have other code, you said you have multiple threads invoking this code. Run a memory profiler, see what resources are taking up the most memory and where. – Anthony Pegram Oct 22 '11 at 02:18
  • redgate is too expensive. any free method available ? – Furkan Gözükara Oct 22 '11 at 02:28
  • As I stated in my answer, both Red Gate and JetBrains offer free trials, unless they've recently changed policy. There are also answers on this website and Google regarding .NET profiling tools. A simple search should give you many leads regarding tools, pricing, and how to use them. – Anthony Pegram Oct 22 '11 at 02:30
  • and if i release my whole code would you bother check it ? it is about 400 lines – Furkan Gözükara Oct 22 '11 at 02:31
  • I'm sorry, but no, I would not review the entire code, I don't have the available time to devote to doing that. Please conduct the research on your own, it would be valuable to do so. – Anthony Pegram Oct 22 '11 at 02:35
  • right now i am checking with jetbrains but how i am supposed to find memory leaks. at each snapshot every object count keep increasing. – Furkan Gözükara Oct 22 '11 at 02:39
  • Find which objects keep being added. One question, are you storing anything in global state, such as using `static` variables? Are you putting large objects into Session (if ASP.NET) or Cache? This could cause your memory pressure to grow over time. – Anthony Pegram Oct 22 '11 at 03:04
  • yes i have static string values which are basically command texts of sql commands. not storing anything in session or cache. first i create 20 threads. each of that 20 threads then starts 9 threads more in infinite while loop. check each thread every 250 ms . i am using thread.sleep method for waiting. so at each 250 ms it checks finished threads and if there is finished thread it creates another one. but i call dispose on finished thread. it is using tpl. so at total i should have 200 threads and 1 main thread which makes 201 threads. but somehow it passes that on runtime. – Furkan Gözükara Oct 22 '11 at 03:08
  • here how i make infinite loop and keep certain number of threads alive all the time. this is the second thread function . so 20 thread starts this thread with tpl. http://pastebin.com/bsvah8DF – Furkan Gözükara Oct 22 '11 at 03:09
  • A handful of static strings would not concern me. Multithreading is not my specialty. I will only say that creating 200 threads seems a bit wasteful. How many processors do you have? Smarter people than me have said that it's not particularly useful to spawn more threads than you actually have CPU cores to run them, but like I said, multithreading is not my specialty, don't take my word for it. – Anthony Pegram Oct 22 '11 at 03:12
  • Threads *do* cause additional memory pressure, however. Each one will allocate a million bytes of virtual memory space immediately. So use only the true number of threads you need. – Anthony Pegram Oct 22 '11 at 03:15
  • @MonsterMMORPG, at this point, I would advise you to start a new question about your multithreading scenario, memory management, etc. The title and immediate feel of this question is about when to invoke `Dispose` and on what objects, and as useful is that is to know, that is not really the problem you are facing. Write a *new question* that focuses more on your actual problem, and you will probably attract the attention of the people that can answer it. – Anthony Pegram Oct 22 '11 at 03:20
  • they are web crawler threads. so basically not cpu core related. it is network related. wait time. i also have core i7 2600k @ 4.5 ghz. so i have 8x4,5 = 36000 MHZ cpu power in basic way. did you check the link i posted ? any problem there ? – Furkan Gözükara Oct 22 '11 at 03:46
  • yes they certainly will allocate memory. but the memory usage should not arise anymore after certain time since i don't allocate anything new. as previous thread finishes i call dispose and create new one. – Furkan Gözükara Oct 22 '11 at 03:48
1

You can't call Dispose() on most of the objects in your list because they do not expose a Dispose() method. You should call Dispose() on an object of any type that implements the IDisposable interface. That's it.

Ed S.
  • 122,712
  • 22
  • 185
  • 265
1

If for some bizarre reason you don't know at run-time if an object has dispose implemented, you can use this dispose-safe function:

/// ---- IsDisposable --------------------------------
///
/// <summary>
/// returns true if an object is disposable, false if not
/// you can optionally dispose of it immediately
/// </summary>

public static Boolean IsDisposable(Object Item, Boolean DeleteIfDisposable)
{
    if (Item is IDisposable)
    {
        if (DeleteIfDisposable)
        {
            IDisposable DisposableItem;
            DisposableItem = (IDisposable)Item;
            DisposableItem.Dispose();
        }
        return true;
    }
    else
        return false;
}
Steve Wellens
  • 20,506
  • 2
  • 28
  • 69