1

I had developed a software in C# using Windows Forms to scrape selected websites for images.

  1. First problem I have is that the websites I monitor constantly change their look and feel, thus making my code in need for updating. I had switched to using XPaths to isolate the divs I look for, but the div ids change too. I have thought of using a text file with the div xpath for each site which the software would read thru, thus saving me the time to edit and recompile the code. Is there a better way to solve this problem ? Maybe CodeDom ?

  2. Secondly, since every website uses different formatting and encoding I had to rewrite parts of code with the HtmlDocument, HtmlWebResponse, HtmlNodes and others for each of them, which ended up accounting for nearly half of my code. I could not put them together since some need extra scraping and paginating and some do not. Is there a way make to simplify this problem ?

  3. Lastly, I have the whole code in one class file with around 600 lines of code. The only methods I have are the backgroundworkers, ui event handlers, a scraping method each for each site, and one method to save the images. Is it alright to have the whole code in one class ? When I used to write in Java, I used to often make use of multiple classes and call them as objects, this helped making changes to particular sections easier. Can I do the same with C# ?

Is there a more efficient approach to making the software ? I was thinking of making a class for each site, so that modifications could be done directly to the class in question, but that would cause a lot of lines to be repeated in each class. Or is it okay to have the whole in one class file ?

Thanks.

PS: This software is for personal use, but I think it is a good opportunity to learn and apply good programming.

Sam Hosseini
  • 813
  • 2
  • 9
  • 17
Monk
  • 69
  • 1
  • 7
  • 3
    This stuff always comes down to identifying patterns and isolating functionality. You have to look at specific functionality and group that functionality into methods and maybe classes depending. I think, if the code isn't proprietary you could get a lot of help here: http://codereview.stackexchange.com/ – Rig Jul 31 '11 at 02:45
  • 2
    "I have the whole code in one class file with around 600 lines of code" - separation of concerns: meaning isolate related functionality into classes. Makes code more managable (unless you take it too far (not likely with 600 lines of code; thinking more like 600,000 lines of code!)) – Mitch Wheat Jul 31 '11 at 02:58
  • 1
    1. Using a separate data store, outside of your code is definitely good practice. You could drop your strings into a project "resource" file (.resx). 3. I'd suggest writing a "Scraper" Interface, and one scraper class per site. Move the site-specific code into a method in the class such as DoScrape(). Then in your main program code you can instantiate each scraper and call myScraper.DoScrape() or whatever you need. Definitely do not repeat any code between classes -- if you have common functionality, move it to a "Helper" class in multiple small methods so that it can be used everywhere. – Ash Eldritch Jul 31 '11 at 03:52
  • @ Rig and Mitch: Thanks a ton for the inputs. Glad to know that splitting to classes was good. I will try eliminate repetitions. @Ash: Thanks for the details. Helper file and a Scraper interface does seem like a great idea. Will try it out. Thanks again. – Monk Jul 31 '11 at 04:16
  • If you're not already using it, the HTMLAgility pack might be worth looking at. – Derek Beattie Jul 31 '11 at 04:36
  • I realized that it was the best HTML library when I stumbled upon it when I was having trouble scraping. I was really impressed by its ease of use. Thanks for pointing it out anyways. It is great library. – Monk Jul 31 '11 at 04:42

0 Answers0