0

I've got my first batch of really massive data that I need for my ASP.NET web application. Up until now (for two years in fact!) nothing has come up that was so costly that I couldn't just do it on Page_Load or store it in ViewState or Session. But now, I need a list of objects that remains fairly static (gets updated once a day at most) and contains about 5000 objects, and building it every time on Page_Load causes the load time to take an obvious hit.

Some Googling regarding caching shows some very elementary examples using stuff like Cache["myStuff"] = myObject; and that's all well and good. But I'd like to know if there is some way I can set my Web Application to run an update at a specific time, regardless of whether or not the application is being accessed or in use.

For example, I want this process to update the list to run once a day at like 4:00am when no one is using anything. So let's say on day 1 in the afternoon, the Page_Load checks to grab the cached object, but it comes back null, so in that case, bite the bullet and request/build the list for the rest of day 1, store it in cache. Throughout the rest of day 1, no one has to build the object because it's in cache and it's still valid.

Day 1 ends, and it's now 4:00am on day 2. No one is using any pages. I want the object that was stored in cache from day 1 to be deleted and re-built, so that when people get to the office in the morning, the first person who accesses the page doesn't need to build the list for today, it's already done.

The only idea I have would be to look into creating a console application that runs on a Windows scheduled task or something. But even then, I'm not sure how I would be able to interact with the ASP.NET web application from an external application like that.

Hopefully I'm making sense. Is what I'd like to accomplish even possible?

CptSupermrkt
  • 6,844
  • 12
  • 56
  • 87
  • What's the purpose? (Trying to get an idea of the reasons behind it) 5000 objects is a lot(!) and I doubt if there is a user who can get a grasp of that amount of information. Is turning that info into pages an option? – lboshuizen Oct 31 '12 at 01:17
  • Yeah, it's not for your average everyday user, it's for our intranet application which is used by various teams to analyze data, so they're very used to large amounts of data. They go back and forth between a lot of data, and I just had this idea to use caching as a way to reduce the amount of redundant calls. Everyone is making massive queries to multiple data sources for thousands of records throughout the day, when all of it is static (at least for one day at a time) – CptSupermrkt Oct 31 '12 at 01:20
  • Just to clarify, none of this data used for the intranet application is modified. The intranet application is just for looking at the data. No one is doing any kind of writes or changes via the intranet web application. – CptSupermrkt Oct 31 '12 at 01:25
  • Caching is a way to reduce load on the backend (database-server) If they are making queries why not expose a way to make those queries possible "directly" against the backend? Your solution seems to be a file download of everything needed – lboshuizen Oct 31 '12 at 01:26
  • Yeah, in normal circumstances, I'd agree. But we're in a very weird/specific network environment where the data is spread across several different servers, and due to our lack of Active Directory, we can't access a server, and then from that server access another server. Internally we call this "the hop problem." So basically what we need to do is grab data from Server A, grab data from Server B, grab data from Server C, then manipulate it to fit the objects necessary for the page. This process takes longer than desired, putting noticable strain on page load times. – CptSupermrkt Oct 31 '12 at 01:31

3 Answers3

3

If you use a distributed cache like MemCached, you could create a Windows Service or Console application for populating the cache. Since both applications (The service and the web app) will use the same cache, it will solve your problem.

Another solution is to have a page just for that (i.e. /startcache.aspx) that will do the cachin on page_load. And you can schedule a request to that page. You could also hide the page from outside the web-app using some kind of firewall (so nobody accesses that page!). Finally, the good part of this solution is that you can reuse your current code, as it will work in the same project.

Hope it helps.

Ivo
  • 8,172
  • 5
  • 27
  • 42
2

Redis is another option for an out of proc 3rd party cache/key value store.

MatthewMartin
  • 32,326
  • 33
  • 105
  • 164
  • 1
    Redis is a great solution, with some timed service (check out Quartz.Net) that re-populates the Cache at set intervals. Depending on what you're trying to do you may find some real value in RavenDB by persisting the processed results as a sort of "cache" – mellodev Oct 31 '12 at 01:25
0

To reduce load on the server and backend and serving all daily data (as I understand from the question)

Why not create a data file (CSV,XML,JSON) and offer that as a data-point.

From your question it seems that the client(application) is capable of running their own queries against that dataset.

Creating a file like that can be done on a fixed time set, saved and as such just one time. No need to include thirt-party solutions (besides their respectable qualities)

lboshuizen
  • 2,746
  • 17
  • 20