0

I am building a website (probably in Wordpress) which takes data from a number of different sources for display on various pages.

The sources:

  • A Twitter feed
  • A Flickr feed
  • A database on a remote server
  • A local database

From each source I will mainly retrieve

  • A short string, e.g. for Twitter, the Tweet, and from the local database the title of a blog page.
  • An associated image, if one exists
  • A link identifying the content at its source

My question is:

What is the best way to a) store the data and b) retrieve the data

My thinking is:

i) Write a script that is run every 2 or so minutes on a cron job
ii) the script retrieves data from all sources and stores it in the local database
iii) application code can then retrieve all data from the one source, the local database

This should make application code easier to manage - we only ever draw data from one source in application code - and that's the main appeal. But is it overkill for a relatively small site?

dianovich
  • 2,288
  • 21
  • 34

1 Answers1

1

I would recommend putting the twitter feed and flickr feed in JavaScript. Both flickr and twitter have REST APIs. By putting it on the client you free up resources on your server, create less complexity, your users won't be waiting around for your server to fetch the data, and you can let twitter and flickr cache the data for you.

This assumes you know JavaScript. Once you get past JavaScript quirks, it's not a bad language. Give Jquery a try. JQuery Twitter plugin Flickery JQuery plugin. There are others, that's just the first results from Google.

As for your data on the local server and remote server, that will depend more on the data that is being fetched. I would go with whatever you can develop the fastest and gives acceptable results. If that means making a REST call from server to sever, then go for it. IF the remote server is slow to respond, I would go the AJAX REST API method.

And for the local database, you are going to have to write server side code for that, so I would do that inside the Wordpress "framework".

Hope that helps.

FoneyOp
  • 336
  • 1
  • 6
  • I haven't ruled out REST requests on the client side, but there are a few problems with that including a) you have to parse data on the client side b) it's harder to merge the data obtained to filter on a particular field, e.g. date created – dianovich Apr 14 '11 at 20:20
  • I agree that server side code is usually faster to write and debug for most developers all things being equal. However client side AJAX in cases like yours are usually noticeably faster since the AJAX requests will be made in parallel not serial and the page can render before the remote calls are made. If that doesn't work for you, sounds like writing a wrapper for their REST api's in PHP and then caching in Memcache or APC might be a good alternative. – FoneyOp Apr 14 '11 at 20:25
  • As far as retrieving data goes, this is fine, it's just a case of manipulating it. If I have say 8 content types, e.g. blog posts, news items, flickr images, tweets, etc. it's going to be easier to, say, display all content ordered by datetime, showing exactly 4 of each type, for a particular author, if I store the data in a database. But I am wondering if I'm being too pedantic about what is essentially a small site that aggregates content on various pages. – dianovich Apr 14 '11 at 23:05
  • Is it possible to be TOO pedantic? – FoneyOp Apr 14 '11 at 23:21