0

I'm using feedparser to print the top 5 Google news titles. I get all the information from the URL the same way as always.

x = 'https://news.google.com/news/feeds?pz=1&cf=all&ned=us&hl=en&topic=t&output=rss'
feed = fp.parse(x)

My problem is that I'm running this script when I start a shell, so that ~2 second lag gets quite annoying. Is this time delay primarily from communications through the network, or is it from parsing the file?

If it's from parsing the file, is there a way to only take what I need (since that is very minimal in this case)?

If it's from the former possibility, is there any way to speed this process up?

Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
jay
  • 477
  • 1
  • 9
  • 17
  • Do you use some framework or it is pure python shell script? – Abbasov Alexander Jun 20 '13 at 20:24
  • 1
    run `python -mcProfile your_script.py` to see where the time is spent. – jfs Jun 20 '13 at 23:05
  • Thanks for the tip Sebastian. As a follow up, it looks like Johannes was right about it being a multitude of factors. The time is pretty evenly distributed between feedparser, opening urllib, and accessing information with http. – jay Jun 22 '13 at 00:07

2 Answers2

2

I suppose that a few delays are adding up:

  • The Python interpreter needs a while to start and import the module
  • Network communication takes a bit
  • Parsing probably consumes only little time but it does

I think there is no straightforward way of speeding things up, especially not the first point. My suggestion is that you have your feeds downloaded on a regularly basis (you could set up a cron job or write a Python daemon) and stored somewhere on your disk (i.e. a plain text file) so you just need to display them at your terminal's startup (echo would probably be the easiest and fastest).

I personally made good experiences with feedparser. I use it to download ~100 feeds every half hour with a Python daemon.

Johannes P
  • 888
  • 8
  • 16
  • For the last point, I personally use speedparser, which goes about 5 times faster for parsing feeds from string. – Andre Miras Aug 15 '14 at 09:55
0

Parse at real time not better case if you want faster result.

You can try does it asynchronously by Celery or by similar other solutions. I like the Celery, it gives many abilities. There are abilities as task as the cron or async and more.

Abbasov Alexander
  • 1,848
  • 1
  • 18
  • 27