1

i need to create a pipe that will pull feeds from multiple sites, say 20. In the output I'll like to have a result with one entry(the most recent) for each site, none duplicated or repeated, and also sorted by date in descending order. Say, in the end, I'll have 20 items in my result to work with, each coming from each of the sites. I hope my question is clear enough. What would be the best way to do this please?

degee
  • 173
  • 2
  • 11

2 Answers2

0

You can do this by creating 2 pipes:

  1. A pipe that fetches a feed (one of the 20), and truncates after 1 item
  2. Another that loops over feed URLs, and for each URL calls the first pipe

Here's an example of the first pipe:

http://pipes.yahoo.com/pipes/pipe.info?_id=106eaba3d5fe1f43805fd4094b11186d

Here's an example of the second:

http://pipes.yahoo.com/pipes/pipe.info?_id=290ef8130a4ff423a8d977493bbc669d

Based on the above two, you should be able to create exactly what you need.

For a similar example, see also this other answer:

Yahoo Pipes how to truncate items per feed with multiple feeds

Community
  • 1
  • 1
janos
  • 120,954
  • 29
  • 226
  • 236
  • thanks for your reply. But I don't think your solution would work for me as I don't intend to put in the urls at runtime. They all have to be in the pipe source and just give me the result of 20 items, each from a specific url, when I run the pipe – degee Jun 04 '14 at 08:54
  • @degee that's very easy to change. I updated the pipe to remove the URL parameter and hardcode the URL in the pipe. I have only 2 feeds in my pipe, you can have 20 by chaining together multiple Union operators. See the updated example pipe. – janos Jun 04 '14 at 12:14
0

I know this is a really old question. Maybe you've found a solution already, but I figured I'd give this a go.

If I understand the question correctly, you are pulling RSS feeds from say 20 links and want the most recent post from each.

You can do that with the YQL module. Like this:

select * from yql.query.multi where queries="SELECT * from feed where url = 'http://www.discofilter.com/feeds/posts/default'|sort (field='published', descending = 'true') | truncate (count=1); SELECT * from feed where url = 'http://dotsanddashes.co.uk/feed/'|sort (field='pubDate', descending = 'true') | truncate (count=1)"

There's some documentation about this.

I read someone else here mention that there's a limit to how many queries you can do at once, but someone else replied and said that's not true. I don't know myself, but you can find out by giving it a try.

The code uses a built in Yahoo Data Table (yql.query.multi) to then pull multiple feeds (within " ; "). And then you use -- 

select * from feed where url = ' '

From there, you use the sorting and filtering: 

|sort (field = 'published', descending = 'true')|truncate (count=1)

It's not exactly elegant, but I think it does what you asked for and in one step.

Stack Johan
  • 379
  • 1
  • 6
  • 23