How Often Can You Aggregate A Feed...?

[rockhard]

I'm building a aggregator/mashup site and I wonder how often you can access the various feeds/sites where the data will be aggregated from, without pissing off the owners of said sites? I would prefer to do this every time someone visits the site but that might not be a good idea if theres a lot of traffic.

 

[LotsOfZeros]

can you run the query once a day and cache the results?

 

[rockhard]

Yes, that's the plan. One more thing is that the site will be around 30 pages total and these will pull data from the same feeds but with different keywords, I'm wondering if I need to spread these out over the day. If the feeds would block me do they do it by host name or ip?

 

[patrick24601]

There is no fixed rule. Every feed owner would have a different threshold.

In my opinion I'd get every bit of the feed once a day and parse through it multiple times on your own server. That way you are beating on your server and not theres.

 

[iisbum]

Depending on how you're aggregating the feeds, I use SimplePie, there might be a caching function in the library that you can configure (default is 20 minutes I think), so even if you use the feed in multiple places the it can use the local cached version.

Typically I take the feeds (about every hour) and put them in MySQL so that I can do what I need with them.

Mubs

 

[rockhard]

Depending on how you're aggregating the feeds, I use SimplePie, there might be a caching function in the library that you can configure (default is 20 minutes I think), so even if you use the feed in multiple places the it can use the local cached version.

Typically I take the feeds (about every hour) and put them in MySQL so that I can do what I need with them.

Mubs

Thanks, I'll look into simple pie. I'm currently using regex.