I'll share are a few other wrinkles I've observed in the paged
friends / followers API responses:
- There are often more (sometime many more) pages than you'd expect
based on the follower/friend counts listed in the profile
- Pages in the middle of a long series of output pages usually but not
always return 5000 elements
- The only reliable way to determine the end of available output pages
is to keep asking until an empty JSON response "[]" is returned
- Friends / follow pages occasionally contain some duplicate entries
from one page to another
On the positive side, the paged output methods are *far* more reliable
for high friend/follow lists than the old ones, when the list is long.
The old methods were more convenient but only worked a small fraction
of the time when the list was very long.
I have test data for britneyspears from a few days ago, at that time
the profile said 1,644,227, which would imply around 328 pages. There
were actually 356 pages, containing 1,772,771 entries. I just looked
at the profile page now and it says 1,745,417, which would still
suggest 349 pages of data, less than what is actually there. And
several of the pages return less than 5000 entries, so you can't
assume that an unfilled page represents the end of the data.
On Jun 8, 6:19 pm, Jesse Stay <
jesses...@gmail.com> wrote:
> Doug, et. al., here's the problem(s) I'm running into. By forcing me to use
> paging for followers/ids and friends/ids, for someone like BritneySpears I
> now have to make over 350 requests to get through all her followers. Now
> I'm having huge rate limit issues because of that, not to mention how long
> it takes to get through the entire list. Would it be possible to set a
> number to specify how many user ids are returned per page so I don't have to
> make so many requests?
> In addition, I'm finding the pages aren't returning consistent data. Some
> are returning less than 5,000 results, and some aren't even returning data
> that should. So even with Paging I'm still unable to get through all of
> @britneyspears' followers. Any suggestions?
>
> @Jesse
>
> On Thu, Jun 4, 2009 at 12:26 AM, Doug Williams <
d...@twitter.com> wrote:
> > I've heard that list sizes greater than 150K-200K start to return timeouts
> > at higher rates. Although I'd enjoy hearing first-hand experiences and
> > recommendations.
> > Thanks,
> > Doug
>
> > On Wed, Jun 3, 2009 at 9:19 PM, Jesse Stay <
jesses...@gmail.com> wrote:
>
> >> In my case specifically it's the Social Graph methods. I didn't realize
> >> you had paging available now. Is there some logic as to when I should
> >> expect to page and when I can just rely on the full result?
> >> Jesse
>
> >> On Wed, Jun 3, 2009 at 9:56 PM, Doug Williams <
d...@twitter.com> wrote:
>
> >>> What methods in particular are you referring to? The social graph methods
> >>> now support paging so retrieving all of that data is now possible, where it
> >>> used to throw 502s. It does however require a bit of application logic to
> >>> assume when paging is necessary (e.g. large follower counts). Additionally,
> >>> we are making changes to the databases which cause latency that result in
> >>> periodic 502s. We are not able to give definitive ETAs on these fixes due to
> >>> priorities that change as unforeseeable critical needs arise.
> >>> More specificity would be beneficial. Do you have a replaceable bug,
> >>> problem, or suggestion that you would like to discuss?
> >>> Thanks,
> >>> Doug
>