Get_tweets gives the same page

[AnnOlChik]

So, I checked get_tweets function with default pages arg 25 and different hashtags. And as output, I got just 25 same pages of tweets. I did this:

from twitter_scraper import get_tweets
for tweet in get_tweets('#brexit'):
    print(tweet['text'], tweet['time'])

When I define the number of pages, it doesn't really change anything.

[bisguzar]

Yes, look like scrolling mechanism not working on hashtag pages.

>>> liste = [] 
... from twitter_scraper import get_tweets 
... for tweet in get_tweets('#mock', 3): 
...     liste.append(tweet['tweetId'])                                                                                                                                                                               

>>> liste.count(liste[0])                                                                                                                                                                                            
3 # first tweet's id repeats 3 times in our scraped tweets


>>> liste = [] 
... from twitter_scraper import get_tweets 
... for tweet in get_tweets('bugraisguzar', 3): 
...     liste.append(tweet['tweetId'])                                                                                                                                                                               

>>> liste.count(liste[0])                                                                                                                                                                                            
1 # first tweet's id is uniq as it should be

We will work on it, help needed

[seaona]

Got the same error. Does it have to do with the way it's queried and the resulting URL? maybe the pagination works different from User profiles to Hashtags?
Any thoughts? would be awesome to make it work!
Ps. by the way, great repo!

[bisguzar]

First at all, thanks. Looks like "max_position" parameter not supported for hashtag pages. I couldn't search it yet. Helps are welcome!

[anushkmittal]

did anyone have any progress? stuck at the same issue :/

[xeliot]

Finally got it to work! See changes in #150

Brief summary of how I fixed it by modifying tweets.py:

Change r = session.get(url, headers=headers) to r = session.get(url+'&max_position', headers=headers). This allows response json to return min_position parameter which will then be used as the max_position parameter in the next session.get

r_json = r.json()

Change r = session.get(url, params={'max_position': last_tweet}, headers=headers) to r = session.get(url, params={'max_position': r_json['min_position']}, headers=headers).

This gets rid of twitter pages repeating on search query.

[BradKML]

Would this pagination method in search applicable to digging all tweets from a user?