Using Selenium to automatically scrap the data from the website.
- Selenium
- RabbitMQ
There's something wrong with the website that prevent selenium to load all the collection. Right now is somewhat falling in the middle of the scrapying progress
As right now scrapying from the website doesn't really work smoothly, trying another way to fetch the data via the timehut mobile app
- mitmproxy
- RabbitMQ
- Use mitmproxy to collect the collection id/information from the request
- Fire one moment request to fetch the header used in the moment request
- Apply those collection_id and the header captured to fire all the request to fetch the moment information
- sudo rabbitmq-server
- Run timehutQueueConsumer.py
- Run mitmproxy -p 8080 -s [script_name]
- or mitmweb -s [script_name]
- Need to reinstall the app and relogin
- Tab one of the moment to fetch all moments