Investigate Firebase Cloud Functions For Creating APIs and Scraping

[RyanMarten]

https://firebase.google.com/products/functions?authuser=0

[RyanMarten]

https://firebase.google.com/docs/functions/?authuser=0#how_does_it_work

This looks perfect for what we are trying to do

[RyanMarten]

Here are some example functions provided by the firebase team:
https://github.com/firebase/functions-samples

This looks like one of the most relevant use cases:
https://firebase.google.com/docs/functions/use-cases?authuser=0#execute_intensive_tasks_in_the_cloud_instead_of_in_your_app

Execute intensive tasks in the cloud instead of your app

This would be our scraping of data

[RyanMarten]

This web crawler and search service from Algolia looks interesting and promising:
https://www.algolia.com/products/crawler/

Firebase has a tutorial on how to use Algolia on their Realtime Database service
https://firebase.google.com/docs/functions/use-cases?authuser=0#execute_intensive_tasks_in_the_cloud_instead_of_in_your_app

[RyanMarten]

https://medium.com/@vinaymavi/serverless-web-scrapping-with-google-cloud-function-24d5f1e24442

@jollyredflames

[RyanMarten]

https://console.firebase.google.com/u/0/project/dreamtune-cdf8a/functions =---> Console

[RyanMarten]

For external calls you need to use the Blaze plan:
https://firebase.google.com/pricing/
https://stackoverflow.com/questions/43415759/use-firebase-cloud-function-to-send-post-request-to-non-google-server

The first 2,000,000 invocations, 400,000 GB-sec, 200,000 CPU-sec, and 5 GB of Internet egress traffic is provided for free each month.

[RyanMarten]

I'll upgrade the account now, I will just have to put limits on it so it doesn't charge me :(

[RyanMarten]

Creating budgets: https://firebase.googleblog.com/2018/03/adding-free-usage-to-blaze-pricing-plan.html

[RyanMarten]

No spending :)

[RyanMarten]

Can use a local emulator for Firebase Functions so you don't use up the cloud calls:
firebase emulators:start

See https://firebase.google.com/docs/functions/local-emulator for more!

[RyanMarten]

@ribhav99 the demo is not working entirely for me. You might want to try a different way of getting the text out of the HTML. Jquery might be helpful, im sure that there are web scraping libraries that also do this work.