Purpose of this project is to help new parents find baby formula during the 2022 Baby Formula Shortage. This application scrapes several online retailers for their baby formula inventory, and sends a report to a list of email subscribers. This project has benefited me personally and a few other moms in our circle in finding formula. The focus is primarily on Store Brand Formula.
Formula Finder can run on any OS. It just requires Python and Google Chrome to get it up and running. Formula Finder scrapes the web using Selenium, for product inventory information. Afterwards it generates an HTML report using PrettyTables. Then Formula Finder logs into an AOL email account, and sends the report out via BCC to a list of email subscribers or recipients.
- Clone the repo
git clone https://github.com/ChocolateTaco/formula-finder.git
- Install the requirements.txt
pip install -r requirements.txt
- Replace the applicable chrome driver in the /input_files/ directory, relevant to your version of Google Chrome
- Sign up for an AOL Email Address
- In AOL Account Security settings, "Generate app password" or One Time Password (OTP)
- In /input_files/ directory, create "emailAlert.txt"
- In "emailAlert.txt" add the AOL email address and the one time password (separated by a line break). NOT the password of the account, but the One Time Password
[email protected] emailsOneTimePassword
- In /input_files/emailSubscribers.txt, update the list of email recipients
- In the installation directory, run the run.py file
python3 run.py
- Scrape StoreBrand Formula Online Retailers
- Costco.com
- Optimize search speed
- Target.com
- Amazon.com
- 🛑 Kroger.com (Anti-Robot Friendly)
- 🛑 SamsClub.com (Anti-Robot Friendly)
- 🛑 Walmart.com (Anti-Robot Friendly)
- Costco.com
- Generate Report
- Gather Product Name, Availability, Price, URL
- Gather Last In Stock Date
- Shorten URL
- Email Results to list of email subscribers
- BCC emails to an email list using AOL SMTP
- Unsubscribe Feature
- Create Virtual Environment
- Automatic Scheduling
I wanted to learn a new stack such as Selenium. I will also be creating other web applications in the future. I liked that Selenium has options to perform web based GUI automation and testing.
One of the goals was to have it be cross compatible with operating systems such as Windows, Mac, and Linux. Firefox also requires additional driver (Gecko Driver) resource that would take an extra step.
Went with AOL for the nostalgia and wide range of available email addresses. Also as of May 30, 2022, Gmail removed the "less secure" mail settings now making python's SMTPLIB incompatible. Though I believe you can still use Gmail if you sign up for an App password.
Maybe. For the time being those domains have strong capchas and bot protection, making it more challenging for projects like Formula Finder to scrape.
- JimSC for the amazing Selenium lessons!