SP1D3R is a powerful and versatile web crawler application built using the Django, Scrapy, and Flutter frameworks. It provides a seamless and efficient way to crawl and scrape data from websites, offering a user-friendly interface for configuration and management.
- Clone the repository:
git clone https://github.com/hrideshmg/SP1D3R.git - Install dependencies:
pip install -r requirements.txt(for Django backend) and flutter pub get (for Flutter frontend). - Configure your database settings in Django's settings.py and utils.py
- Launch the Django development server:
python manage.py runserver - Navigate to the Flutter app directory: cd sp1d3r_flutter and run the app:
flutter run
Built as a part of an internal AmFOSS hackathon by @cr3ativ3cod3r, @hkx05, @shraav and me