Building retrievers with multiple docs (web pages or files)

[zhiheng-huang]

Description

Support scraping pages under a given domain. As a result. we now support building retrievers with multiple webpages or documents.

make formatting
All unit tests pass
make test
Verified the following toy experiments run okay.
poetry run python experiments/index_and_query_from_docs.py
poetry run python experiments/index_and_query_from_webpage.py

Related Issue

No

Type of Change

• 📚 Examples / docs / tutorials / dependencies update
• 🔧 Bug fix (non-breaking change which fixes an issue)
• 🥂 Improvement (non-breaking change which improves an existing feature)
• 🚀 New feature (non-breaking change which adds functionality)
• 💥 Breaking change (fix or feature that would cause existing functionality to change)
• 🔐 Security fix

Checklist

• I've read the CODE_OF_CONDUCT.md document.
• I've read the CONTRIBUTING.md guide.
• I've updated the code style using make codestyle.
• I've written tests for all new methods and classes that I created.
• I've written the docstring in Google format for all the methods and classes that I used.

[jotyy]

This test output file should not be submitted.