Project Proposal Feedback

[pooyanjamshidi]

Feedback on RoostAI Proposal:

First off, I am sorry for the delay in providing my feedback on your proposal! second, great job on outlining the problem and your approach! You’ve clearly defined the need for a centralized platform for USC and the benefits it could bring. The use of a Retrieval-Augmented Generation (RAG) framework is smart, and I can see that you’ve really thought through the technical aspects. Let me walk you through some thoughts and suggestions that can help tighten things up a bit:

Strengths:

• Problem Clarity: You’ve nailed the problem statement. The lack of a unified platform at USC is something I’ve seen too, and your idea to develop a chatbot that pulls info from multiple sources will definitely help improve user experience on campus.
• Technical Setup: I like how you’re approaching the project with the RAG framework. Scraping and indexing data, then retrieving it with semantic search, is a solid plan. Your pipeline from web scraping to deploying the chatbot seems well thought out.
• Evaluation Plan: You’re covering your bases with quantitative metrics like latency and qualitative ones like user studies. This is exactly what you need to do for thorough validation.
• Scalability: The fact that you’re considering cloud deployment and load balancing shows you’re thinking long-term, which is great for this type of project.

Suggestions:

1. Data Privacy: You’ll want to think about privacy when scraping and using USC-specific data. While most of what you’re scraping is public, you need to make sure you’re not pulling in any sensitive info accidentally. It might also be worth mentioning any steps you're taking to comply with data protection regulations like GDPR, especially since this will involve user interactions.

2. Challenges of Web Scraping: Scraping can get messy sometimes. There might be access restrictions (e.g., password-protected content, rate limits), or links can break. How will your system handle these cases? If possible, look into whether you can get API access for a more structured data feed instead of relying entirely on scraping.

3. Limitations of LLMs: Be careful with user expectations. LLMs aren’t perfect, and they’ll definitely return irrelevant or incorrect info at times. You might want to talk about how you’ll handle situations where the chatbot can’t provide a good answer—will it direct users to a human or suggest other resources?

4. Handling Ambiguity in Queries: University-related questions can be vague or open-ended. It’d be good to plan for how the chatbot will deal with these. Will it ask follow-up questions to clarify the user’s intent?

5. Dataset Maintenance: You’ve talked about scraping and indexing USC data, but how often are you planning to update this knowledge base? You’ll need to make sure that the chatbot stays current. Maybe mention how frequently the updates will happen.

6. User Experience and Evaluation: You’ve got solid technical metrics, but don’t forget to evaluate the overall user experience. How will you measure whether people actually find the chatbot helpful? Maybe as part of your user study, you can ask users to rate the usefulness and ease of use of the bot.

7. Accessibility: It might be good to touch on making the chatbot accessible to all users, including those with disabilities. This could be a nice addition and make the bot more widely usable (e.g., screen reader compatibility, voice input).

Other Ideas:

• Collaboration with USC IT: If you haven’t already, it might be worth getting in touch with the USC IT team. They could help you integrate the chatbot into official university platforms and apps down the road.

• Learning from User Feedback: Maybe the bot can have a feature where users can provide feedback or correct its responses, allowing the chatbot to learn and improve over time.

---

Overall, this is a strong proposal! With a bit of refinement around privacy, dataset updates, and handling LLM limitations, you’ll be in a great spot. Looking forward to seeing how RoostAI develops!