feat: add memory, structured agents, swarm stream and retry bug fixes

[mozharovsky]

What

1. Core Architecture Improvements:

   • Introduced MessageStore for robust conversation history management
   • Added MessageIndex for semantic search capabilities over messages
   • Implemented ContextManager for intelligent context optimization
   • Simplified agent execution loop in Swarm for better reliability

2. Breaking Changes:

   • Removed implicit last agent preservation - users must now explicitly specify the agent for execution
   • Updated REPL commands and behavior to match new architecture
   • Changed stream handler interface to provide more context in responses

3. Error Handling & Reliability:

   • Implemented proper retry backoff for async generators
   • Added new error types for better error handling:
     • MaxAgentSwitchesError
     • MaxResponseContinuationsError
     • RetryError
   • Enhanced error messages and logging

4. Performance & Memory:

   • Implemented efficient message trimming algorithm
   • Added context optimization strategies (window, summarize, RAG)
   • Improved token usage tracking and management

5. Developer Experience:

   • Refactored REPL with new commands and better state management
   • Added comprehensive docstrings and examples
   • Improved type hints and validation

Why

• Improves conversation context management with multiple optimization strategies
• Adds semantic search capabilities for better context retrieval
• Reduces token usage through intelligent context pruning
• Makes the codebase more modular and maintainable
• Fixes memory leaks in long-running conversations

Scope Check

• Changes are focused (no scope creep)
• Less than 500 lines changed
• Core functionality only (no nice-to-have features)
• Tests added/updated
• Documentation added/updated

Future Improvements

• Add persistent storage backends for MessageStore
• Implement more advanced context optimization strategies
• Add support for custom embedding models
• Optimize batch processing for large message sets
• Add telemetry for context optimization effectiveness

Self Review

• PR is small and focused
• Commits are clear and logical
• No debug code left
• Tested locally