async-go

Producer-Consumer Pattern Implementation in Go

Producers       Channel       Consumers
[P1] ----\                  /----> [C1]
[P2] -----\   ┌─────────┐  /-----> [C2]
[P3] ------\  │ Message │ /------> [C3]
[P4] -------► │ Queue   │ -------> [C4]
[P5] ------/  │ Buffer  │ \------> [C5]
[P6] -----/   └─────────┘  \-----> [C6]
[P7] ----/                  \----> [C7]

Metrics & Monitoring
┌────────────────────┐
│ ● Active Count     │
│ ● Message Latency  │
│ ● Error Rate       │
└────────────────────┘

Components

1. Producers (P1-P7)

   • Independent message producers
   • Asynchronous message generation
   • Configurable production rates
   • Built-in backpressure handling

2. Message Queue Buffer

   • Thread-safe implementation
   • Configurable buffer size
   • FIFO (First-In-First-Out) processing
   • Memory-efficient design

3. Consumers (C1-C7)

   • Parallel message processing
   • Independent consumption rates
   • Error handling and retry mechanisms
   • Scalable consumer groups

Features

• High Performance

  • Non-blocking operations
  • Optimized memory usage
  • Efficient message routing

• Reliability

  • Message persistence
  • Transaction support
  • Guaranteed message delivery
  • Fault tolerance

• Scalability

  • Horizontal scaling
  • Dynamic producer/consumer addition
  • Load balancing
  • Cluster support

• Monitoring

  • Real-time metrics
  • Performance analytics
  • Health checks
  • Alert system

Getting Started

Prerequisites

go version >= 1.20

Installation

go install github.com/shivanshuraj1333/producer-consumer-pattern

Configuration

yaml
buffer:
size: 1000
workers: 5
batchSize: 100
producers:
maxRetries: 3
timeout: 5s
consumers:
concurrency: 10
processTimeout: 30s

Monitoring & Metrics

The system provides various metrics for monitoring:

• Message throughput
• Queue depth
• Processing latency
• Error rates
• Resource utilization

Performance

Scenario	Messages/sec	Latency (ms)
Light	10,000	< 1
Medium	50,000	< 5
Heavy	100,000	< 10

Error Handling

• Automatic retry mechanism
• Dead letter queues
• Error logging and tracking
• Circuit breaker pattern

Best Practices

1. Configure buffer size based on memory constraints
2. Implement proper error handling
3. Monitor queue depth regularly
4. Use batch processing for better performance
5. Implement backpressure mechanisms

Contributing

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add some amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details.

Support

For support, please open an issue in the GitHub repository or contact the maintainers.

Acknowledgments

• Thanks to all contributors
• Inspired by various message queue systems
• Built with Go's concurrent patterns

Roadmap

• Distributed clustering support
• Enhanced monitoring dashboard
• Additional persistence options
• WebSocket interface
• REST API endpoints