This repository contains implementations of the MapReduce distributed computing framework in multiple programming languages. Each implementation is based on the principles outlined in Google's MapReduce paper.
- Distributed Processing: Execute computations across multiple machines
- Fault Tolerance: Handles node failures and network interruptions
- Scalability: Easily scale with additional worker nodes
- Custom Functions: Define custom map and reduce operations
- Built-in Examples: Includes word count and other common MapReduce tasks
A Go-based MapReduce framework featuring:
- Clean and straightforward implementation
- Plugin system for custom MapReduce tasks
- Efficient worker coordination
- Example WordCount implementation
A high-performance Rust implementation offering:
- High concurrency with Rust's safety guarantees
- Async runtime for efficient I/O
- Strong type system and memory safety
- Comprehensive test coverage
The system follows a coordinator-worker architecture:
- Manages task distribution
- Handles worker coordination
- Tracks job progress
- Manages fault tolerance
- Execute map and reduce tasks
- Handle data processing
- Communicate with coordinator
- Manage intermediate results
.
├── golang/ # Go implementation
│ ├── src/ # Source code
│ ├── apps/ # MapReduce applications
│ └── files/ # Sample input files
│
└── rust/ # Rust implementation
├── src/ # Source code
├── examples/ # Example implementations
└── tests/ # Test suites
Contributions are welcome! Feel free to:
- Add new implementations in other languages
- Improve existing implementations
- Add new MapReduce applications
- Fix bugs or improve documentation
Please see the individual implementation directories for specific contribution guidelines.
This project is licensed under the MIT License - see the individual implementation directories for specific license details.