Set up complete MQTT broker infrastructure with RabbitMQ and Redis for transit vehicle tracking

.env.example

@@ -0,0 +1,9 @@
+# RabbitMQ Configuration
+RABBITMQ_USER=admin
+RABBITMQ_PASS=admin
+
+# MQTT Configuration
+MQTT_TOPIC=transit/vehicles/#
+
+# Redis Configuration
+REDIS_DB=0

.gitignore

@@ -0,0 +1,40 @@
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+.Python
+venv/
+env/
+ENV/
+*.egg-info/
+dist/
+build/
+
+# Environment variables
+.env
+
+# IDE
+.vscode/
+.idea/
+*.swp
+*.swo
+*~
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Docker
+*.log
+
+# Redis
+dump.rdb
+
+# Node.js (if needed later)
+node_modules/
+package-lock.json
+
+# Temporary files
+tmp/
+*.tmp

.python-version

@@ -0,0 +1 @@
+3.14

QUICKSTART.md

@@ -0,0 +1,132 @@
+# Quick Start Guide
+
+This guide will help you get the databus-mqtt system up and running in minutes.
+
+## Prerequisites
+
+- Docker (20.10 or later)
+- Docker Compose (or Docker with Compose plugin)
+
+## Installation Steps
+
+1. **Clone the repository**
+   ```bash
+   git clone https://github.com/simovilab/databus-mqtt.git
+   cd databus-mqtt
+   ```
+
+2. **Configure environment (optional)**
+   ```bash
+   cp .env.example .env
+   # Edit .env if you want to change default settings
+   ```
+
+3. **Start all services**
+   ```bash
+   docker-compose up -d
+   ```
+
+4. **Verify services are running**
+   ```bash
+   docker-compose ps
+   ```
+
+   You should see three containers running:
+   - `databus-rabbitmq` - MQTT broker
+   - `databus-redis` - In-memory database
+   - `databus-subscriber` - MQTT to Redis bridge
+
+## Testing the System
+
+### 1. Access RabbitMQ Management UI
+
+Open your browser and navigate to: http://localhost:15672
+
+- Username: `admin` (or value from .env)
+- Password: `admin` (or value from .env)
+
+### 2. Publish Test Messages
+
+Install the paho-mqtt library:
+```bash
+pip install paho-mqtt
+```
+
+Run the example publisher:
+```bash
+python examples/publisher_example.py
+```
+
+This will start publishing simulated vehicle tracking data.
+
+### 3. Query Stored Data
+
+Install the redis library:
+```bash
+pip install redis
+```
+
+Run the query example:
+```bash
+python examples/query_redis.py
+```
+
+This will display all vehicle data stored in Redis.
+
+### 4. Monitor Subscriber Logs
+
+Watch the subscriber service process messages:
+```bash
+docker-compose logs -f subscriber
+```
+
+## Common Commands
+
+### View all logs
+```bash
+docker-compose logs -f
+```
+
+### Stop all services
+```bash
+docker-compose down
+```
+
+### Restart a service
+```bash
+docker-compose restart subscriber
+```
+
+### View Redis data directly
+```bash
+docker exec -it databus-redis redis-cli
+# Then run Redis commands like:
+# KEYS *
+# GET vehicle:BUS-001
+```
+
+## Next Steps
+
+- Customize the MQTT topic in `.env` file
+- Modify `subscriber/subscriber.py` to add custom processing logic
+- Integrate with your transit vehicle tracking system
+- Add additional services to the Docker Compose file
+
+## Troubleshooting
+
+**Services won't start:**
+- Check if ports 1883, 5672, 6379, or 15672 are already in use
+- Ensure Docker has sufficient resources allocated
+
+**Can't connect to MQTT broker:**
+- Wait 30 seconds for RabbitMQ to fully initialize
+- Check logs: `docker-compose logs rabbitmq`
+
+**No data in Redis:**
+- Ensure the subscriber is running: `docker-compose ps subscriber`
+- Check subscriber logs: `docker-compose logs subscriber`
+- Verify you're publishing to the correct topic
+
+## Support
+
+For issues and questions, please open an issue on GitHub.

README.md

@@ -1,2 +1,209 @@
 # databus-mqtt
+
 MQTT broker for real-time transit vehicles tracking and telemetry data
+
+## Overview
+
+This repository provides a complete infrastructure for collecting real-time transit vehicle tracking and telemetry data using MQTT protocol. The system uses RabbitMQ as the MQTT broker and Redis as an in-memory database for quick data retrieval.
+
+## Architecture
+
+The system consists of three main components:
+
+1. **RabbitMQ MQTT Broker**: Receives MQTT messages from transit vehicles
+2. **Redis**: In-memory database for fast data storage and retrieval
+3. **Subscriber Service**: Python service that subscribes to MQTT topics and stores data in Redis
+
+```
+Transit Vehicles → MQTT → RabbitMQ → Subscriber Service → Redis
+                         (Port 1883)                    (Port 6379)
+```
+
+## Features
+
+- **MQTT Protocol Support**: Full MQTT broker capabilities via RabbitMQ
+- **Real-time Data Processing**: Immediate storage of vehicle tracking data
+- **Fast Data Retrieval**: Redis in-memory storage for sub-millisecond access
+- **Scalable Architecture**: Docker-based containerized services
+- **Management Interface**: RabbitMQ management UI on port 15672
+- **Automatic Reconnection**: Built-in retry logic for service reliability
+- **TTL Support**: Automatic data expiration (1 hour default)
+
+## Prerequisites
+
+- Docker
+- Docker Compose
+
+## Quick Start
+
+1. Clone the repository:
+```bash
+git clone https://github.com/simovilab/databus-mqtt.git
+cd databus-mqtt
+```
+
+2. Copy the environment file and adjust settings if needed:
+```bash
+cp .env.example .env
+```
+
+3. Start the services:
+```bash
+docker-compose up -d
+```
+
+4. Check service status:
+```bash
+docker-compose ps
+```
+
+## Configuration
+
+### Environment Variables
+
+Edit the `.env` file to customize the configuration:
+
+- `RABBITMQ_USER`: RabbitMQ username (default: admin)
+- `RABBITMQ_PASS`: RabbitMQ password (default: admin)
+- `MQTT_TOPIC`: MQTT topic pattern to subscribe to (default: transit/vehicles/#)
+- `REDIS_DB`: Redis database number (default: 0)
+
+### RabbitMQ Configuration
+
+RabbitMQ configuration is located in `config/rabbitmq/`:
+- `enabled_plugins`: Enables MQTT and management plugins
+- `rabbitmq.conf`: Main RabbitMQ configuration file
+
+## Usage
+
+### Accessing Services
+
+- **RabbitMQ Management UI**: http://localhost:15672 (login with credentials from .env)
+- **MQTT Port**: localhost:1883
+- **Redis Port**: localhost:6379
+
+### Publishing Test Messages
+
+You can publish test messages using any MQTT client. Example using mosquitto_pub:
+
+```bash
+mosquitto_pub -h localhost -p 1883 -u admin -P admin \
+  -t "transit/vehicles/bus/1234" \
+  -m '{"vehicle_id": "1234", "lat": 40.7128, "lon": -74.0060, "speed": 25, "heading": 180}'
+```
+
+### Querying Data from Redis
+
+Connect to Redis and query stored data:
+
+```bash
+docker exec -it databus-redis redis-cli
+
+# Get all vehicle keys
+KEYS vehicle:*
+
+# Get data for a specific vehicle
+GET vehicle:1234
+
+# Get recent vehicles by timeline
+ZRANGE vehicles:timeline -10 -1
+```
+
+## Data Storage Schema
+
+The subscriber service stores data in Redis with the following structure:
+
+- **Key Pattern**: `vehicle:{vehicle_id}`
+- **Value**: JSON string containing vehicle data with metadata
+- **TTL**: 1 hour (3600 seconds)
+- **Timeline Index**: Sorted set `vehicles:timeline` for time-based queries
+
+Example stored data:
+```json
+{
+  "vehicle_id": "1234",
+  "lat": 40.7128,
+  "lon": -74.0060,
+  "speed": 25,
+  "heading": 180,
+  "_timestamp": "2024-01-15T10:30:00.123456",
+  "_topic": "transit/vehicles/bus/1234"
+}
+```
+
+## Monitoring
+
+### View Subscriber Logs
+
+```bash
+docker-compose logs -f subscriber
+```
+
+### View RabbitMQ Logs
+
+```bash
+docker-compose logs -f rabbitmq
+```
+
+### Check Redis Memory Usage
+
+```bash
+docker exec -it databus-redis redis-cli INFO memory
+```
+
+## Development
+
+### Project Structure
+
+```
+databus-mqtt/
+├── config/
+│   └── rabbitmq/          # RabbitMQ configuration
+│       ├── enabled_plugins
+│       └── rabbitmq.conf
+├── subscriber/            # MQTT subscriber service
+│   ├── Dockerfile
+│   ├── requirements.txt
+│   └── subscriber.py
+├── docker-compose.yml     # Docker Compose configuration
+├── .env.example          # Environment variables template
+└── README.md             # This file
+```
+
+### Extending the Subscriber
+
+The subscriber service (`subscriber/subscriber.py`) can be extended to:
+- Add custom data validation
+- Implement additional storage backends
+- Add data transformation logic
+- Integrate with other services
+
+## Stopping the Services
+
+```bash
+docker-compose down
+```
+
+To remove volumes as well:
+```bash
+docker-compose down -v
+```
+
+## Troubleshooting
+
+### RabbitMQ won't start
+- Check if ports 1883, 5672, or 15672 are already in use
+- Verify Docker has sufficient resources
+
+### Subscriber can't connect
+- Ensure RabbitMQ is fully started (check `docker-compose logs rabbitmq`)
+- Verify credentials in `.env` file
+- Check network connectivity between containers
+
+### Redis connection issues
+- Verify Redis container is running: `docker-compose ps redis`
+- Check Redis logs: `docker-compose logs redis`
+
+## License
+
+Apache License 2.0 - See LICENSE file for details

config/rabbitmq/enabled_plugins

@@ -0,0 +1,2 @@
+[rabbitmq_management, rabbitmq_mqtt].
+