Using the Faker with Vagrant

The Faker is a simple tool written to allow you to verify data can successfully flow through your instance of the connector. This guide assumes you have set up your instance of the connector via Vagrant (vagrant up), and SSH'd into your connector instance (vagrant ssh).

1. Start by updating your reader and writer config files to tell the connector where data will be flowing from and to:

1.a. Update the gnip.host value to http://localhost:5001

sudo vi /etc/datasift/gnip-reader/reader.json

{
  "kafka": {
      "topic": "twitter",
      "servers": "localhost:6667",
      "retry-backoff": 1000,
      "reconnect-backoff": 1000
  },
  "gnip": {
      "account": "ACCOUNT",
      "label": "LABEL",
      "product": "PRODUCT",
      "username": "USER",
      "password": "PASSWORD",
      "host": "http://localhost:5001"
  },
  "hosebird" : {
      "retries": 10,
      "buffer_size": 10000,
      "buffer_timeout": 500
  },
  "metrics": {
      "host": "localhost",
      "port": 8125,
      "prefix": "hosebird.reader",
      "reporting-time": 1
  }
}

1.b. Update the datasift.base_url and datasift.port values to http://localhost:5002

sudo vi /etc/datasift/datasift-writer/writer.json

{
  "zookeeper": {
    "socket": "localhost:2181"
  },
  "kafka": {
    "topic": "twitter",
    "broker": "localhost",
    "port": 6667
  },
  "datasift": {
    "base_url": "http://localhost",
    "port": 5002,
    "username": "YOUR_DATASIFT_USERNAME",
    "api_key": "YOUR_DATASIFT_API_KEY",
    "source_id": "YOUR_DATASIFT_SOURCE_ID",
    "bulk_size": 100000,
    "bulk_items": 1000,
    "bulk_interval": 1000
  },
  "metrics": {
    "host": "localhost",
    "port": 8125,
    "prefix": "datasift.writer",
    "reporting-time": 1
  }
}

2. After saving these changes, restart both the reader and writer services:

sudo supervisorctl restart gnip-reader datasift-writer

3. Now the connector knows where your Faker data will flow from and to, we can start the Faker services:

3.a. Start the gnip-faker with default settings

cd /vagrant/gnip-faker/
./server.js

3.b. Start the datasift-ingestion-faker with default settings

node /vagrant/datasift-ingestion-faker/server.js

4. Take a look at your Grafana Dashboard at localhost:3000; you should now see data flowing through your connector