This is a critical bug fix update for the Connector.
Upgrade steps
To apply the fixes, and take advantage of the new feature, it will be necessary to create a new AMI or Vagrant VM using packer or vagrant, following the Quick Deployment steps in README.md.
If you are currently running an instance of the Connector, it is strongly recommended that you provision a new AMI, and launch a new instance in parallel. Once the new Connector instance has been configured, and is running correctly, the gnip-reader service on the old Connector should be stopped: sudo supervisorctl stop gnip-reader. This will allow datasift-writer to fully consume and send the data stored in the Kafka queue. The read and send metrics within the datasift-writer row in Grafana should indicate the point at which no more items are available.
Unfortunately a manual upgrade with an existing provisioned Connector is not advised, due a change of Java runtime and significant changes to the Chef provisioning recipes to address the below issues.
Bug fixes
- Fixes issue where datasift-writer does not commit Kafka offset correctly. Issue: #46
- Fixes issue where datasift-writer would throw KeyException errors on handshake with https://in.datasift.com. Issue #45
- Fixes issue where datasift-writer would not attempt reconnection to Kafka when VM was rebooted, and required a restart to operate correctly. Issue #32
Changes
- datasift-writer now applies back-off policy automatically when DataSift ingestion endpoint responds with HTTP code 429, in addition to 413.
- Default EC2 instance type for building an AMI with Packer has been changed from t2.micro to t2.small. This was to accommodate increased memory usage during provisioning. The Connector launched from the built AMI can still be run under a t2.micro instance. Please refer to the Pricing section in README.md for details.