Code

Version3 package refactor#54

Code

Open

jasonyunicorn wants to merge 110 commits intomasterrewardStyle/kinetic:masterfrom version3_package_refactorrewardStyle/kinetic:version3_package_refactorCopy head branch name to clipboard

+6,658-2,029

Commits

Commits on Mar 27, 2017

• new listener

  twang-rs

  committed
  f6f2f66

Commits on Mar 29, 2017

• refactored listener, added kinetic

  Show description for dc17c6f
  twang-rs

  committed
  dc17c6f

• go vet & golint fixes

  twang-rs

  committed
  f94ded4

• gosimple fixes

  twang-rs

  committed
  41e8fb3

• more go vet & go race fixes

  twang-rs

  committed
  9481b6a

Commits on Mar 31, 2017

• Simplify configuration implementation

  Show description for a1d14e4
  twang-rs

  committed
  a1d14e4

• Disable ListenFn test

  Show description for 60f11c0
  twang-rs

  committed
  60f11c0

• Fix race condition in ListenFn test

  Show description for c78e129
  twang-rs

  committed
  c78e129

Commits on Apr 5, 2017

• producer -- work in progress

  twang-rs

  committed
  2fcc21d

Commits on Apr 6, 2017

• lint & build fixes

  twang-rs

  committed
  0d27501

Commits on Apr 7, 2017

• producer refactor, wip

  twang-rs

  committed
  29483f6

Commits on May 5, 2017

• producer (kinesis + firehose)

  twang-rs

  committed
  69c785e

Commits on May 9, 2017

• wip

  twang-rs

  committed
  5805f55

Commits on May 10, 2017

• Stabilized StreamReader abstraction changes

  

  Jason Yu

  committed
  d4de4d0

Commits on May 11, 2017

• Fix errors and golint warnings

  

  Jason Yu

  committed
  1f865bf

• Fix more errors

  

  Jason Yu

  committed
  d9e62af

Commits on May 12, 2017

• Fix a Nil reference exception

  

  Jason Yu

  committed
  4b69801

Commits on May 13, 2017

• Added a default queueDepth which was causing the listener to hang

  

  Jason Yu

  committed
  1acb629

Commits on May 16, 2017

• Refactored StreamReader interface based on new design considerations. Introduced first draft of KclReader. Code cleanup based on code review.

  

  Jason Yu

  committed
  7f564b2

• Fixed golint errors. Implemented sendMessage function to write to stdout.

  

  Jason Yu

  committed
  6edab84

• Call processAction from ensureClient. Handle decoding and unmarshalling correctly.

  

  Jason Yu

  committed
  98dfd11

• Fix gosimple error by simplifying error handling logic

  

  Jason Yu

  committed
  d41dd36

Commits on May 18, 2017

• Fix unmarshalling / base64 decoding step of the KCL Multilang Daemon's processRecord message

  

  Jason Yu

  committed
  af64e1a

• Added a custom unmarshaller to handle html angle brackets

  

  Jason Yu

  committed
  c408a5f

• Fix gosimple error

  

  Jason Yu

  committed
  cc09647

Commits on May 19, 2017

• Merge pull request #45 from rewardStyle/version3_listener_unstable

  Show description for 6977a19
  

  Jason Yu

  authored
  6977a19

Commits on May 22, 2017

• Added a function to set the stream reader instead of setting the kinetic stream

  

  Jason Yu

  committed
  f5a6907

• Git branching clean up

  

  Jason Yu

  committed
  5cdc2a2

Commits on May 25, 2017

• Added integration testing to test sending multiple messages through a kinesis stream using the kinetic object with a producer and a listener

  

  Jason Yu

  committed
  77f05e9

• Update glide files to pull in assertion package. Fix golint issues.

  

  Jason Yu

  committed
  28d14bd

• Adjust the kinetic producer / listener configs to fix race condition causing sporadic unit test failures

  

  Jason Yu

  committed
  3282271

Commits on May 26, 2017

• Added sample application to create a stand-alone executable for running/testing the kinetic producer and listener. Also made enhancements to unit tests.

  

  Jason Yu

  committed
  636b47d

• Fix golint errors

  

  Jason Yu

  committed
  90b6288

• Added exception logging and more code clean up

  

  Jason Yu

  committed
  6a24e8e

Commits on Jun 1, 2017

• Refactored executable test program to add the ability to 1) stream to an AWS Kinesis stream and 2) run for either a fixed or indefinite duration of time. Fixed potential deadlock conditions and ear…

  Show description for 538f5b8
  

  Jason Yu

  committed
  538f5b8

Commits on Jun 2, 2017

• Modified the README.md file to update usage examples for the producer / listener

  

  Jason Yu

  committed
  9702c17

Commits on Jun 7, 2017

• Added back KCL reader implementation and fixed bugs encountered when interacting with the MultilangDaemon

  

  Jason Yu

  committed
  a957c86

Commits on Jun 13, 2017

• Producer refactor changes after team code review

  Show description for 0bd3a3c
  

  Jason Yu

  committed
  0bd3a3c

Commits on Jun 22, 2017

• Refactored kinetic to remove the association between the producer/writer and the listener/reader by requiring the readers (kinesisreader/kclreader) and writers (kinesiswriter/firehosewriter) to hav…

  Show description for baebb8a
  

  Jason Yu

  committed
  baebb8a

Commits on Jun 26, 2017

• Modified the signature of PutRecords to accept a context variable and a lambda function for processing failed messages. Refactored the firehosewriter and streamwriter to implement the new stream wr…

  Show description for 251fc65
  

  Jason Yu

  committed
  251fc65

• Refactor readers and writers to have an AwsConfig field instead of using AwsOptions as an embedded type in order to simplify code. Note: The producer and listener still have AwsOptions as an embedd…

  Show description for 04d6790
  

  Jason Yu

  committed
  04d6790

Commits on Jun 27, 2017

• Renamed MessageFn to MessageHandler and created a new type called MessageProcessor for handling the user-supplied functions for Listen, Retrieve, etc.. Changed PutRecords interface to process the m…

  Show description for 2381724
  

  Jason Yu

  committed
  2381724

Commits on Jun 28, 2017

• Upgraded to the latest version of the aws-sdk-go library and simplified the ReadCloserWrapper in getRecords(). Added per-request response timeouts to both GetRecordsRequest and PutRecordsRequest. P…

  Show description for 76666ac
  

  Jason Yu

  committed
  76666ac

Commits on Jun 29, 2017

• Fixed a bug with setting the batchSize to 500. Added validation logic to prevent the user from setting the batchSize to an invalid value. Add stats to kclreader.go

  

  Jason Yu

  committed
  4217118

Commits on Jul 5, 2017

• Simplified ReadCloserWrapper by removing the OnReadFn field which is no longer needed because of the aws-sdk-go version upgrade. Listenerconfig and producerconfig now have aws.Config as a field ins…

  Show description for 75d9289
  

  Jason Yu

  committed
  75d9289

Commits on Jul 6, 2017

• Make attempts counter in producer "thread safe". Changes to testexec for output logging, command line interface and documentation.

  

  Jason Yu

  committed
  8ad9b48

• Run gofmt.

  

  Jason Yu

  committed
  eda4de4

Commits on Jul 10, 2017

• Added a DefaultStatsCollector to both the listener package and the producer package which both implement the StatsCollector interface using the rcrowley/go-metrics library. Added unit tests and mod…

  Show description for 2a6685c
  

  Jason Yu

  committed
  2a6685c

• Fixed a bug with the producer's stats collection for the Sent and Failed counters by moving the stats collection into the writers.

  

  Jason Yu

  committed
  1d4a236

• Converted magic strings for stats into constants and added missing stats in PrintStats.

  

  Jason Yu

  committed
  99cb2e8

Commits on Jul 11, 2017

• Renamed the producer stats to make more sense.

  

  Jason Yu

  committed
  2e20f0c

Commits on Jul 12, 2017

• Added new producer stats to the StatsCollector interface to capture retried message statistics. Added a new command line input to the testexec program to allow testing for deadlocks with the producer.

  

  Jason Yu

  committed
  b12dc44

Commits on Jul 18, 2017

• Initial commit of producer refactor based on producer redsign to 1) remove the retry channel completely 2) and have go routines hold on to failed messages to retry them instead and 3) add a rate li…

  Show description for 8f919fc
  

  Jason Yu

  committed
  8f919fc

• Fixed a bug that introduced nil messages into the workers retry messages by converting the closure function that handles what to do with failed messages to a synchronous call that waits on a wait g…

  Show description for 1a8ec24
  

  Jason Yu

  committed
  1a8ec24

Commits on Jul 19, 2017

• Removed the worker object entirely replaced it with a helper function within the producer for running concurrent worker routines to call sendBatch. Added the worker's 'inbox' channel in the the sta…

  Show description for 0c6ec56
  

  Jason Yu

  committed
  0c6ec56

• Clean up the command line arguments for the test executable and updated README.md

  

  Jason Yu

  committed
  a4f7a28

• Fixed a bug in the produce function where we weren't waiting for the rateLimiter to reset by added a reset notification channel in the rateLimiter that notifies when the token count has been reset.

  

  Jason Yu

  committed
  529ef19

• Fixed broken unit test

  

  Jason Yu

  committed
  19ebef5

Commits on Jul 21, 2017

• Replaced the rateLimiter implementation with a golang library (github.com/golang/time/rate) to resolve an unusually high throughput exceeded exception caused by a race condition. Added a message si…

  Show description for f2159ad
  

  Jason Yu

  committed
  f2159ad

Commits on Jul 24, 2017

• Implemented shard monitoring for the kinesiswriter and the produce function of the producer. Re-arranged the rate limiter properties to be part of the kineswriter and firehosewriter configuration i…

  Show description for a685a90
  

  Jason Yu

  committed
  a685a90

• Added new stats to track the number of concurrent workers for the producer. Fixed a bug with the determination of the number of open shards for a stream.

  

  Jason Yu

  committed
  e26a55c

• Added more thorough error checking around rate limiting configuration values

  

  Jason Yu

  committed
  13421ee

Commits on Jul 25, 2017

• Added a data spill callback function as a configurable parameter.

  

  Jason Yu

  committed
  c66a9e1

• Reset rate limiters upon a detected shard count change event.

  

  Jason Yu

  committed
  b3f5a5f

• Modified the rate limiting implementation in the produce function to fill a batch or timeout first before waiting on the count and size tokens needed to send the batch. Renamed newWorker helper fun…

  Show description for 9788d1e
  

  Jason Yu

  committed
  9788d1e

Commits on Jul 26, 2017

• Minor refactor of listener to clean up a bit.

  

  Jason Yu

  committed
  c892a14

Commits on Jul 27, 2017

• Added rate limiting to the listener. Added payload size as one of the return parameters for the GetRecord and GetRecords interface.

  

  Jason Yu

  committed
  e100d7d

• Remove glide files

  

  Jason Yu

  committed
  385c90c

• Eliminated uncessary packages by moving files around

  

  Jason Yu

  committed
  efc4176

Commits on Jul 28, 2017

• Renamed listener to consumer. Introduced functional options for configuring consumer, producer, kinesisreader, kclreader, kinesiswriter and firehosewriter objects

  

  Jason Yu

  committed
  e4a35be

• Consolidated the producer and consumer packages into the main kinetic package. Simplied the kinetic parameter configuration by applying the functional options pattern.

  

  Jason Yu

  committed
  56194f7

• Documented new data types and functions to fix golint errors. Fixed broken unit tests.

  

  Jason Yu

  committed
  a61c732

• Regenerated glide.yaml and glide.lock files.

  

  Jason Yu

  committed
  a6c3ba5

• Fix vendoring issues.

  

  Jason Yu

  committed
  83b9b32

• Additional changes to glide.yaml file.

  

  Jason Yu

  committed
  adcd33a

• Refactored the produce/consumer constructor methods to not require a StreamWriter/StreamReader. Converted the writer/reader to configurable options which can be configured using a functional option…

  Show description for a352c0b
  

  Jason Yu

  committed
  a352c0b

• Fix broken unit test

  

  Jason Yu

  committed
  f7a09a8

Commits on Jul 31, 2017

• Fix some megacheck errors and updated README.md files.

  

  Jason Yu

  committed
  58b6aa2

Commits on Aug 2, 2017

• Removed the WaitGroup argument from the messageHandler and MessageProcessor callback functions and refactored the APIs to call the callback function asynchronously. Made KCL action types private. R…

  Show description for 66a064f
  

  Jason Yu

  committed
  66a064f

• Refactor KclReader implementation to add startup and shutdown functions. Renamed provisioned through exceeded stats for producer and consumer. Added randomization to dummy messages and fixed partio…

  Show description for 40157db
  

  Jason Yu

  committed
  40157db

Commits on Aug 11, 2017

• Added checkpointing in KclReader and exposed new functions through the StreamReader interface. Refactored KclReader process method to allow checkpoint calls to KCL in between long-poll processRecor…

  Show description for 8c11d72
  

  Jason Yu

  committed
  8c11d72

Commits on Aug 14, 2017

• Expose function option methods (KclReaderLogLevel and KclReaderStats)

  

  Jason Yu

  committed
  9856250

Commits on Aug 18, 2017

• Renamed checkpoint to checkpointer. Removed capacity constraint on checkpointer. Removed node expirary on checkpointer. Added checkpoint stats in consumer's stats collector. Added periodic update o…

  Show description for 80ea7c3
  

  Jason Yu

  committed
  80ea7c3

Commits on Aug 22, 2017

• Refactored the checkpointer implementation to use a doubly linked list (instead of a binary search tree). Modified markDone so delete elements from the list upon forming a chain of sequence numbers…

  Show description for da9aabc
  

  Jason Yu

  committed
  da9aabc

Commits on Aug 23, 2017

• Refactored the signature of the function option methods to operate on the main (public) objects instead of the respective (private) option configuration structs.

  

  Jason Yu

  committed
  739e252

• Added the ability to configure the KinesisReader's GetRecords concurrency by adding a new concurrency field to the kinesisReaderOptions struct. Also added a new function method option to be able to…

  Show description for c37399e
  

  Jason Yu

  committed
  c37399e

• gofmt.

  

  Jason Yu

  committed
  819161d

• Remove Checkpoint methods from StreamReader interface.

  

  Jason Yu

  committed
  fcac48b

Commits on Aug 25, 2017

• Added documentation in doc.go and updated README.md file. Fixed a logical bug in the producer's doWork exit condition. Added read response timeout to FirehoseWriter.

  

  Jason Yu

  committed
  1cd9bfb

• Fix race condition in checkpoint_test.go that was causing tests to fail.

  

  Jason Yu

  committed
  ed36ce9

Commits on Aug 29, 2017

• Moved the rate limiting logic from the Consumer to the KinesisReader. Simplified the GetRecord / GetRecords API to only return an error (instead of returning count and size as well). Updated docume…

  Show description for 6a5bfe2
  

  Jason Yu

  committed
  6a5bfe2

Commits on Aug 30, 2017

• Remove throttle function from KinesisReader as it is now redundant with the count/size rate limiter.

  

  Jason Yu

  committed
  0176c83

Commits on Oct 2, 2017

• Removed unnecessary CloseWithContext function.

  

  Jason Yu

  committed
  e8ffae2

• strip out pipe of death

  

  Ian Chiles

  committed
  116fb8c

Commits on Oct 4, 2017

• Define reader as a bufio.NewReader that reads from STDIN

  

  Jason Yu

  committed
  d48871e

• FU staticcheck

  

  Jason Yu

  committed
  83c9709

• staticcheck and gosimple site is down

  

  Jason Yu

  committed
  bfbb8a2

Commits on Oct 5, 2017

• Added TeeReader for debugging

  

  Jason Yu

  committed
  3c41824

• Added TeeReader for debugging

  

  Jason Yu

  committed
  fde947e

• Print errors with file creation

  

  Jason Yu

  committed
  e79cb77

• Panic

  

  Jason Yu

  committed
  c08d8e4

Commits on Oct 6, 2017

• Revert TeeReader stuff (for debugging) and fix megacheck errors.

  

  Jason Yu

  committed
  854c5f6

• Simplified Consumer.ListenWithContext by eliminatin the select statement

  

  Jason Yu

  committed
  4eaca88

Commits on Oct 9, 2017

• Uncomment out staticcheck and gosimple dependencies in circle.yml file.

  

  Jason Yu

  committed
  94e63ce

• Add default KclReader callback functions

  

  Jason Yu

  committed
  e266d0a

Commits on Oct 27, 2017

• Added a bufio writer for KclReader to write to Stdout.

  

  Jason Yu

  committed
  3b561fa

• Added function option methods for configuring the KclReader's reader/writer.

  

  Jason Yu

  committed
  e0d3689

• Change from fmt.Println to fmt.FPrintln to write to the correct io.Writer

  

  Jason Yu

  committed
  f3225b0

• Fix a bug where kcl_reader was writing to KCL twice.

  

  Jason Yu

  committed
  3efae98

Commits on Mar 1, 2018

• [LM-594] Fixed firehose writer timeout type

  

  Dan Travis

  committed
  46d513e