diff --git a/CHANGELOG.rst b/CHANGELOG.rst index 40267a84af..adef48448d 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -7,14 +7,134 @@ Note that ``RB_ID=#`` and ``PHAB_ID=#`` correspond to associated message in comm Unreleased ---------- -19.1.0 +19.2.0 ------- +Added +~~~~~ +* finatra-kafka: Expose timeout duration in FinagleKafkaConsumerBuilder dest(). ``PHAB_ID=D269701`` + +* finatra-kafka-streams: Expose all existing RocksDb configurations. See + `c.t.f.k.config.FinatraRocksDBConfig` for details on flag names, + descriptions and default values. ``PHAB_ID=D272068`` + +* finatra-kafka-streams: Added two RocksDB flags related to block cache tuning, + `cache_index_and_filter_blocks` and `pin_l0_filter_and_index_blocks_in_cache`. + ``PHAB_ID=D269516`` + +* finatra-kafka: Adding an implicit implementation of + `c.t.app.Flaggable[c.t.finatra.kafka.domain.SeekStrategy]` + and `c.t.app.Flaggable[org.apache.kafka.clients.consumer.OffsetResetStrategy]`. + ``PHAB_ID=D271098`` + +* finatra-http: Added support to serve `c.t.io.Reader` as a streaming response in + `c.t.finatra.http.internal.marshalling.CallbackConverter`. ``PHAB_ID=D266863`` + +* finatra-kafka: Expose endOffsets() in FinagleKafkaConsumer. ``PHAB_ID=D263573`` + +* finatra-kafka-streams: Adding missing ScalaDocs. Adding metric for elapsed state + restore time. RocksDB configuration now contains a flag for adjusting the number + of cache shard bits, `rocksdb.block.cache.shard.bits`. ``PHAB_ID=D255771`` + +* finatra-jackson: Added @Pattern annotation to support finatra/jackson for regex pattern + validation on string values. ``PHAB_ID=D259719`` + +Changed +~~~~~~~ + +* finatra-kafka-streams: Refactor package names. All classes moved from + com.twitter.finatra.streams to com.twitter.finatra.kafkastreams. ``PHAB_ID=D268027`` + +* finatra-kafka-streams: Delete deprecated and unused classes. ``PHAB_ID=D267921`` + +* finatra-kafka-streams: `c.t.finatra.streams.transformer.domain.Time` is now the canonical + representation of time for watermarks and timers. `RichLong` implicit from + `com.twitter.finatra.streams.converters.time` has been renamed to `RichFinatraKafkaStreamsLong`. + ``PHAB_ID=D255736`` + +* finatra-jackson: Fix `CaseClassField` annotation reflection for Scala 2.12. ``PHAB_ID=D264423`` + +* finatra-kafka-streams: Combine FinatraTransformer with FinatraTransformerV2. ``PHAB_ID=D254411`` + +* finatra-thrift: The return type of `ReqRepDarkTrafficFilterModule#newFilter` has been changed from + `DarkTrafficFilter[MethodIface]` to `Filter.TypeAgnostic`. ``PHAB_ID=D261868`` + +* finatra-kafka: Add lookupBootstrapServers function that takes timeout as a parameter. + ``PHAB_ID=D256997`` + +* finatra-thrift: If a Controller is not configured with exactly one endpoint + per method, it will throw an AssertionError instead of logging an error message. + An attempt to use non-legacy functionality with a legacy Controller will throw + an AssertionError. ``PHAB_ID=D260230`` + +* finatra-kafka: Add flags for controlling rocksdb internal LOG file growth. + - `rocksdb.log.info.level` Allows the setting of rocks log levels + `DEBUG_LEVEL`, `INFO_LEVEL`, `WARN_LEVEL`, `ERROR_LEVEL`, `FATAL_LEVEL`, + `HEADER_LEVEL`. + - `rocksdb.log.max.file.size` The maximal size of the info log file. + - `rocksdb.log.keep.file.num` Maximal info log files to be kept. + ``PHAB_ID=D259579`` + +* finatra-kafka: Add admin routes for properties and topology information + - `/admin/kafka/streams/properties` Dumps the + `KafkaStreamsTwitterServer#properties` as plain text in the TwitterServer + admin page. + - `/admin/kafka/streams/topology` Dumps the + `KafkaStreamsTwitterServer#topology` as plain text in the TwitterServer + admin page. + ``PHAB_ID=D259597`` + +* inject-server: EmbeddedTwitterServer that fails to start will now continue to + throw the startup failure on calls to methods that require a successfully started server. + ``PHAB_ID=D265543`` + +Fixed +~~~~~ + +* finatra-kafka-streams: `FinatraTopologyTester` did not set + `TopologyTestDriver#initialWallClockTimeMs` on initialization causing diverging wall clock time + when `TopologyTestDriver#advanceWallClockTime` advanced time. The divergence was between + system time set by `org.joda.time.DateTimeUtils.setCurrentMillisFixed` and internal mock timer + `TopologyTestDriver#mockWallClockTime`. `FinatraTopologyTester.inMemoryStatsReceiver` is reset on + `TopologyFeatureTest#beforeEach` for all test that extend `TopologyFeatureTest`. + ``PHAB_ID=D269013`` + +* finatra-kafka-streams: Improve watermark assignment/propagation upon reading the first + message and when caching key value stores are used. ``PHAB_ID=D262054`` + +* finatra-jackson: Support inherited annotations in case class deserialization. Case class + deserialization support does not properly find inherited Jackson annotations. This means + that code like this: + + ``` + trait MyTrait { + @JsonProperty("differentName") + def name: String + } + case class MyCaseClass(name: String) extends MyTrait + ``` + + would not properly expect an incoming field with name `differentName` to parse into the + case class `name` field. This commit provides support for capturing inherited annotations + on case class fields. Annotations processed in order, thus if the same annotation appears + in the class hierarchy multiple times, the value configured on the class will win otherwise + will be in the order of trait linearization with the "last" declaration prevailing. + ``PHAB_ID=D260376`` + +* finatra: Remove extraneous dependency on old `javax.servlet` ServletAPI dependency. + The fixes #478. ``PHAB_ID=D259671`` + +Closed +~~~~~~ + +19.1.0 +------ + Added ~~~~~ * finatra-kafka-streams: SumAggregator and CompositeSumAggregator only support enhanced window - aggregations for the sum operation. Deprecate SumAggregator and CompositeSumAggregator and create + aggregations for the sum operation. Deprecate SumAggregator and CompositeSumAggregator and create an AggregatorTransformer class that can perform arbitrary aggregations. ``PHAB_ID=D257138`` * finatra-streams: Open-source Finatra Streams. Finatra Streams is an integration @@ -100,11 +220,11 @@ Changed now-removed `c.t.f.b.Server` have been modified or removed. ``PHAB_ID=D254339`` -* finatra-kafka-streams: Finatra Queryable State methods currently require the window size - to be passed into query methods for windowed key value stores. This is unnecessary, as - the queryable state class can be passed the window size at construction time. We also now - save off all FinatraKeyValueStores in a global manager class to allow query services - (e.g. thrift) to access the same KeyValueStore implementation that the FinatraTransformer +* finatra-kafka-streams: Finatra Queryable State methods currently require the window size + to be passed into query methods for windowed key value stores. This is unnecessary, as + the queryable state class can be passed the window size at construction time. We also now + save off all FinatraKeyValueStores in a global manager class to allow query services + (e.g. thrift) to access the same KeyValueStore implementation that the FinatraTransformer is using. ``PHAB_ID=D256920`` Fixed diff --git a/build.sbt b/build.sbt index 92490aba9c..4eb0413774 100644 --- a/build.sbt +++ b/build.sbt @@ -4,7 +4,7 @@ import scoverage.ScoverageKeys concurrentRestrictions in Global += Tags.limit(Tags.Test, 1) // All Twitter library releases are date versioned as YY.MM.patch -val releaseVersion = "19.1.0" +val releaseVersion = "19.2.0" lazy val buildSettings = Seq( version := releaseVersion, @@ -74,7 +74,6 @@ lazy val versions = new { val scalaCheck = "1.13.4" val scalaGuice = "4.1.0" val scalaTest = "3.0.0" - val servletApi = "2.5" val slf4j = "1.7.21" val snakeyaml = "1.12" val specs2 = "2.4.17" @@ -645,7 +644,6 @@ lazy val http = project "com.twitter" %% "finagle-exp" % versions.twLibVersion, "com.twitter" %% "finagle-http" % versions.twLibVersion, "commons-fileupload" % "commons-fileupload" % versions.commonsFileupload, - "javax.servlet" % "servlet-api" % versions.servletApi, "com.novocode" % "junit-interface" % "0.11" % Test ), unmanagedResourceDirectories in Test += baseDirectory( diff --git a/doc/README.md b/doc/README.md deleted file mode 100644 index edb6f7574f..0000000000 --- a/doc/README.md +++ /dev/null @@ -1,25 +0,0 @@ -# Finatra Documentation using [Github Pages](https://pages.github.com/) - -Fast, testable, Scala services built on [TwitterServer][twitter-server] and [Finagle][finagle]. - -Deploy using the `pushsite.bash` script ---------------------------------------- - -``` -$ ./pushsite.bash -``` - -* Changes should be visible at [https://twitter.github.io/finatra](https://twitter.github.io/finatra). - -
-
-
-
-
- -#### Copyright 2013-2018 Twitter, Inc. - -Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0 - -[twitter-server]: https://github.com/twitter/twitter-server -[finagle]: https://github.com/twitter/finagle diff --git a/doc/src/sphinx/FinatraLifecycle.svg b/doc/src/sphinx/FinatraLifecycle.svg new file mode 100644 index 0000000000..6dd3784d9f --- /dev/null +++ b/doc/src/sphinx/FinatraLifecycle.svg @@ -0,0 +1,2 @@ + +
Add admin http server routes
<span>Add admin http server routes</span>
AdminHttpServer#premain
<span>AdminHttpServer#premain</span>
Start server
<span>Start server</span>
TwitterServer#main
TwitterServer#main
TwitterModule#singletonStartup
TwitterModule#singletonStartup
Load modules
Load modules
Modules post injector startup
Modules post injector startup
Warmup
Warmup
Before post warmup
Before post warmup
Post Warmup
Post Warmup
1. announce admin interface
2. add user defined admin routes
3. bind external port(s)
4. announce external interface(s)
[Not supported by viewer]
After Post Warmup
After Post Warmup
enable health endpoint
[Not supported by viewer]
Set app started
Set app started
App#run
App#run<br>
Server#start
Server#start<br>
bind admin port
bind admin port
trigger GC for 
object promotion
[Not supported by viewer]
Post injector startup
Post injector startup
1. resolve Finagle clients
2. configure Server routing
[Not supported by viewer]
install Guice modules
and create injector
[Not supported by viewer] \ No newline at end of file diff --git a/doc/src/sphinx/_static/FinatraLifecycle.png b/doc/src/sphinx/_static/FinatraLifecycle.png index eb09d0a47c..e483fc610e 100644 Binary files a/doc/src/sphinx/_static/FinatraLifecycle.png and b/doc/src/sphinx/_static/FinatraLifecycle.png differ diff --git a/doc/src/sphinx/_static/finatra_logo_text.png b/doc/src/sphinx/_static/finatra_logo_text.png deleted file mode 100644 index be14e04e04..0000000000 Binary files a/doc/src/sphinx/_static/finatra_logo_text.png and /dev/null differ diff --git a/doc/src/sphinx/_static/test-classes.png b/doc/src/sphinx/_static/test-classes.png deleted file mode 100644 index c556d32778..0000000000 Binary files a/doc/src/sphinx/_static/test-classes.png and /dev/null differ diff --git a/doc/src/sphinx/user-guide/getting-started/flags.rst b/doc/src/sphinx/user-guide/getting-started/flags.rst index a23ac71271..7a6a89cef1 100644 --- a/doc/src/sphinx/user-guide/getting-started/flags.rst +++ b/doc/src/sphinx/user-guide/getting-started/flags.rst @@ -17,12 +17,23 @@ of "*environment*\ " string within code, e.g. .. code:: bash - if (env == "production") { ... } + if (env == "production") { ... } It is generally good practice to make Flags *granular* controls that are fully orthogonal to one another. They can then be independently managed for each deploy and this scales consistently as the number of supported "environments" scales. +Global Flags +------------ + +`TwitterUtil `__ `Flags `__ +also has the concept of a "global" flag. That is, a flag that is "global" to the JVM process (as it is +generally defined as a Scala object). In the discussion of Flags with Finatra we **do not** mean +"global" flags unless it is explicitly stated. + +See the `scaladoc `__ for +`c.t.app.GlobalFlag` for more information. + But I have a lot of Flags ------------------------- diff --git a/doc/src/sphinx/user-guide/http/server.rst b/doc/src/sphinx/user-guide/http/server.rst index fb351fb4e3..edfb4d6674 100644 --- a/doc/src/sphinx/user-guide/http/server.rst +++ b/doc/src/sphinx/user-guide/http/server.rst @@ -264,7 +264,8 @@ specifically the server documentation `here `__ you could choose to +The default Response Classifier for HTTP servers is `HttpResponseClassifier.ServerErrorsAsFailures `__, +which classifies any HTTP 5xx response code as a failure. To configure server-side `Response Classification `__ you could choose to set the classifier directly on the underlying Finagle server by overriding the `configureHttpServer` (or `configureHttpsServer`) in your server, e.g., .. code:: scala diff --git a/doc/src/sphinx/user-guide/json/validations.rst b/doc/src/sphinx/user-guide/json/validations.rst index 4761e42599..3bb51891ad 100644 --- a/doc/src/sphinx/user-guide/json/validations.rst +++ b/doc/src/sphinx/user-guide/json/validations.rst @@ -10,6 +10,7 @@ The validations framework integrates Finatra's custom `case class` deserializer - ``@CountryCode`` - ``@FutureTime`` - ``@PastTime`` +- ``@Pattern`` - ``@Max`` - ``@Min`` - ``@NotEmpty`` diff --git a/doc/src/sphinx/user-guide/kafka-streams/index.rst b/doc/src/sphinx/user-guide/kafka-streams/index.rst index cd113f9652..718ccb9e07 100644 --- a/doc/src/sphinx/user-guide/kafka-streams/index.rst +++ b/doc/src/sphinx/user-guide/kafka-streams/index.rst @@ -23,7 +23,7 @@ a fully functional service can be written by simply configuring the Kafka Stream Transformers ~~~~~~~~~~~~ -Implement custom `transformers `__ using `FinatraTransformerV2 `__. +Implement custom `transformers `__ using `FinatraTransformer `__. Aggregations ^^^^^^^^^^^^ diff --git a/doc/src/sphinx/user-guide/logging/asyncappender.rst b/doc/src/sphinx/user-guide/logging/asyncappender.rst index 123f8ff229..3fba9a0a56 100644 --- a/doc/src/sphinx/user-guide/logging/asyncappender.rst +++ b/doc/src/sphinx/user-guide/logging/asyncappender.rst @@ -55,8 +55,8 @@ basics.html#verbosity-levels>`__ of the metrics via a Finagle `Tunable `__. Users need to create a JSON file and place it in the `src/main/resources` folder in -`com/twitter/tunables/finagle/instances.json` to whitelist the Logback metrics. -To whitelist all Logback metrics the JSON file could contain the following: +`com/twitter/tunables/finagle/instances.json` to acceptlist the Logback metrics. +To acceptlist all Logback metrics the JSON file could contain the following: .. code-block:: json diff --git a/doc/src/sphinx/user-guide/testing/embedded.rst b/doc/src/sphinx/user-guide/testing/embedded.rst index 69179c3452..1b62bba5c8 100644 --- a/doc/src/sphinx/user-guide/testing/embedded.rst +++ b/doc/src/sphinx/user-guide/testing/embedded.rst @@ -29,6 +29,46 @@ You'll notice that this hierarchy generally follows the server trait hierarchy a and |c.t.finatra.thrift.ThriftServer|_ extend from |c.t.server.TwitterServer|_ which extends from |c.t.app.App|_. +Testing With `Global Flags` +--------------------------- + +The embedded servers and the embedded app allow for passing `TwitterUtil `__ `Flags `__ +to the server under test via the `flags `__ +constructor argument (a map of flag name to flag value) which is meant to mimic setting flag values +via the command line. + +However it is **not recommended** that users set any |GlobalFlag|_ value in this manner. In normal +usage, the value of a |GlobalFlag|_ is **only read once during the initialization of the JVM process**. + +If you wish to test with toggled values of a |GlobalFlag|_ you should prefer using +|FlagLet|_ or |FlagLetClear|_ in tests instead of passing the |GlobalFlag|_ value via the `flags `__ +arg of an embedded server or embedded app. For example, + +.. code:: scala + + import com.twitter.finatra.http.EmbeddedHttpServer + import com.twitter.finagle.http.Status + import com.twitter.inject.server.FeatureTest + + class ExampleServerFeatureTest extends FeatureTest { + override val server = new EmbeddedHttpServer(new ExampleServer) + + test("ExampleServer#perform feature") { + + someGlobalFlag.let("a value") { + // any read of the `someGlobalFlag` value in this closure will be "a value" + server.httpGet( + path = "/", + andExpect = Status.Ok) + + ??? + } + } + } + +See the `scaladoc `_ for `c.t.app.Flag` +for more information on using |FlagLet|_ or |FlagLetClear|_. + InMemoryStatsReceiver --------------------- @@ -82,3 +122,12 @@ More Information .. |EmbeddedThriftServer| replace:: `EmbeddedThriftServer` .. _EmbeddedThriftServer: https://github.com/twitter/finatra/blob/develop/thrift/src/test/scala/com/twitter/finatra/thrift/EmbeddedThriftServer.scala + +.. |GlobalFlag| replace:: `GlobalFlag` +.. _GlobalFlag: https://github.com/twitter/util/blob/f2a05474ec41f34146d710bdc2a789efd6da9d21/util-app/src/main/scala/com/twitter/app/GlobalFlag.scala + +.. |FlagLet| replace:: `Flag.let` +.. _FlagLet: http://twitter.github.io/util/docs/com/twitter/app/Flag.html#let[R](t:T)(f:=%3ER):R + +.. |FlagLetClear| replace:: `Flag.letClear` +.. _FlagLetClear: http://twitter.github.io/util/docs/com/twitter/app/Flag.html#letClear[R](f:=%3ER):R diff --git a/doc/src/sphinx/user-guide/thrift/filters.rst b/doc/src/sphinx/user-guide/thrift/filters.rst index c57b89fcb9..c70691d9c2 100644 --- a/doc/src/sphinx/user-guide/thrift/filters.rst +++ b/doc/src/sphinx/user-guide/thrift/filters.rst @@ -142,7 +142,7 @@ E.g., for the `UserFilter` defined above (shown with common filters in a recomme .filter[AccessLoggingFilter] .filter[StatsFilter] .filter[ExceptionMappingFilter] - .filter[ClientIdWhitelistFilter] + .filter[ClientIdAcceptlistFilter] .filter[FinagleRequestScopeFilter] .filter[UserFilter] .exceptionMapper[FinatraThriftExceptionMapper] diff --git a/doc/src/sphinx/user-guide/thrift/server.rst b/doc/src/sphinx/user-guide/thrift/server.rst index aa9eac2442..efff4c199f 100644 --- a/doc/src/sphinx/user-guide/thrift/server.rst +++ b/doc/src/sphinx/user-guide/thrift/server.rst @@ -237,7 +237,8 @@ the server documentation `here `__ +The default Response Classifier for Thrift servers is `ThriftResponseClassifier.ThriftExceptionsAsFailures `__, +which classifies any deserialized Thrift Exception as a failure. To configure server-side `Response Classification `__ you could choose to set the classifier directly on the underlying Finagle server by overriding the `configureThriftServer` in your server, e.g., diff --git a/examples/thrift-server/thrift-example-server/src/main/scala/com/twitter/calculator/CalculatorController.scala b/examples/thrift-server/thrift-example-server/src/main/scala/com/twitter/calculator/CalculatorController.scala index 7bfd7ec88d..ce8dca3689 100644 --- a/examples/thrift-server/thrift-example-server/src/main/scala/com/twitter/calculator/CalculatorController.scala +++ b/examples/thrift-server/thrift-example-server/src/main/scala/com/twitter/calculator/CalculatorController.scala @@ -7,18 +7,18 @@ import com.twitter.util.Future import javax.inject.Singleton @Singleton -class CalculatorController extends Controller with Calculator.BaseServiceIface { +class CalculatorController extends Controller(Calculator) { - override val addNumbers = handle(AddNumbers) { args: AddNumbers.Args => + handle(AddNumbers) { args: AddNumbers.Args => info(s"Adding numbers $args.a + $args.b") Future.value(args.a + args.b) } - override val addStrings = handle(AddStrings) { args: AddStrings.Args => + handle(AddStrings) { args: AddStrings.Args => Future.value((args.a.toInt + args.b.toInt).toString) } - override val increment = handle(Increment) { args: Increment.Args => + handle(Increment) { args: Increment.Args => Future.value(args.a + 1) } } diff --git a/examples/thrift-server/thrift-example-server/src/test/scala/com/twitter/calculator/CalculatorServerFeatureTest.scala b/examples/thrift-server/thrift-example-server/src/test/scala/com/twitter/calculator/CalculatorServerFeatureTest.scala index fe7ede9564..932902abc1 100644 --- a/examples/thrift-server/thrift-example-server/src/test/scala/com/twitter/calculator/CalculatorServerFeatureTest.scala +++ b/examples/thrift-server/thrift-example-server/src/test/scala/com/twitter/calculator/CalculatorServerFeatureTest.scala @@ -12,13 +12,13 @@ class CalculatorServerFeatureTest extends FeatureTest { val client = server.thriftClient[Calculator[Future]](clientId = "client123") - test("whitelist#clients allowed") { + test("acceptlist#clients allowed") { await(client.increment(1)) should equal(2) await(client.addNumbers(1, 2)) should equal(3) await(client.addStrings("1", "2")) should equal("3") } - test("blacklist#clients blocked with UnknownClientIdException") { + test("denylist#clients blocked with UnknownClientIdException") { val clientWithUnknownId = server.thriftClient[Calculator[Future]](clientId = "unlisted-client") intercept[UnknownClientIdError] { await(clientWithUnknownId.increment(2)) diff --git a/http/src/main/scala/BUILD b/http/src/main/scala/BUILD index 246d9ce2d8..933d524076 100644 --- a/http/src/main/scala/BUILD +++ b/http/src/main/scala/BUILD @@ -47,7 +47,6 @@ scala_library( "finatra/utils/src/main/scala", "twitter-server/server/src/main/scala", "util/util-app/src/main/scala", - "util/util-collection/src/main/scala", "util/util-core/src/main/scala", "util/util-lint/src/main/scala", "util/util-logging/src/main/scala", diff --git a/http/src/main/scala/com/twitter/finatra/http/internal/marshalling/CallbackConverter.scala b/http/src/main/scala/com/twitter/finatra/http/internal/marshalling/CallbackConverter.scala index 8d4327b721..7912c0fca7 100644 --- a/http/src/main/scala/com/twitter/finatra/http/internal/marshalling/CallbackConverter.scala +++ b/http/src/main/scala/com/twitter/finatra/http/internal/marshalling/CallbackConverter.scala @@ -6,23 +6,22 @@ import com.twitter.finatra.http.internal.marshalling.CallbackConverter.url import com.twitter.finatra.http.response.{ResponseBuilder, StreamingResponse} import com.twitter.finatra.json.FinatraObjectMapper import com.twitter.finatra.json.internal.streaming.JsonStreamParser -import com.twitter.io.Buf +import com.twitter.io.{Buf, Reader} import com.twitter.util.{Future, FuturePool, Promise} import javax.inject.Inject import scala.concurrent.{ExecutionContext => ScalaExecutionContext, Future => ScalaFuture} import scala.util.{Failure, Success} private object CallbackConverter { - val url = "https://twitter.github.io/finatra/user-guide/http/controllers.html#controllers-and-routing" + val url = + "https://twitter.github.io/finatra/user-guide/http/controllers.html#controllers-and-routing" } - private[http] class CallbackConverter @Inject()( messageBodyManager: MessageBodyManager, responseBuilder: ResponseBuilder, mapper: FinatraObjectMapper, - jsonStreamParser: JsonStreamParser -) { + jsonStreamParser: JsonStreamParser) { /* Public */ @@ -76,6 +75,12 @@ private[http] class CallbackConverter @Inject()( StreamingResponse.jsonArray(toBuf = mapper.writeValueAsBuf, asyncStream = asyncStream) streamingResponse.toFutureFinagleResponse + } else if (runtimeClassEq[ResponseType, Reader[_]]) { request: Request => + val reader = requestCallback(request).asInstanceOf[Reader[_]] + val streamingResponse = StreamingResponse.jsonArray( + toBuf = mapper.writeValueAsBuf, + asyncStream = Reader.toAsyncStream(reader)) + streamingResponse.toFutureFinagleResponse } else if (runtimeClassEq[ResponseType, Future[_]]) { request: Request => requestCallback(request).asInstanceOf[Future[_]].map(createHttpResponse(request)) } else if (runtimeClassEq[ResponseType, StreamingResponse[_, _]]) { request: Request => @@ -141,8 +146,8 @@ private[http] class CallbackConverter @Inject()( contentType: String ): Response = { assert( - contentType == responseBuilder.jsonContentType || - contentType == responseBuilder.plainTextContentType + contentType == responseBuilder.jsonContentType || + contentType == responseBuilder.plainTextContentType ) val orig = Response(status) @@ -190,8 +195,11 @@ private[http] class CallbackConverter @Inject()( typeArgs.head.runtimeClass == classOf[Option[_]] } - private def toTwitterFuture[A](scalaFuture: ScalaFuture[A]) - (implicit executor: ScalaExecutionContext): Future[A] = { + private def toTwitterFuture[A]( + scalaFuture: ScalaFuture[A] + )( + implicit executor: ScalaExecutionContext + ): Future[A] = { val p = new Promise[A]() scalaFuture.onComplete { case Success(value) => p.setValue(value) @@ -209,10 +217,8 @@ private[http] class CallbackConverter @Inject()( private[this] val immediatePoolExcCtx = new ExecutionContext(FuturePool.immediatePool) /** ExecutionContext adapter using a FuturePool; see bijection/TwitterExecutionContext */ - private[this] class ExecutionContext( - pool: FuturePool, - report: Throwable => Unit - ) extends ScalaExecutionContext { + private[this] class ExecutionContext(pool: FuturePool, report: Throwable => Unit) + extends ScalaExecutionContext { def this(pool: FuturePool) = this(pool, ExecutionContext.ignore) override def execute(runnable: Runnable): Unit = { pool(runnable.run()) diff --git a/http/src/main/scala/com/twitter/finatra/http/internal/marshalling/MessageBodyManager.scala b/http/src/main/scala/com/twitter/finatra/http/internal/marshalling/MessageBodyManager.scala index dcd11cf5d5..75e0186dbd 100644 --- a/http/src/main/scala/com/twitter/finatra/http/internal/marshalling/MessageBodyManager.scala +++ b/http/src/main/scala/com/twitter/finatra/http/internal/marshalling/MessageBodyManager.scala @@ -29,7 +29,7 @@ import scala.collection.mutable * [[com.twitter.finatra.http.HttpServer.messageBodyModule]]. * * When the MessageBodyManager is obtained from the injector (which is configured with the framework - * [[com.twitter.finatra.http.modules.MessageBodyModule]] the framework default implementations for + * [[com.twitter.finatra.http.modules.MessageBodyModule]]) the framework default implementations for * the reader and writer will be provided accordingly (along with the configured server injector). * * @param injector the configured [[com.twitter.inject.Injector]] for the server. diff --git a/http/src/main/scala/com/twitter/finatra/http/internal/routing/Route.scala b/http/src/main/scala/com/twitter/finatra/http/internal/routing/Route.scala index c9a2d09cdb..cdbc3806b6 100644 --- a/http/src/main/scala/com/twitter/finatra/http/internal/routing/Route.scala +++ b/http/src/main/scala/com/twitter/finatra/http/internal/routing/Route.scala @@ -7,7 +7,6 @@ import com.twitter.finatra.http.internal.request.RequestWithRouteParams import com.twitter.finatra.http.internal.routing.Route._ import com.twitter.util.Future import java.lang.annotation.Annotation -import scala.language.existentials import scala.reflect.ClassTag private[http] object Route { diff --git a/http/src/test/java/BUILD b/http/src/test/java/BUILD index 2acce68fc1..6087959358 100644 --- a/http/src/test/java/BUILD +++ b/http/src/test/java/BUILD @@ -9,7 +9,6 @@ junit_tests( "3rdparty/jvm/junit", "3rdparty/jvm/org/scalatest", "3rdparty/jvm/org/slf4j:slf4j-api", - "finagle/finagle-base-http/src/main/java", "finagle/finagle-base-http/src/main/scala", "finagle/finagle-core/src/main/scala", "finagle/finagle-http/src/main/scala", diff --git a/http/src/test/scala/com/twitter/finatra/http/EmbeddedHttpServer.scala b/http/src/test/scala/com/twitter/finatra/http/EmbeddedHttpServer.scala index cad431fad6..b8e4101687 100644 --- a/http/src/test/scala/com/twitter/finatra/http/EmbeddedHttpServer.scala +++ b/http/src/test/scala/com/twitter/finatra/http/EmbeddedHttpServer.scala @@ -24,7 +24,7 @@ import scala.collection.JavaConverters._ * we default to Stage.DEVELOPMENT. This makes it possible to only mock objects that are used in a given test, * at the expense of not checking that the entire object graph is valid. As such, you should always have at * least one Stage.PRODUCTION test for your service (which eagerly creates all classes at startup) - * @param useSocksProxy Use a tunneled socks proxy for external service discovery/calls (useful for manually run external + * @param useSocksProxy Use a tunneled socks proxy for external service discovery/calls (useful for manually running external * integration tests that connect to external services). * @param defaultRequestHeaders Headers to always send to the embedded server. * @param defaultHttpSecure Default all requests to the server to be HTTPS. diff --git a/http/src/test/scala/com/twitter/finatra/http/ExternalHttpClient.scala b/http/src/test/scala/com/twitter/finatra/http/ExternalHttpClient.scala index ae633cbb27..1bd22a0beb 100644 --- a/http/src/test/scala/com/twitter/finatra/http/ExternalHttpClient.scala +++ b/http/src/test/scala/com/twitter/finatra/http/ExternalHttpClient.scala @@ -26,6 +26,9 @@ private[twitter] trait ExternalHttpClient { self: EmbeddedTwitterServer => /** Provide an override to the underlying server's mapper */ def mapperOverride: Option[FinatraObjectMapper] + /** Provide an override to the external HTTPS client */ + private[twitter] def httpsClientOverride: Option[JsonAwareEmbeddedHttpClient] = None + /* Overrides */ /** Logs the external http and/or https host and port of the underlying EmbeddedHttpServer */ @@ -108,18 +111,11 @@ private[twitter] trait ExternalHttpClient { self: EmbeddedTwitterServer => mapper, disableLogging = self.disableLogging ) - closeOnExit { - if (isStarted) { - Closable.make { deadline => - info(s"Closing embedded http client: ${client.label}", disableLogging) - client.close(deadline) - } - } else Closable.nop - } + closeOnExit(client) client } - final lazy val httpsClient: JsonAwareEmbeddedHttpClient = { + final lazy val httpsClient: JsonAwareEmbeddedHttpClient = httpsClientOverride.getOrElse { val client = new JsonAwareEmbeddedHttpClient( "httpsClient", httpsExternalPort(), @@ -130,14 +126,16 @@ private[twitter] trait ExternalHttpClient { self: EmbeddedTwitterServer => mapper, disableLogging = self.disableLogging ) - closeOnExit { - if (isStarted) { - Closable.make { deadline => - info(s"Closing embedded http client: ${client.label}", disableLogging) - client.close(deadline) - } - } else Closable.nop - } + closeOnExit(client) client } + + final def closeOnExit(client: JsonAwareEmbeddedHttpClient): Unit = closeOnExit { + if (isStarted) { + Closable.make { deadline => + info(s"Closing embedded http client: ${client.label}", disableLogging) + client.close(deadline) + } + } else Closable.nop + } } diff --git a/http/src/test/scala/com/twitter/finatra/http/tests/integration/json/JacksonIntegrationServerFeatureTest.scala b/http/src/test/scala/com/twitter/finatra/http/tests/integration/json/JacksonIntegrationServerFeatureTest.scala index 0a2da498b2..5a3bba7a5d 100644 --- a/http/src/test/scala/com/twitter/finatra/http/tests/integration/json/JacksonIntegrationServerFeatureTest.scala +++ b/http/src/test/scala/com/twitter/finatra/http/tests/integration/json/JacksonIntegrationServerFeatureTest.scala @@ -1,5 +1,6 @@ package com.twitter.finatra.http.tests.integration.json +import com.twitter.finagle.http.{Request, Response} import com.twitter.finagle.http.Status.BadRequest import com.twitter.finatra.http.{Controller, EmbeddedHttpServer, HttpServer} import com.twitter.finatra.http.filters.CommonFilters @@ -20,6 +21,19 @@ class JacksonIntegrationServerFeatureTest extends FeatureTest { post("/personWithThings") { _: PersonWithThingsRequest => "Accepted" } + + get("/users/lookup") { request: UserLookupRequest => + Map( + "ids" -> request.ids, + "names" -> request.names, + "format" -> request.format, + "userFormat" -> request.userFormat, + "statusFormat" -> request.statusFormat, + "acceptHeader" -> request.acceptHeader, + "validationPassesForIds" -> request.validationPassesForIds, + "validationPassesForNames" -> request.validationPassesForNames + ) + } } ) } @@ -47,4 +61,22 @@ class JacksonIntegrationServerFeatureTest extends FeatureTest { andExpect = BadRequest, withJsonBody = """{"errors":["things: Unable to parse"]}""") } + + test("/GET UserLookup") { + + val response: Response = server.httpGet( + "/users/lookup?ids=21345", + headers = Map("accept" -> "application/vnd.foo+json") + ) + + response.status.code shouldBe 200 + val responseMap = server.mapper.parse[Map[String, String]](response.contentString) + responseMap("ids") should be("21345") + responseMap("format") should be(null) + responseMap("userFormat") should be(null) + responseMap("statusFormat") should be(null) + responseMap("validationPassesForIds").toBoolean should be(true) + responseMap("validationPassesForNames").toBoolean should be(true) + responseMap("acceptHeader") should be("application/vnd.foo+json") + } } diff --git a/http/src/test/scala/com/twitter/finatra/http/tests/integration/json/PersonWithThingsRequest.scala b/http/src/test/scala/com/twitter/finatra/http/tests/integration/json/PersonWithThingsRequest.scala deleted file mode 100644 index 27462051b5..0000000000 --- a/http/src/test/scala/com/twitter/finatra/http/tests/integration/json/PersonWithThingsRequest.scala +++ /dev/null @@ -1,12 +0,0 @@ -package com.twitter.finatra.http.tests.integration.json - -import com.twitter.finatra.validation.Size - -case class PersonWithThingsRequest( - id: Int, - name: String, - age: Option[Int], - @Size(min = 1, max = 10) things: Map[String, Things]) - -case class Things( - @Size(min = 1, max = 2) names: Seq[String]) diff --git a/http/src/test/scala/com/twitter/finatra/http/tests/integration/json/caseclasses.scala b/http/src/test/scala/com/twitter/finatra/http/tests/integration/json/caseclasses.scala new file mode 100644 index 0000000000..eb3173a95c --- /dev/null +++ b/http/src/test/scala/com/twitter/finatra/http/tests/integration/json/caseclasses.scala @@ -0,0 +1,131 @@ +package com.twitter.finatra.http.tests.integration.json + +import com.twitter.finatra.request.{Header, QueryParam} +import com.twitter.finatra.validation.{MethodValidation, Size, ValidationResult} + +case class PersonWithThingsRequest( + id: Int, + name: String, + age: Option[Int], + @Size(min = 1, max = 10) things: Map[String, Things]) + +case class Things( + @Size(min = 1, max = 2) names: Seq[String]) + +trait TestRequest { + protected[this] val ValidFormats: Seq[String] = Seq("compact", "default", "detailed") + + protected[this] def validateFormat(formatValue: Option[String], formatKey: String): ValidationResult = { + if (formatValue.isEmpty) { + ValidationResult.Valid + } else { + val actualFormat = formatValue.get + val errorMsg = s"Bad parameter value: <$actualFormat>." + + s" The only format values allowed for <$formatKey> are ${ValidFormats.mkString(",")}" + ValidationResult.validate(ValidFormats.contains(actualFormat), errorMsg) + } + } + + protected[this] def validateListOfLongIds(commaSeparatedListOfIds: String): Boolean = { + val actualIdsString = commaSeparatedListOfIds.trim() + if (actualIdsString.isEmpty) { + false + } else { + actualIdsString + .split("\\,").map { anEntry => + anEntry.trim.nonEmpty && anEntry.matches("\\d+") + }.forall(_ == true) + } + } + + protected[this] def validateListOfUsers(users: Option[String]): Boolean = { + users.forall { names => + val namesTrimmed = names.trim() + if (namesTrimmed.isEmpty) { + false + } else { + namesTrimmed + .split("\\,").map { anEntry => + anEntry.trim.nonEmpty + }.forall(_ == true) + } + } + } + + protected[this] def extractListOfLongIds(idsString: String): Seq[Long] = { + val items = idsString.trim().split(",") + items.map { anItem => + anItem.toLong + }.toSeq + } + + def listOfStrings(namesValue: Option[String]): Seq[String] = { + { + namesValue.map { + _.split(",").toSeq + } + }.getOrElse(Seq[String]()) + } + + def createErrorMessage(paramName: String, badValue: String, errMsg: String): String = { + s"Bad Value: '$badValue' for parameter '$paramName'. $errMsg" + } + + def createErrorMessage(paramName: String, badValue: Option[String], errMsg: String): String = { + createErrorMessage(paramName, badValue.getOrElse("None"), errMsg) + } +} +case class UserLookupRequest( + @QueryParam ids: Option[String] = None, + @QueryParam names: Option[String] = None, + @QueryParam format: Option[String] = None, + @QueryParam("user.format") userFormat: Option[String] = None, + @QueryParam("status.format") statusFormat: Option[String] = None, + @Header("Accept") acceptHeader: Option[String] = None) + extends TestRequest { + + lazy val validationPassesForIds: Boolean = ids.forall(validateListOfLongIds) + lazy val validationPassesForNames: Boolean = validateListOfUsers(names) + + @MethodValidation + def validateIds(): ValidationResult = + ValidationResult.validate( + validationPassesForIds, + createErrorMessage("ids", ids, "Must be a comma separated list of decimal numbers.") + ) + + @MethodValidation + def validateNames(): ValidationResult = { + ValidationResult.validate( + validationPassesForNames, + createErrorMessage( + "names", + names, + "Must be a comma separated list of names." + ) + ) + } + + @MethodValidation + def validateUserFormat(): ValidationResult = + validateFormat(userFormat, "user.format") + + @MethodValidation + def validateStatusFormat(): ValidationResult = + validateFormat(statusFormat, "status.format") + + @MethodValidation + def validateMinimalRequestParams: ValidationResult = { + // in case one of the validations failed, don't add this error message + val atLeastOneValidEntry = + (!validationPassesForIds || !validationPassesForNames) || + listOfIds.nonEmpty || listOfNames.nonEmpty + ValidationResult.validate( + atLeastOneValidEntry, + "At least one valid id or one valid name must be provided" + ) + } + + private[this] val listOfIds: Seq[Long] = ids.fold(Seq.empty[Long])(extractListOfLongIds) + private[this] val listOfNames: Seq[String] = listOfStrings(names) +} diff --git a/http/src/test/scala/com/twitter/finatra/http/tests/marshalling/CallbackConverterIntegrationTest.scala b/http/src/test/scala/com/twitter/finatra/http/tests/marshalling/CallbackConverterIntegrationTest.scala index 54f3a894b5..18d25e1bcb 100644 --- a/http/src/test/scala/com/twitter/finatra/http/tests/marshalling/CallbackConverterIntegrationTest.scala +++ b/http/src/test/scala/com/twitter/finatra/http/tests/marshalling/CallbackConverterIntegrationTest.scala @@ -206,16 +206,16 @@ class CallbackConverterIntegrationTest extends IntegrationTest with Mockito { val converted = callbackConverter.convertToFutureResponse(asyncStreamRequest) - val response = Await.result(converted(request)) + val response = await(converted(request)) assertOk(response, "List(1, 2)") } test("AsyncStream response") { val converted = callbackConverter.convertToFutureResponse(asyncStreamResponse) - val response = Await.result(converted(Request())) + val response = await(converted(Request())) response.status should equal(Status.Ok) - Await.result(Reader.readAll(response.reader)).utf8str should equal("[1,2,3]") + await(Reader.readAll(response.reader)).utf8str should equal("[1,2,3]") } test("AsyncStream request and response") { @@ -228,9 +228,17 @@ class CallbackConverterIntegrationTest extends IntegrationTest with Mockito { val converted = callbackConverter.convertToFutureResponse(asyncStreamRequestAndResponse) - val response = Await.result(converted(request)) + val response = await(converted(request)) response.status should equal(Status.Ok) - Await.result(Reader.readAll(response.reader)).utf8str should equal("""["1","2"]""") + await(Reader.readAll(response.reader)).utf8str should equal("""["1","2"]""") + } + + test("Reader response") { + val converted = callbackConverter.convertToFutureResponse(readerResponse) + + val response = await(converted(Request())) + response.status should equal(Status.Ok) + await(Reader.readAll(response.reader)).utf8str should equal("[1.1,2.2,3.3]") } test("Null") { @@ -369,18 +377,22 @@ class CallbackConverterIntegrationTest extends IntegrationTest with Mockito { AsyncStream(1, 2, 3) } + def readerResponse(request: Request): Reader[Double] = { + Reader.fromSeq(Seq(1.1, 2.2, 3.3)) + } + private def assertOk(response: Response, expectedBody: String): Unit = { response.status should equal(Status.Ok) response.contentString should equal(expectedBody) } private def assertOk(convertedFunc: (Request) => Future[Response], withBody: String): Unit = { - val response = Await.result(convertedFunc(Request())) + val response = await(convertedFunc(Request())) assertOk(response, withBody) } private def assertStatus(convertedFunc: (Request) => Future[Response], expectedStatus: Status): Unit = { - val response = Await.result(convertedFunc(Request())) + val response = await(convertedFunc(Request())) response.status should equal(expectedStatus) } } diff --git a/inject/inject-app/src/main/java/BUILD b/inject/inject-app/src/main/java/BUILD index 02c5043ee5..b4a4dbe850 100644 --- a/inject/inject-app/src/main/java/BUILD +++ b/inject/inject-app/src/main/java/BUILD @@ -11,3 +11,10 @@ java_library( "3rdparty/jvm/com/google/inject:guice", ], ) + +# TODO: Remove this and references to it, +# when a fix for https://github.com/pantsbuild/pants/issues/7200 has landed. +files( + name = "pants-workaround", + sources = rglobs("*.java"), +) diff --git a/inject/inject-app/src/main/scala/BUILD b/inject/inject-app/src/main/scala/BUILD index 5d226540c7..98bd34cfa6 100644 --- a/inject/inject-app/src/main/scala/BUILD +++ b/inject/inject-app/src/main/scala/BUILD @@ -22,6 +22,7 @@ scala_library( "3rdparty/jvm/org/slf4j:jul-to-slf4j", "3rdparty/jvm/org/slf4j:log4j-over-slf4j", "3rdparty/jvm/org/slf4j:slf4j-api", + "finatra/inject/inject-app/src/main/java:pants-workaround", "finatra/inject/inject-core/src/main/scala", "finatra/inject/inject-slf4j/src/main/scala", "util/util-app/src/main/scala", diff --git a/inject/inject-app/src/test/scala/com/twitter/inject/app/BindDSL.scala b/inject/inject-app/src/test/scala/com/twitter/inject/app/BindDSL.scala index ffdcf2fa27..98ad4daa0c 100644 --- a/inject/inject-app/src/test/scala/com/twitter/inject/app/BindDSL.scala +++ b/inject/inject-app/src/test/scala/com/twitter/inject/app/BindDSL.scala @@ -58,7 +58,7 @@ private[twitter] trait BindDSL { self => def bindClass[T](clazz: Class[T]): ClassDSL[T] = new ClassDSL[T](clazz) /** For Java Compatibility */ - def bindClass[T](clazz: Class[T], instance: T): self.type = { + def bindClass[T](clazz: Class[T], instance: T): BindDSL.this.type = { addInjectionServiceModule(new TwitterModule { override def configure(): Unit = { bind(clazz).toInstance(instance) @@ -68,7 +68,7 @@ private[twitter] trait BindDSL { self => } /** For Java Compatibility */ - def bindClass[T](clazz: Class[T], annotation: Annotation, instance: T): self.type = { + def bindClass[T](clazz: Class[T], annotation: Annotation, instance: T): BindDSL.this.type = { addInjectionServiceModule(new TwitterModule { override def configure(): Unit = { bind(clazz).annotatedWith(annotation).toInstance(instance) @@ -78,7 +78,7 @@ private[twitter] trait BindDSL { self => } /** For Java Compatibility */ - def bindClass[T, Ann <: Annotation](clazz: Class[T], annotationClazz: Class[Ann], instance: T): self.type = { + def bindClass[T, Ann <: Annotation](clazz: Class[T], annotationClazz: Class[Ann], instance: T): BindDSL.this.type = { addInjectionServiceModule(new TwitterModule { override def configure(): Unit = { bind(clazz).annotatedWith(annotationClazz).toInstance(instance) @@ -88,7 +88,7 @@ private[twitter] trait BindDSL { self => } /** For Java Compatibility */ - def bindClass[T, U <: T](clazz: Class[T], instanceClazz: Class[U]): self.type = { + def bindClass[T, U <: T](clazz: Class[T], instanceClazz: Class[U]): BindDSL.this.type = { addInjectionServiceModule(new TwitterModule { override def configure(): Unit = { bind(clazz).to(instanceClazz) @@ -98,7 +98,7 @@ private[twitter] trait BindDSL { self => } /** For Java Compatibility */ - def bindClass[T, U <: T](clazz: Class[T], annotation: Annotation, instanceClazz: Class[U]): self.type = { + def bindClass[T, U <: T](clazz: Class[T], annotation: Annotation, instanceClazz: Class[U]): BindDSL.this.type = { addInjectionServiceModule(new TwitterModule { override def configure(): Unit = { bind(clazz).annotatedWith(annotation).to(instanceClazz) @@ -108,7 +108,7 @@ private[twitter] trait BindDSL { self => } /** For Java Compatibility */ - def bindClass[T, Ann <: Annotation, U <: T](clazz: Class[T], annotationClazz: Class[Ann], instanceClazz: Class[U]): self.type = { + def bindClass[T, Ann <: Annotation, U <: T](clazz: Class[T], annotationClazz: Class[Ann], instanceClazz: Class[U]): BindDSL.this.type = { addInjectionServiceModule(new TwitterModule { override def configure(): Unit = { bind(clazz).annotatedWith(annotationClazz).to(instanceClazz) @@ -120,7 +120,7 @@ private[twitter] trait BindDSL { self => /* Private */ private[app] class TypeDSL[T: TypeTag] { - def toInstance(instance: T): self.type = { + def toInstance(instance: T): BindDSL.this.type = { addInjectionServiceModule(new TwitterModule { override def configure(): Unit = { bind(asManifest[T]).toInstance(instance) @@ -129,7 +129,7 @@ private[twitter] trait BindDSL { self => self } - def to[U <: T: TypeTag]: self.type = { + def to[U <: T: TypeTag]: BindDSL.this.type = { addInjectionServiceModule(new TwitterModule { override def configure(): Unit = { bind(asManifest[T]).to(asManifest[U]) @@ -138,7 +138,7 @@ private[twitter] trait BindDSL { self => self } - def to[U <: T: TypeTag](instanceClazz: Class[U]): self.type = { + def to[U <: T: TypeTag](instanceClazz: Class[U]): BindDSL.this.type = { addInjectionServiceModule(new TwitterModule { override def configure(): Unit = { bind(asManifest[T]).to(instanceClazz) @@ -155,7 +155,7 @@ private[twitter] trait BindDSL { self => } private[app] class TypeAnnotationDSL[T: TypeTag, Ann <: Annotation: TypeTag] extends TypeDSL[T] { - override def toInstance(instance: T): self.type = { + override def toInstance(instance: T): BindDSL.this.type = { addInjectionServiceModule(new TwitterModule { override def configure(): Unit = { bind(asManifest[T], asManifest[Ann]).toInstance(instance) @@ -164,7 +164,7 @@ private[twitter] trait BindDSL { self => self } - override def to[U <: T: TypeTag]: self.type = { + override def to[U <: T: TypeTag]: BindDSL.this.type = { addInjectionServiceModule(new TwitterModule { override def configure(): Unit = { bind(asManifest[T]).annotatedWith(asManifest[Ann]).to(asManifest[U]) @@ -173,7 +173,7 @@ private[twitter] trait BindDSL { self => self } - override def to[U <: T: TypeTag](instanceClazz: Class[U]): self.type = { + override def to[U <: T: TypeTag](instanceClazz: Class[U]): BindDSL.this.type = { addInjectionServiceModule(new TwitterModule { override def configure(): Unit = { bind(asManifest[T]).annotatedWith(asManifest[Ann]).to(instanceClazz) @@ -184,7 +184,7 @@ private[twitter] trait BindDSL { self => } private[app] class TypeAnnotationClassDSL[T: TypeTag, Ann <: Annotation](annotationClazz: Class[Ann]) extends TypeDSL[T] { - override def toInstance(instance: T): self.type = { + override def toInstance(instance: T): BindDSL.this.type = { addInjectionServiceModule(new TwitterModule { override def configure(): Unit = { bind(asManifest[T]).annotatedWith(annotationClazz).toInstance(instance) @@ -193,7 +193,7 @@ private[twitter] trait BindDSL { self => self } - override def to[U <: T: TypeTag]: self.type = { + override def to[U <: T: TypeTag]: BindDSL.this.type = { addInjectionServiceModule(new TwitterModule { override def configure(): Unit = { bind(asManifest[T]).annotatedWith(annotationClazz).to(asManifest[U]) @@ -202,7 +202,7 @@ private[twitter] trait BindDSL { self => self } - override def to[U <: T: TypeTag](instanceClazz: Class[U]): self.type = { + override def to[U <: T: TypeTag](instanceClazz: Class[U]): BindDSL.this.type = { addInjectionServiceModule(new TwitterModule { override def configure(): Unit = { bind(asManifest[T]).annotatedWith(annotationClazz).to(instanceClazz) @@ -213,7 +213,7 @@ private[twitter] trait BindDSL { self => } private[app] class TypeWithNamedAnnotationDSL[T: TypeTag](annotation: Annotation) extends TypeDSL[T] { - override def toInstance(instance: T): self.type = { + override def toInstance(instance: T): BindDSL.this.type = { addInjectionServiceModule(new TwitterModule { override def configure(): Unit = { bind(asManifest[T]).annotatedWith(annotation).toInstance(instance) @@ -222,7 +222,7 @@ private[twitter] trait BindDSL { self => self } - override def to[U <: T: TypeTag]: self.type = { + override def to[U <: T: TypeTag]: BindDSL.this.type = { addInjectionServiceModule(new TwitterModule { override def configure(): Unit = { bind(asManifest[T]).annotatedWith(annotation).to(asManifest[U]) @@ -231,7 +231,7 @@ private[twitter] trait BindDSL { self => self } - override def to[U <: T: TypeTag](instanceClazz: Class[U]): self.type = { + override def to[U <: T: TypeTag](instanceClazz: Class[U]): BindDSL.this.type = { addInjectionServiceModule(new TwitterModule { override def configure(): Unit = { bind(asManifest[T]).annotatedWith(annotation).to(instanceClazz) @@ -242,7 +242,7 @@ private[twitter] trait BindDSL { self => } private[app] class ClassDSL[T](clazz: Class[T]) { - def toInstance(instance: T): self.type = { + def toInstance(instance: T): BindDSL.this.type = { addInjectionServiceModule(new TwitterModule { override def configure(): Unit = { bind(clazz).toInstance(instance) @@ -251,7 +251,7 @@ private[twitter] trait BindDSL { self => self } - def to[U <: T : TypeTag : ClassTag]: self.type = { + def to[U <: T : TypeTag : ClassTag]: BindDSL.this.type = { addInjectionServiceModule(new TwitterModule { override def configure(): Unit = { bind(clazz).to(typeLiteral[U](asManifest[U])) @@ -260,7 +260,7 @@ private[twitter] trait BindDSL { self => self } - def to[U <: T](instanceClazz: Class[U]): self.type = { + def to[U <: T](instanceClazz: Class[U]): BindDSL.this.type = { addInjectionServiceModule(new TwitterModule { override def configure(): Unit = { bind(clazz).to(instanceClazz) @@ -275,7 +275,7 @@ private[twitter] trait BindDSL { self => } private[app] class ClassAnnotationDSL[T, Ann <: Annotation](clazz: Class[T], annotationClazz: Class[Ann]) extends ClassDSL(clazz) { - override def toInstance(instance: T): self.type = { + override def toInstance(instance: T): BindDSL.this.type = { addInjectionServiceModule(new TwitterModule { override def configure(): Unit = { bind(clazz).annotatedWith(annotationClazz).toInstance(instance) @@ -284,7 +284,7 @@ private[twitter] trait BindDSL { self => self } - override def to[U <: T : TypeTag : ClassTag]: self.type = { + override def to[U <: T : TypeTag : ClassTag]: BindDSL.this.type = { addInjectionServiceModule(new TwitterModule { override def configure(): Unit = { bind(clazz).annotatedWith(annotationClazz).to(typeLiteral[U](asManifest[U])) @@ -293,7 +293,7 @@ private[twitter] trait BindDSL { self => self } - override def to[U <: T](instanceClazz: Class[U]): self.type = { + override def to[U <: T](instanceClazz: Class[U]): BindDSL.this.type = { addInjectionServiceModule(new TwitterModule { override def configure(): Unit = { bind(clazz).annotatedWith(annotationClazz).to(instanceClazz) @@ -304,7 +304,7 @@ private[twitter] trait BindDSL { self => } private[app] class ClassWithNamedAnnotationDSL[T](clazz: Class[T], annotation: Annotation) extends ClassDSL(clazz) { - override def toInstance(instance: T): self.type = { + override def toInstance(instance: T): BindDSL.this.type = { addInjectionServiceModule(new TwitterModule { override def configure(): Unit = { bind(clazz).annotatedWith(annotation).toInstance(instance) @@ -313,7 +313,7 @@ private[twitter] trait BindDSL { self => self } - override def to[U <: T : TypeTag : ClassTag]: self.type = { + override def to[U <: T : TypeTag : ClassTag]: BindDSL.this.type = { addInjectionServiceModule(new TwitterModule { override def configure(): Unit = { bind(clazz).annotatedWith(annotation).to(typeLiteral[U](asManifest[U])) @@ -323,7 +323,7 @@ private[twitter] trait BindDSL { self => } - override def to[U <: T](instanceClazz: Class[U]): self.type = { + override def to[U <: T](instanceClazz: Class[U]): BindDSL.this.type = { addInjectionServiceModule(new TwitterModule { override def configure(): Unit = { bind(clazz).annotatedWith(annotation).to(instanceClazz) diff --git a/inject/inject-core/src/test/java/BUILD b/inject/inject-core/src/test/java/BUILD index 696e8ff3f8..7061434bc8 100644 --- a/inject/inject-core/src/test/java/BUILD +++ b/inject/inject-core/src/test/java/BUILD @@ -5,3 +5,10 @@ java_library( "3rdparty/jvm/com/google/inject:guice", ], ) + +# TODO: Remove this and references to it, +# when a fix for https://github.com/pantsbuild/pants/issues/7200 has landed. +files( + name = "pants-workaround", + sources = rglobs("*.java"), +) diff --git a/inject/inject-core/src/test/scala/BUILD b/inject/inject-core/src/test/scala/BUILD index 5a21658624..5d2a47a275 100644 --- a/inject/inject-core/src/test/scala/BUILD +++ b/inject/inject-core/src/test/scala/BUILD @@ -52,6 +52,7 @@ scala_library( "3rdparty/jvm/org/slf4j:slf4j-api", "3rdparty/jvm/org/specs2:mock", "finatra/inject/inject-core/src/main/scala", + "finatra/inject/inject-core/src/test/java:pants-workaround", "finatra/inject/inject-slf4j/src/main/scala", "util/util-core/src/main/scala", "util/util-slf4j-api/src/main/scala", diff --git a/inject/inject-request-scope/src/main/java/BUILD b/inject/inject-request-scope/src/main/java/BUILD index 5413211c51..2b40551b24 100644 --- a/inject/inject-request-scope/src/main/java/BUILD +++ b/inject/inject-request-scope/src/main/java/BUILD @@ -10,3 +10,10 @@ java_library( "3rdparty/jvm/com/google/inject:guice", ], ) + +# TODO: Remove this and references to it, +# when a fix for https://github.com/pantsbuild/pants/issues/7200 has landed. +files( + name = "pants-workaround", + sources = rglobs("*.java"), +) diff --git a/inject/inject-request-scope/src/main/scala/BUILD b/inject/inject-request-scope/src/main/scala/BUILD index 0177fdab2b..96e8d941c1 100644 --- a/inject/inject-request-scope/src/main/scala/BUILD +++ b/inject/inject-request-scope/src/main/scala/BUILD @@ -18,6 +18,7 @@ scala_library( "3rdparty/jvm/org/slf4j:slf4j-api", "finagle/finagle-core/src/main/scala", "finatra/inject/inject-core/src/main/scala", + "finatra/inject/inject-request-scope/src/main/java:pants-workaround", "finatra/inject/inject-slf4j/src/main/scala", "util/util-app/src/main/scala", "util/util-core/src/main/scala", diff --git a/inject/inject-server/src/main/scala/com/twitter/inject/server/TwitterServer.scala b/inject/inject-server/src/main/scala/com/twitter/inject/server/TwitterServer.scala index d5f07b2fee..414f523de7 100644 --- a/inject/inject-server/src/main/scala/com/twitter/inject/server/TwitterServer.scala +++ b/inject/inject-server/src/main/scala/com/twitter/inject/server/TwitterServer.scala @@ -172,7 +172,7 @@ trait TwitterServer /* Overrides */ override final def main(): Unit = { - super.main() // Call inject.App.main() to create Injector + super[App].main() // Call inject.App.main() to create Injector info("Startup complete, server awaiting.") Awaiter.any(awaitables.asScala, period = 1.second) @@ -190,7 +190,7 @@ trait TwitterServer */ @Lifecycle override protected def postInjectorStartup(): Unit = { - super.postInjectorStartup() + super[App].postInjectorStartup() if (resolveFinagleClientsOnStartup) { info("Resolving Finagle clients before warmup") @@ -278,7 +278,7 @@ trait TwitterServer */ @Lifecycle override protected def beforePostWarmup(): Unit = { - super.beforePostWarmup() + super[App].beforePostWarmup() // trigger gc before accepting traffic prebindWarmup() @@ -298,7 +298,7 @@ trait TwitterServer */ @Lifecycle override protected def postWarmup(): Unit = { - super.postWarmup() + super[App].postWarmup() if (disableAdminHttpServer) { info("Disabling the Admin HTTP Server since disableAdminHttpServer=true") @@ -324,7 +324,7 @@ trait TwitterServer */ @Lifecycle override protected def afterPostWarmup(): Unit = { - super.afterPostWarmup() + super[App].afterPostWarmup() if (!disableAdminHttpServer) { info("admin http server started on port " + PortUtils.getPort(adminHttpServer)) diff --git a/inject/inject-server/src/test/scala/com/twitter/inject/server/EmbeddedHttpClient.scala b/inject/inject-server/src/test/scala/com/twitter/inject/server/EmbeddedHttpClient.scala index ec1496b731..e0da16da78 100644 --- a/inject/inject-server/src/test/scala/com/twitter/inject/server/EmbeddedHttpClient.scala +++ b/inject/inject-server/src/test/scala/com/twitter/inject/server/EmbeddedHttpClient.scala @@ -73,7 +73,7 @@ private[twitter] class EmbeddedHttpClient private[twitter] ( .withStreaming(_streamResponses) .withLabel(label) if (tls) { - client = client.withTlsWithoutValidation + client = configureTls(client) } private[twitter] val service: Service[Request, Response] = @@ -196,6 +196,9 @@ private[twitter] class EmbeddedHttpClient private[twitter] ( protected def disableLogging(suppress: Boolean): Boolean = suppress || this._disableLogging + protected[twitter] def configureTls(client: Http.Client): Http.Client = + client.withTlsWithoutValidation + /* Private */ // Deletes request headers with null values in map. diff --git a/inject/inject-server/src/test/scala/com/twitter/inject/server/EmbeddedTwitterServer.scala b/inject/inject-server/src/test/scala/com/twitter/inject/server/EmbeddedTwitterServer.scala index 839f6e34ca..e2224ace51 100644 --- a/inject/inject-server/src/test/scala/com/twitter/inject/server/EmbeddedTwitterServer.scala +++ b/inject/inject-server/src/test/scala/com/twitter/inject/server/EmbeddedTwitterServer.scala @@ -205,8 +205,17 @@ class EmbeddedTwitterServer( started = true //mutation starting = false //mutation } + + //because start() can be called lazily multiple times, + //subsequent calls will not enter the initialization loop. + //we need to throw here if there was an error for the initial start() call. + throwIfStartupFailed() } + /** If the [[startupFailedThrowable]] is defined, [[throwStartupFailedException]] */ + private def throwIfStartupFailed(): Unit = + if (startupFailedThrowable.isDefined) throwStartupFailedException() + /** Assert the underlying TwitterServer has started */ def assertStarted(started: Boolean = true): Unit = { assert(isInjectable) @@ -448,9 +457,7 @@ class EmbeddedTwitterServer( for (_ <- 1 to maxStartupTimeSeconds) { info("Waiting for warmup phases to complete...", disableLogging) - if (startupFailedThrowable.isDefined) { - throwStartupFailedException() - } + throwIfStartupFailed() if ((isInjectable && injectableServer.started) || (!isInjectable && nonInjectableServerStarted)) { diff --git a/inject/inject-server/src/test/scala/com/twitter/inject/server/tests/EmbeddedTwitterServerIntegrationTest.scala b/inject/inject-server/src/test/scala/com/twitter/inject/server/tests/EmbeddedTwitterServerIntegrationTest.scala index af060d4883..20916814f4 100644 --- a/inject/inject-server/src/test/scala/com/twitter/inject/server/tests/EmbeddedTwitterServerIntegrationTest.scala +++ b/inject/inject-server/src/test/scala/com/twitter/inject/server/tests/EmbeddedTwitterServerIntegrationTest.scala @@ -88,6 +88,29 @@ class EmbeddedTwitterServerIntegrationTest extends Test { } } + test("server#failed startup throws startup error on future method calls") { + val server = new EmbeddedTwitterServer( + twitterServer = new TwitterServer {}, + flags = Map("foo.bar" -> "true"), + disableTestLogging = true + ) + + try { + val e = intercept[Exception] { + server.assertHealthy() + } + + val e2 = intercept[Exception] { //accessing the injector requires a started server + server.injector + } + + e.getMessage.contains("Error parsing flag \"foo.bar\": flag undefined") should be(true) + e.getMessage equals(e2.getMessage) + } finally { + server.close() + } + } + test("server#injector error") { val server = new EmbeddedTwitterServer( stage = Stage.PRODUCTION, diff --git a/inject/inject-thrift-client/src/main/java/BUILD b/inject/inject-thrift-client/src/main/java/BUILD index ab42cd3967..d69eb3ee08 100644 --- a/inject/inject-thrift-client/src/main/java/BUILD +++ b/inject/inject-thrift-client/src/main/java/BUILD @@ -10,3 +10,10 @@ java_library( "3rdparty/jvm/com/google/inject:guice", ], ) + +# TODO: Remove this and references to it, +# when a fix for https://github.com/pantsbuild/pants/issues/7200 has landed. +files( + name = "pants-workaround", + sources = rglobs("*.java"), +) diff --git a/inject/inject-thrift-client/src/main/scala/BUILD b/inject/inject-thrift-client/src/main/scala/BUILD index e0880ee409..f7d1270e22 100644 --- a/inject/inject-thrift-client/src/main/scala/BUILD +++ b/inject/inject-thrift-client/src/main/scala/BUILD @@ -25,6 +25,7 @@ scala_library( "finatra/inject/inject-app/src/main/scala", "finatra/inject/inject-core/src/main/scala", "finatra/inject/inject-slf4j/src/main/scala", + "finatra/inject/inject-thrift-client/src/main/java:pants-workaround", "finatra/inject/inject-thrift/src/main/scala", "finatra/inject/inject-utils/src/main/scala", "scrooge/scrooge-core/src/main/scala", diff --git a/inject/inject-thrift-client/src/main/scala/com/twitter/inject/thrift/filters/ThriftClientFilterChain.scala b/inject/inject-thrift-client/src/main/scala/com/twitter/inject/thrift/filters/ThriftClientFilterChain.scala index 00642e0c64..2f4e9a0352 100644 --- a/inject/inject-thrift-client/src/main/scala/com/twitter/inject/thrift/filters/ThriftClientFilterChain.scala +++ b/inject/inject-thrift-client/src/main/scala/com/twitter/inject/thrift/filters/ThriftClientFilterChain.scala @@ -66,7 +66,7 @@ import org.joda.time.Duration * @tparam Req Request type for this filter chain * @tparam Rep Response type for this filter chain * @see [[com.twitter.inject.thrift.filters.ThriftClientFilterBuilder]] - * @see [[com.twitter.finagle.thrift.ThriftServiceIface]] + * @see [[com.twitter.finagle.thrift.service.ThriftServicePerEndpoint]] */ @deprecated("Use ThriftMethodBuilderClientModule and ThriftMethodBuilder", "2018-01-12") class ThriftClientFilterChain[Req <: ThriftStruct, Rep]( @@ -90,7 +90,7 @@ class ThriftClientFilterChain[Req <: ThriftStruct, Rep]( private val scopedStatsReceiver = scopeStatsReceiver() - /** @see [[com.twitter.finagle.thrift.ThriftServiceIface#statsFilter]] */ + /** @see [[com.twitter.finagle.thrift.service.ThriftServicePerEndpoint#statsFilter]] */ // method invocations - incremented every time we call/invoke the method. /** Example scope: clnt/thrift/Adder/add1String/method/invocations */ private val invocationsCounter = scopedStatsReceiver.counter("invocations") diff --git a/inject/inject-thrift-client/src/main/scala/com/twitter/inject/thrift/internal/filters/LatencyFilter.scala b/inject/inject-thrift-client/src/main/scala/com/twitter/inject/thrift/internal/filters/LatencyFilter.scala index eef0aecb7e..328686bcff 100644 --- a/inject/inject-thrift-client/src/main/scala/com/twitter/inject/thrift/internal/filters/LatencyFilter.scala +++ b/inject/inject-thrift-client/src/main/scala/com/twitter/inject/thrift/internal/filters/LatencyFilter.scala @@ -1,6 +1,6 @@ package com.twitter.inject.thrift.internal.filters -import com.twitter.finagle.{WriteException, BackupRequestLost, Service, SimpleFilter} +import com.twitter.finagle.{FailureFlags, Service, SimpleFilter} import com.twitter.finagle.stats.StatsReceiver import com.twitter.util.{Throw, Try, Stopwatch, Future} import java.util.concurrent.TimeUnit @@ -27,14 +27,8 @@ private[thrift] class LatencyFilter[Req, Rep]( // Based on `c.t.finagle.service.StatsFilter#isBlackholeResponse` private def isBlackHoleResponse(rep: Try[Rep]): Boolean = rep match { - case Throw(BackupRequestLost) | Throw(WriteException(BackupRequestLost)) => + case Throw(f: FailureFlags[_]) if f.isFlagged(FailureFlags.Ignorable) => // We blackhole this request. It doesn't count for anything. - // After the Failure() patch, this should no longer need to - // be a special case. - // - // In theory, we should probably unwind the whole cause - // chain to look for a BackupRequestLost, but in practice it - // is wrapped only once. true case _ => false diff --git a/inject/inject-thrift-client/src/main/scala/com/twitter/inject/thrift/modules/FilteredThriftClientModule.scala b/inject/inject-thrift-client/src/main/scala/com/twitter/inject/thrift/modules/FilteredThriftClientModule.scala index bc8d3b66c4..5c66895f11 100644 --- a/inject/inject-thrift-client/src/main/scala/com/twitter/inject/thrift/modules/FilteredThriftClientModule.scala +++ b/inject/inject-thrift-client/src/main/scala/com/twitter/inject/thrift/modules/FilteredThriftClientModule.scala @@ -6,13 +6,8 @@ import com.google.inject.Provides import com.twitter.finagle._ import com.twitter.finagle.service.Retries.Budget import com.twitter.finagle.stats.StatsReceiver -import com.twitter.finagle.thrift.{ - ClientId, - MethodIfaceBuilder, - ServiceIfaceBuilder, - ThriftService, - ThriftServiceIface -} +import com.twitter.finagle.thrift.service.Filterable +import com.twitter.finagle.thrift.{ClientId, MethodIfaceBuilder, ServiceIfaceBuilder, ThriftService} import com.twitter.inject.annotations.Flag import com.twitter.inject.conversions.duration._ import com.twitter.inject.exceptions.PossiblyRetryable @@ -66,7 +61,7 @@ object FilteredThriftClientModule { @deprecated("Use the com.twitter.inject.thrift.modules.ThriftMethodBuilderClientModule", "2018-01-08") abstract class FilteredThriftClientModule[ FutureIface <: ThriftService: ClassTag, - ServiceIface <: ThriftServiceIface.Filterable[ServiceIface]: ClassTag + ServiceIface <: Filterable[ServiceIface]: ClassTag ]( implicit serviceBuilder: ServiceIfaceBuilder[ServiceIface], methodBuilder: MethodIfaceBuilder[ServiceIface, FutureIface] diff --git a/inject/inject-thrift-client/src/test/scala/com/twitter/inject/thrift/DoEverythingFilteredThriftClientModuleFeatureTest.scala b/inject/inject-thrift-client/src/test/scala/com/twitter/inject/thrift/DoEverythingFilteredThriftClientModuleFeatureTest.scala index 123c1444e1..aa22b133c2 100644 --- a/inject/inject-thrift-client/src/test/scala/com/twitter/inject/thrift/DoEverythingFilteredThriftClientModuleFeatureTest.scala +++ b/inject/inject-thrift-client/src/test/scala/com/twitter/inject/thrift/DoEverythingFilteredThriftClientModuleFeatureTest.scala @@ -44,7 +44,7 @@ class DoEverythingFilteredThriftClientModuleFeatureTest // per-method -- all the requests in this test were to the same method server.assertCounter("clnt/greeter-thrift-client/Greeter/hi/invocations", 1) - /* assert counters added by ThriftServiceIface#statsFilter */ + /* assert counters added by ThriftServicePerEndpoint#statsFilter */ server.assertCounter("clnt/greeter-thrift-client/Greeter/hi/requests", 4) server.assertCounter("clnt/greeter-thrift-client/Greeter/hi/success", 2) server.assertCounter("clnt/greeter-thrift-client/Greeter/hi/failures", 2) diff --git a/inject/inject-thrift-client/src/test/scala/com/twitter/inject/thrift/DoEverythingReqRepThriftMethodBuilderClientModuleFeatureTest.scala b/inject/inject-thrift-client/src/test/scala/com/twitter/inject/thrift/DoEverythingReqRepThriftMethodBuilderClientModuleFeatureTest.scala index 0f470cec94..a5fb556f1b 100644 --- a/inject/inject-thrift-client/src/test/scala/com/twitter/inject/thrift/DoEverythingReqRepThriftMethodBuilderClientModuleFeatureTest.scala +++ b/inject/inject-thrift-client/src/test/scala/com/twitter/inject/thrift/DoEverythingReqRepThriftMethodBuilderClientModuleFeatureTest.scala @@ -64,7 +64,7 @@ class DoEverythingReqRepThriftMethodBuilderClientModuleFeatureTest } // per-method -- all the requests in this test were to the same method - /* assert counters added by ThriftServiceIface#statsFilter */ + /* assert counters added by ThriftServicePerEndpoint#statsFilter */ server.assertCounter("clnt/greeter-thrift-client/Greeter/hi/requests", 3) server.assertCounter("clnt/greeter-thrift-client/Greeter/hi/success", 1) server.assertCounter("clnt/greeter-thrift-client/Greeter/hi/failures", 2) @@ -83,7 +83,7 @@ class DoEverythingReqRepThriftMethodBuilderClientModuleFeatureTest assert(!response.headers.empty) // per-method -- all the requests in this test were to the same method - /* assert counters added by ThriftServiceIface#statsFilter */ + /* assert counters added by ThriftServicePerEndpoint#statsFilter */ server.assertCounter("clnt/greeter-thrift-client/Greeter/hello/requests", 3) server.assertCounter("clnt/greeter-thrift-client/Greeter/hello/success", 1) server.assertCounter("clnt/greeter-thrift-client/Greeter/hello/failures", 2) @@ -103,7 +103,7 @@ class DoEverythingReqRepThriftMethodBuilderClientModuleFeatureTest ) // per-method -- all the requests in this test were to the same method - /* assert counters added by ThriftServiceIface#statsFilter */ + /* assert counters added by ThriftServicePerEndpoint#statsFilter */ server.assertCounter("clnt/greeter-thrift-client/Greeter/bye/requests", 3) server.assertCounter("clnt/greeter-thrift-client/Greeter/bye/success", 1) server.assertCounter("clnt/greeter-thrift-client/Greeter/bye/failures", 2) diff --git a/inject/inject-thrift-client/src/test/scala/com/twitter/inject/thrift/DoEverythingThriftMethodBuilderClientModuleFeatureTest.scala b/inject/inject-thrift-client/src/test/scala/com/twitter/inject/thrift/DoEverythingThriftMethodBuilderClientModuleFeatureTest.scala index e861cf46b7..6fb69da04c 100644 --- a/inject/inject-thrift-client/src/test/scala/com/twitter/inject/thrift/DoEverythingThriftMethodBuilderClientModuleFeatureTest.scala +++ b/inject/inject-thrift-client/src/test/scala/com/twitter/inject/thrift/DoEverythingThriftMethodBuilderClientModuleFeatureTest.scala @@ -82,7 +82,7 @@ class DoEverythingThriftMethodBuilderClientModuleFeatureTest extends FeatureTest server.assertStat("route/hi/GET/response_size", Seq(5)) // per-method -- all the requests in this test were to the same method - /* assert counters added by ThriftServiceIface#statsFilter */ + /* assert counters added by ThriftServicePerEndpoint#statsFilter */ server.assertCounter("clnt/greeter-thrift-client/Greeter/hi/requests", 3) server.assertCounter("clnt/greeter-thrift-client/Greeter/hi/success", 1) server.assertCounter("clnt/greeter-thrift-client/Greeter/hi/failures", 2) @@ -100,7 +100,7 @@ class DoEverythingThriftMethodBuilderClientModuleFeatureTest extends FeatureTest server.assertStat("route/hello/GET/response_size", Seq(9)) // per-method -- all the requests in this test were to the same method - /* assert counters added by ThriftServiceIface#statsFilter */ + /* assert counters added by ThriftServicePerEndpoint#statsFilter */ server.assertCounter("clnt/greeter-thrift-client/Greeter/hello/requests", 3) server.assertCounter("clnt/greeter-thrift-client/Greeter/hello/success", 1) server.assertCounter("clnt/greeter-thrift-client/Greeter/hello/failures", 2) @@ -122,7 +122,7 @@ class DoEverythingThriftMethodBuilderClientModuleFeatureTest extends FeatureTest server.assertStat("route/bye/GET/response_size", Seq(20)) // per-method -- all the requests in this test were to the same method - /* assert counters added by ThriftServiceIface#statsFilter */ + /* assert counters added by ThriftServicePerEndpoint#statsFilter */ server.assertCounter("clnt/greeter-thrift-client/Greeter/bye/requests", 3) server.assertCounter("clnt/greeter-thrift-client/Greeter/bye/success", 1) server.assertCounter("clnt/greeter-thrift-client/Greeter/bye/failures", 2) @@ -155,7 +155,7 @@ class DoEverythingThriftMethodBuilderClientModuleFeatureTest extends FeatureTest server.assertStat("route/echo/GET/response_size", Seq(9)) // per-method -- all the requests in this test were to the same method - /* assert counters added by ThriftServiceIface#statsFilter */ + /* assert counters added by ThriftServicePerEndpoint#statsFilter */ server.assertCounter("clnt/echo-thrift-client/EchoService/echo/requests", 1) server.assertCounter("clnt/echo-thrift-client/EchoService/echo/success", 1) server.assertCounter("clnt/echo-thrift-client/EchoService/echo/failures", 0) diff --git a/inject/inject-thrift-client/src/test/scala/com/twitter/inject/thrift/InheritanceThriftMethodBuilderClientModuleFeatureTest.scala b/inject/inject-thrift-client/src/test/scala/com/twitter/inject/thrift/InheritanceThriftMethodBuilderClientModuleFeatureTest.scala index 20e699dd61..c007501467 100644 --- a/inject/inject-thrift-client/src/test/scala/com/twitter/inject/thrift/InheritanceThriftMethodBuilderClientModuleFeatureTest.scala +++ b/inject/inject-thrift-client/src/test/scala/com/twitter/inject/thrift/InheritanceThriftMethodBuilderClientModuleFeatureTest.scala @@ -47,7 +47,7 @@ class InheritanceThriftMethodBuilderClientModuleFeatureTest server.httpGet(path = "/echo?msg=Hello!", andExpect = Ok, withBody = "Hello!") // per-method -- all the requests in this test were to the same method - /* assert counters added by ThriftServiceIface#statsFilter */ + /* assert counters added by ThriftServicePerEndpoint#statsFilter */ server.assertCounter("clnt/serviceB-thrift-client/ServiceA/echo/requests", 1) server.assertCounter("clnt/serviceB-thrift-client/ServiceA/echo/success", 1) server.assertCounter("clnt/serviceB-thrift-client/ServiceA/echo/failures", 0) @@ -63,7 +63,7 @@ class InheritanceThriftMethodBuilderClientModuleFeatureTest server.httpGet(path = "/ping", andExpect = Ok, withBody = "pong") // per-method -- all the requests in this test were to the same method - /* assert counters added by ThriftServiceIface#statsFilter */ + /* assert counters added by ThriftServicePerEndpoint#statsFilter */ server.assertCounter("clnt/serviceB-thrift-client/ServiceB/ping/requests", 1) server.assertCounter("clnt/serviceB-thrift-client/ServiceB/ping/success", 1) server.assertCounter("clnt/serviceB-thrift-client/ServiceB/ping/failures", 0) diff --git a/inject/inject-thrift/src/main/scala/com/twitter/inject/exceptions/PossiblyRetryable.scala b/inject/inject-thrift/src/main/scala/com/twitter/inject/exceptions/PossiblyRetryable.scala index 7d9643066c..54faaedee8 100644 --- a/inject/inject-thrift/src/main/scala/com/twitter/inject/exceptions/PossiblyRetryable.scala +++ b/inject/inject-thrift/src/main/scala/com/twitter/inject/exceptions/PossiblyRetryable.scala @@ -1,7 +1,7 @@ package com.twitter.inject.exceptions import com.twitter.finagle.mux.ClientDiscardedRequestException -import com.twitter.finagle.{BackupRequestLost, CancelledConnectionException, CancelledRequestException, Failure, FailureFlags, service => ctfs} +import com.twitter.finagle.{CancelledConnectionException, CancelledRequestException, Failure, FailureFlags, service => ctfs} import com.twitter.finagle.service.{ReqRep, ResponseClass} import com.twitter.util.{Return, Throw, Try} import scala.util.control.NonFatal @@ -64,19 +64,18 @@ object PossiblyRetryable { } private[inject] def isCancellation(t: Throwable): Boolean = t match { - case BackupRequestLost => true case _: CancelledRequestException => true case _: CancelledConnectionException => true case _: ClientDiscardedRequestException => true - case f: Failure if f.isFlagged(FailureFlags.Interrupted) => true + case f: FailureFlags[_] if f.isFlagged(FailureFlags.Ignorable) => true + case f: FailureFlags[_] if f.isFlagged(FailureFlags.Interrupted) => true case f: Failure if f.cause.isDefined => isCancellation(f.cause.get) case _ => false } private[inject] def isNonRetryable(t: Throwable) : Boolean = t match { - case BackupRequestLost => true case _: NonRetryableException => true - case f: Failure if f.isFlagged(FailureFlags.Ignorable) => true + case f: FailureFlags[_] if f.isFlagged(FailureFlags.Ignorable) => true case f: Failure if f.cause.isDefined => isNonRetryable(f.cause.get) case _ => false } diff --git a/inject/inject-thrift/src/test/scala/com/twitter/inject/tests/exceptions/PossiblyRetryableTest.scala b/inject/inject-thrift/src/test/scala/com/twitter/inject/tests/exceptions/PossiblyRetryableTest.scala index d5c80846dc..d28e1a405e 100644 --- a/inject/inject-thrift/src/test/scala/com/twitter/inject/tests/exceptions/PossiblyRetryableTest.scala +++ b/inject/inject-thrift/src/test/scala/com/twitter/inject/tests/exceptions/PossiblyRetryableTest.scala @@ -2,7 +2,6 @@ package com.twitter.inject.tests.exceptions import com.twitter.finagle.mux.ClientDiscardedRequestException import com.twitter.finagle.{ - BackupRequestLost, CancelledConnectionException, CancelledRequestException, Failure, @@ -21,7 +20,6 @@ class PossiblyRetryableTest extends Test { with com.twitter.inject.exceptions.NonRetryableException test("test isCancellation") { - assertIsCancellation(BackupRequestLost) assertIsCancellation(new CancelledRequestException) assertIsCancellation(new CancelledConnectionException(new Exception("cause"))) assertIsCancellation(new ClientDiscardedRequestException("cause")) @@ -30,7 +28,6 @@ class PossiblyRetryableTest extends Test { } test("test isNonRetryable") { - assertIsNonRetryable(BackupRequestLost) assertIsNonRetryable(Failure("int", FailureFlags.Ignorable)) } diff --git a/jackson/src/main/java/BUILD b/jackson/src/main/java/BUILD index 7f74ff7855..d283a8ec7e 100644 --- a/jackson/src/main/java/BUILD +++ b/jackson/src/main/java/BUILD @@ -10,3 +10,10 @@ java_library( "finatra/jackson/src/main/scala", ], ) + +# TODO: Remove this and references to it, +# when a fix for https://github.com/pantsbuild/pants/issues/7200 has landed. +files( + name = "pants-workaround", + sources = rglobs("*.java"), +) diff --git a/jackson/src/main/java/com/twitter/finatra/json/internal/caseclass/validation/validators/PatternInternal.java b/jackson/src/main/java/com/twitter/finatra/json/internal/caseclass/validation/validators/PatternInternal.java new file mode 100644 index 0000000000..3451e88bda --- /dev/null +++ b/jackson/src/main/java/com/twitter/finatra/json/internal/caseclass/validation/validators/PatternInternal.java @@ -0,0 +1,20 @@ +package com.twitter.finatra.json.internal.caseclass.validation.validators; + +import com.twitter.finatra.validation.Validation; + +import java.lang.annotation.Retention; +import java.lang.annotation.Target; + +import static java.lang.annotation.ElementType.PARAMETER; +import static java.lang.annotation.RetentionPolicy.RUNTIME; + +@Target({PARAMETER}) +@Retention(RUNTIME) +@Validation(validatedBy = PatternValidator.class) +public @interface PatternInternal { + + /** + * @return the regular expression to match + */ + String regexp(); +} diff --git a/jackson/src/main/resources/com/twitter/finatra/json/validation.properties b/jackson/src/main/resources/com/twitter/finatra/json/validation.properties index 1b58cfc278..e935dc88f4 100644 --- a/jackson/src/main/resources/com/twitter/finatra/json/validation.properties +++ b/jackson/src/main/resources/com/twitter/finatra/json/validation.properties @@ -9,3 +9,4 @@ com.twitter.finatra.json.internal.caseclass.validation.validators.RangeInternal com.twitter.finatra.json.internal.caseclass.validation.validators.SizeInternal = size [%s] is not between %s and %s com.twitter.finatra.json.internal.caseclass.validation.validators.TimeGranularityInternal = [%s] is not %s granularity com.twitter.finatra.json.internal.caseclass.validation.validators.UUIDInternal = [%s] is not a valid UUID +com.twitter.finatra.json.internal.caseclass.validation.validators.PatternInternal = [%s] does not match regex %s diff --git a/jackson/src/main/resources/com/twitter/json/validation.properties b/jackson/src/main/resources/com/twitter/json/validation.properties deleted file mode 100644 index 1b58cfc278..0000000000 --- a/jackson/src/main/resources/com/twitter/json/validation.properties +++ /dev/null @@ -1,11 +0,0 @@ -com.twitter.finatra.json.internal.caseclass.validation.validators.CountryCodeInternal = [%s] is not a valid country code -com.twitter.finatra.json.internal.caseclass.validation.validators.FutureTimeInternal = [%s] is not in the future -com.twitter.finatra.json.internal.caseclass.validation.validators.MinInternal = [%s] is not greater than or equal to %s -com.twitter.finatra.json.internal.caseclass.validation.validators.MaxInternal = [%s] is not less than or equal to %s -com.twitter.finatra.json.internal.caseclass.validation.validators.NotEmptyInternal = cannot be empty -com.twitter.finatra.json.internal.caseclass.validation.validators.OneOfInternal = [%s] is not one of [%s] -com.twitter.finatra.json.internal.caseclass.validation.validators.PastTimeInternal = [%s] is not in the past -com.twitter.finatra.json.internal.caseclass.validation.validators.RangeInternal = [%s] is not between %s and %s -com.twitter.finatra.json.internal.caseclass.validation.validators.SizeInternal = size [%s] is not between %s and %s -com.twitter.finatra.json.internal.caseclass.validation.validators.TimeGranularityInternal = [%s] is not %s granularity -com.twitter.finatra.json.internal.caseclass.validation.validators.UUIDInternal = [%s] is not a valid UUID diff --git a/jackson/src/main/scala/BUILD b/jackson/src/main/scala/BUILD index c0e5e9a8ac..9b31a133cd 100644 --- a/jackson/src/main/scala/BUILD +++ b/jackson/src/main/scala/BUILD @@ -33,6 +33,7 @@ scala_library( "finatra/inject/inject-core/src/main/scala", "finatra/inject/inject-slf4j/src/main/scala", "finatra/inject/inject-utils/src/main/scala", + "finatra/jackson/src/main/java:pants-workaround", "finatra/jackson/src/main/resources", "util/util-app/src/main/scala", "util/util-core/src/main/scala", diff --git a/jackson/src/main/scala/com/twitter/finatra/json/internal/caseclass/exceptions/JsonInjectException.scala b/jackson/src/main/scala/com/twitter/finatra/json/internal/caseclass/exceptions/JsonInjectException.scala index d15da84e87..862f6363c9 100644 --- a/jackson/src/main/scala/com/twitter/finatra/json/internal/caseclass/exceptions/JsonInjectException.scala +++ b/jackson/src/main/scala/com/twitter/finatra/json/internal/caseclass/exceptions/JsonInjectException.scala @@ -1,7 +1,6 @@ package com.twitter.finatra.json.internal.caseclass.exceptions import com.google.inject.Key -import scala.language.existentials import scala.util.control.NoStackTrace case class JsonInjectException( diff --git a/jackson/src/main/scala/com/twitter/finatra/json/internal/caseclass/exceptions/JsonInjectionNotSupportedException.scala b/jackson/src/main/scala/com/twitter/finatra/json/internal/caseclass/exceptions/JsonInjectionNotSupportedException.scala index 30bc90fe8d..17d61c0574 100644 --- a/jackson/src/main/scala/com/twitter/finatra/json/internal/caseclass/exceptions/JsonInjectionNotSupportedException.scala +++ b/jackson/src/main/scala/com/twitter/finatra/json/internal/caseclass/exceptions/JsonInjectionNotSupportedException.scala @@ -1,7 +1,5 @@ package com.twitter.finatra.json.internal.caseclass.exceptions -import scala.language.existentials - case class JsonInjectionNotSupportedException(parentClass: Class[_], fieldName: String) extends Exception( "Injection of fields (e.g. @Inject, @QueryParam, @Header) not " + diff --git a/jackson/src/main/scala/com/twitter/finatra/json/internal/caseclass/jackson/CaseClassField.scala b/jackson/src/main/scala/com/twitter/finatra/json/internal/caseclass/jackson/CaseClassField.scala index d9c13bbbad..526f753873 100644 --- a/jackson/src/main/scala/com/twitter/finatra/json/internal/caseclass/jackson/CaseClassField.scala +++ b/jackson/src/main/scala/com/twitter/finatra/json/internal/caseclass/jackson/CaseClassField.scala @@ -7,13 +7,15 @@ import com.fasterxml.jackson.databind.`type`.TypeFactory import com.fasterxml.jackson.databind.annotation.JsonDeserialize import com.fasterxml.jackson.databind.node.TreeTraversingParser import com.fasterxml.jackson.databind.util.ClassUtil -import com.twitter.finatra.json.internal.caseclass.exceptions.CaseClassValidationException.PropertyPath import com.twitter.finatra.json.internal.caseclass.exceptions.{ CaseClassValidationException, FinatraJsonMappingException } -import com.twitter.finatra.json.internal.caseclass.reflection.CaseClassSigParser -import com.twitter.finatra.json.internal.caseclass.reflection.DefaultMethodUtils.defaultFunction +import com.twitter.finatra.json.internal.caseclass.reflection.{ + CaseClassSigParser, + ConstructorParam, + DefaultMethodUtils +} import com.twitter.finatra.json.internal.caseclass.utils.AnnotationUtils._ import com.twitter.finatra.json.internal.caseclass.utils.FieldInjection import com.twitter.finatra.request.{FormParam, Header, QueryParam} @@ -23,7 +25,6 @@ import com.twitter.inject.Logging import com.twitter.inject.conversions.string._ import java.lang.annotation.Annotation import scala.annotation.tailrec -import scala.language.existentials import scala.reflect.NameTransformer private[finatra] object CaseClassField { @@ -33,56 +34,122 @@ private[finatra] object CaseClassField { namingStrategy: PropertyNamingStrategy, typeFactory: TypeFactory ): Seq[CaseClassField] = { - val allAnnotations = constructorAnnotations(clazz) - val constructorParams = CaseClassSigParser.parseConstructorParams(clazz) + val constructorParams: Seq[ConstructorParam] = CaseClassSigParser.parseConstructorParams(clazz) assert( - allAnnotations.size == constructorParams.size, + clazz.getConstructors.head.getParameterCount == constructorParams.size, "Non-static inner 'case classes' not supported" ) + // field name to list of parsed annotations + val annotationsMap: Map[String, Seq[Annotation]] = findAnnotations(clazz, constructorParams) + val companionObject = Class.forName(clazz.getName + "$").getField("MODULE$").get(null) val companionObjectClass = companionObject.getClass - for { - (constructorParam, idx) <- constructorParams.zipWithIndex - annotations = allAnnotations(idx) - name = jsonNameForField(annotations, namingStrategy, constructorParam.name) - deserializer = deserializerOrNone(annotations) - } yield { + for ((constructorParam, idx) <- constructorParams.zipWithIndex) yield { + val fieldAnnotations: Seq[Annotation] = annotationsMap.getOrElse(constructorParam.name, Nil) + val name = jsonNameForField(fieldAnnotations, namingStrategy, constructorParam.name) + val deserializer = deserializerOrNone(fieldAnnotations) + CaseClassField( name = name, javaType = JacksonTypes.javaType(typeFactory, constructorParam.scalaType), parentClass = clazz, - defaultFuncOpt = defaultFunction(companionObjectClass, companionObject, idx), - annotations = annotations, + defaultFuncOpt = + DefaultMethodUtils.defaultFunction(companionObjectClass, companionObject, idx), + annotations = fieldAnnotations, deserializer = deserializer ) } } - private[finatra] def constructorAnnotations(clazz: Class[_]): Seq[Array[Annotation]] = { - clazz.getConstructors.head.getParameterAnnotations.toSeq + /** Finds the sequence of Annotations per field in the clazz, keyed by field name */ + private[finatra] def findAnnotations( + clazz: Class[_], + constructorParams: Seq[ConstructorParam] + ): Map[String, Seq[Annotation]] = { + // for case classes, the annotations are only visible on the constructor. + val clazzConstructorAnnotations: Array[Array[Annotation]] = + clazz.getConstructors.head.getParameterAnnotations + + // find case class field annotations + val clazzAnnotations: Map[String, Seq[Annotation]] = (for { + (field, index) <- constructorParams.zipWithIndex + fieldAnnotations = clazzConstructorAnnotations(index) + } yield { + field.name -> fieldAnnotations.toSeq + }).toMap + + // find inherited annotations + val inheritedAnnotations: Map[String, Seq[Annotation]] = + findDeclaredMethodAnnotations(clazz, Map.empty[String, Seq[Annotation]]) + + // Merge the two maps: if the same annotation for a given field occurs in both lists, we keep + // the clazz annotation to in effect "override" what was specified by inheritance. That is, it + // is not expected that annotations are ever additive (in the sense that you can configure a + // single field through multiple declarations of the same annotation) but rather either-or. + clazzAnnotations.map { + case (field: String, annotations: Seq[Annotation]) => + val inherited: Seq[Annotation] = + inheritedAnnotations.getOrElse(field, Nil) + // want to prefer what is coming in from clazz annotations over inherited + field -> mergeAnnotationLists(annotations, inherited) + } } - private def jsonNameForField( + private[this] def findDeclaredMethodAnnotations( + clazz: Class[_], + found: Map[String, Seq[Annotation]] + ): Map[String, Seq[Annotation]] = { + // clazz declared method annotations + val interfaceDeclaredAnnotations: Map[String, Seq[Annotation]] = + clazz.getDeclaredMethods + .map { method => + method.getName -> method.getDeclaredAnnotations.toSeq + }.toMap.map { + case (key, values) => + key -> mergeAnnotationLists(values, found.getOrElse(key, Seq.empty[Annotation])) + } + + // interface declared method annotations + clazz.getInterfaces.foldLeft(interfaceDeclaredAnnotations) { + (acc: Map[String, Seq[Annotation]], interface: Class[_]) => + acc.map { + case (key, values) => + key -> mergeAnnotationLists( + values, + findDeclaredMethodAnnotations(interface, acc).getOrElse(key, Seq.empty[Annotation])) + } + } + } + + /** Prefer values in A over B */ + private[this] def mergeAnnotationLists( + a: Seq[Annotation], + b: Seq[Annotation] + ): Seq[Annotation] = { + a ++ b.filterNot(bAnnotation => a.exists(_.annotationType() == bAnnotation.annotationType())) + } + + private[this] def jsonNameForField( annotations: Seq[Annotation], namingStrategy: PropertyNamingStrategy, name: String ): String = { findAnnotation[JsonProperty](annotations) match { - case Some(jsonProperty) if jsonProperty.value.nonEmpty => jsonProperty.value + case Some(jsonProperty) if jsonProperty.value.nonEmpty => + jsonProperty.value case _ => - val decodedName = NameTransformer.decode(name) //decode unicode escaped field names - namingStrategy.nameForField( //apply json naming strategy (e.g. snake_case) + val decodedName = NameTransformer.decode(name) // decode unicode escaped field names + namingStrategy.nameForField( // apply json naming strategy (e.g. snake_case) /* config = */ null, /* field = */ null, - /* defaultName = */ decodedName - ) + /* defaultName = */ decodedName) } } - private def deserializerOrNone( - annotations: Array[Annotation] + private[this] def deserializerOrNone( + annotations: Seq[Annotation] ): Option[JsonDeserializer[Object]] = { for { jsonDeserializer <- findAnnotation[JsonDeserialize](annotations) @@ -98,8 +165,8 @@ private[finatra] case class CaseClassField( parentClass: Class[_], defaultFuncOpt: Option[() => Object], annotations: Seq[Annotation], - deserializer: Option[JsonDeserializer[Object]] -) extends Logging { + deserializer: Option[JsonDeserializer[Object]]) + extends Logging { private val isOption = javaType.getRawClass == classOf[Option[_]] private val isString = javaType.getRawClass == classOf[String] @@ -107,7 +174,7 @@ private[finatra] case class CaseClassField( private val fieldInjection = new FieldInjection(name, javaType, parentClass, annotations) private lazy val firstTypeParam = javaType.containedType(0) private lazy val requiredFieldException = CaseClassValidationException( - PropertyPath.leaf(attributeName), + CaseClassValidationException.PropertyPath.leaf(attributeName), Invalid(s"$attributeType is required", ErrorCode.RequiredFieldMissing) ) @@ -141,7 +208,9 @@ private[finatra] case class CaseClassField( ): Object = { if (fieldInjection.isInjectable) fieldInjection - .inject(context, codec).orElse(defaultValue).getOrElse(throwRequiredFieldException()) + .inject(context, codec) + .orElse(defaultValue) + .getOrElse(throwRequiredFieldException()) else { val fieldJsonNode = objectJsonNode.get(name) if (fieldJsonNode != null && !fieldJsonNode.isNull) @@ -213,10 +282,7 @@ private[finatra] case class CaseClassField( private case class AttributeInfo(`type`: String, fieldName: String) @tailrec - private def findAttributeInfo( - fieldName: String, - annotations: Seq[Annotation] - ): AttributeInfo = { + private def findAttributeInfo(fieldName: String, annotations: Seq[Annotation]): AttributeInfo = { if (annotations.isEmpty) { AttributeInfo("field", fieldName) } else { diff --git a/jackson/src/main/scala/com/twitter/finatra/json/internal/caseclass/utils/AnnotationUtils.scala b/jackson/src/main/scala/com/twitter/finatra/json/internal/caseclass/utils/AnnotationUtils.scala index a0a2e0baca..01509dbf8e 100644 --- a/jackson/src/main/scala/com/twitter/finatra/json/internal/caseclass/utils/AnnotationUtils.scala +++ b/jackson/src/main/scala/com/twitter/finatra/json/internal/caseclass/utils/AnnotationUtils.scala @@ -7,8 +7,8 @@ private[finatra] object AnnotationUtils { def filterIfAnnotationPresent[A <: Annotation: Manifest]( annotations: Seq[Annotation] ): Seq[Annotation] = { - annotations filter { annot => - isAnnotationPresent[A](annot) + annotations.filter { annotation => + isAnnotationPresent[A](annotation) } } @@ -16,7 +16,7 @@ private[finatra] object AnnotationUtils { filterSet: Set[Class[_ <: Annotation]], annotations: Seq[Annotation] ): Seq[Annotation] = { - annotations filter { annotation => + annotations.filter { annotation => filterSet.contains(annotation.annotationType) } } @@ -25,13 +25,13 @@ private[finatra] object AnnotationUtils { target: Class[_ <: Annotation], annotations: Seq[Annotation] ): Option[Annotation] = { - annotations find { annotation => + annotations.find { annotation => annotation.annotationType() == target } } def findAnnotation[A <: Annotation: Manifest](annotations: Seq[Annotation]): Option[A] = { - annotations collectFirst { + annotations.collectFirst { case annotation if annotationEquals[A](annotation) => annotation.asInstanceOf[A] } diff --git a/jackson/src/main/scala/com/twitter/finatra/json/internal/caseclass/validation/validators/PatternValidator.scala b/jackson/src/main/scala/com/twitter/finatra/json/internal/caseclass/validation/validators/PatternValidator.scala new file mode 100644 index 0000000000..2d245cba1b --- /dev/null +++ b/jackson/src/main/scala/com/twitter/finatra/json/internal/caseclass/validation/validators/PatternValidator.scala @@ -0,0 +1,92 @@ +package com.twitter.finatra.json.internal.caseclass.validation.validators + +import com.twitter.finatra.json.internal.caseclass.validation.validators.PatternValidator._ +import com.twitter.finatra.validation.ValidationResult.{Invalid, Valid} +import com.twitter.finatra.validation._ +import com.twitter.util.{Return, Throw, Try} +import scala.util.matching.Regex + +private[finatra] object PatternValidator { + def errorMessage(resolver: ValidationMessageResolver, value: Any, regex: String): String = { + resolver.resolve(classOf[Pattern], value, regex) + } +} + +/** + * Validates whether given [[CharSequence]] value matches with the specified regular expression + * + * @example {{{ + * case class ExampleRequest(@Pattern(regexp= "exampleRegex") exampleValue : String) + * }}} + */ +private[finatra] class PatternValidator( + validationMessageResolver: ValidationMessageResolver, + annotation: Pattern) + extends Validator[Pattern, Any](validationMessageResolver, annotation) { + + private val regexp: String = annotation.regexp() + private val regex: Try[Regex] = Try(regexp.r) + + /* Public */ + + override def isValid(value: Any): ValidationResult = { + val validateRegexResult = validateRegex + if (validateRegexResult.isValid) { + value match { + case arrayValue: Array[_] => + validationResult(arrayValue) + case traversableValue: Traversable[_] => + validationResult(traversableValue) + case stringValue: String => + validationResult(stringValue) + case _ => + throw new IllegalArgumentException( + s"Class [${value.getClass}}] is not supported by ${this.getClass}") + } + } else validateRegexResult + } + + /* Private */ + + private def validationResult(value: Traversable[_]): ValidationResult = { + ValidationResult.validate( + value.forall(x => validateValue(x.toString)), + errorMessage(validationMessageResolver, value, regexp), + errorCode(value, regexp) + ) + } + + private def validationResult(value: String): ValidationResult = { + ValidationResult.validate( + validateValue(value), + errorMessage(validationMessageResolver, value, regexp), + errorCode(value, regexp) + ) + } + + // validate the value after validate the regex + private def validateValue(value: String): Boolean = { + regex.get().findFirstIn(value) match { + case None => false + case _ => true + } + } + + private def validateRegex: ValidationResult = + regex match { + case Return(_) => Valid + case Throw(ex) => Invalid(ex.getClass.getName, errorCode(ex, regexp)) + } + + private def errorCode(t: Throwable, regex: String) = { + ErrorCode.PatternSyntaxError(t.getMessage, regex) + } + + private def errorCode(value: String, regex: String) = { + ErrorCode.PatternNotMatched(value, regex) + } + + private def errorCode(value: Traversable[_], regex: String) = { + ErrorCode.PatternNotMatched(value mkString ",", regex) + } +} diff --git a/jackson/src/main/scala/com/twitter/finatra/json/internal/streaming/AsyncJsonParser.scala b/jackson/src/main/scala/com/twitter/finatra/json/internal/streaming/AsyncJsonParser.scala new file mode 100644 index 0000000000..bc89653d36 --- /dev/null +++ b/jackson/src/main/scala/com/twitter/finatra/json/internal/streaming/AsyncJsonParser.scala @@ -0,0 +1,128 @@ +package com.twitter.finatra.json.internal.streaming + +import com.fasterxml.jackson.core.async.ByteArrayFeeder +import com.fasterxml.jackson.core.json.async.NonBlockingJsonParser +import com.fasterxml.jackson.core.{JsonFactory, JsonParser, JsonToken} +import com.twitter.io.Buf +import java.nio.ByteBuffer +import scala.collection.mutable.ListBuffer + +/** + * Integrate Jackson JSON Parser to parse JSON using non-blocking input source. + * + * Jackson JsonParser would validate the input JSON and throw + * [[com.fasterxml.jackson.core.JsonParseException]] if unexpected token occurs. + * Finatra AsyncJsonParser would decide where to split up the input Buf, return whole + * serialized JSON objects and keep the unfinished bytes in the ByteBuffer. + */ +private[finatra] class AsyncJsonParser { + private[this] val parser: JsonParser = new JsonFactory().createNonBlockingByteArrayParser() + private[this] val feeder: ByteArrayFeeder = + parser.asInstanceOf[NonBlockingJsonParser].getNonBlockingInputFeeder() + + // All mutable state is synchronized on `this` + + // we can only parse the whole object/array/primitive type, + // and keep unfinished bytes in this buffer. + private[this] var remaining = ByteBuffer.allocate(0) + // keep track the object's starting position in remaining byteBuffer + private[this] var position: Int = 0 + // offset that records the length of parsed bytes in parser, accumulated every chunk + private[this] var offset: Int = 0 + // offset of a sliced buf, this offset should not be accumulated + private[this] var slicedBufOffset = 0 + private[this] var depth: Int = 0 + + /** + * Parse the Buf to slice it into a List of objects/arrays/primitive types, + * keep the unparsed part and wait for the next feed + */ + private[finatra] def feedAndParse(buf: Buf): Seq[Buf] = synchronized { + assertState() + + buf match { + case Buf.ByteArray.Owned((bytes, begin, end)) => + feeder.feedInput(bytes, begin, end) + slicedBufOffset = begin + case Buf.ByteArray.Shared(bytes) => + feeder.feedInput(bytes, 0, bytes.length) + case b => + val bytes = new Array[Byte](b.length) + b.write(bytes, 0) + feeder.feedInput(bytes, 0, bytes.length) + } + + remaining = ByteBufferUtils.append(remaining, buf, position) + val result: ListBuffer[Buf] = ListBuffer.empty + + while (parser.nextToken() != JsonToken.NOT_AVAILABLE) { + updateOpenBrace() + if (startInitialArray) { + // exclude the initial `[` + position = parser.getCurrentLocation.getByteOffset.toInt - slicedBufOffset - offset + } else if (startObjectArray) { + // include the starting token of the object or array + position = parser.getCurrentLocation.getByteOffset.toInt - slicedBufOffset - offset - 1 + } else if (endObjectArray) { + result += getSlicedBuf() + } else if (endInitialArray) { + depth = -1 + } else if (depth == 1) { + result += getSlicedBuf() + } else { + // fall through expected; a valid JsonToken, such as: FIELD_NAME, VALUE_NUMBER_INT, etc. + } + } + + result + } + + private def getSlicedBuf(): Buf = { + + remaining.position(position) + val newPosition = parser.getCurrentLocation.getByteOffset.toInt - slicedBufOffset - offset + val buf = remaining.slice() + buf.limit(newPosition - position) + + remaining.position(newPosition) + remaining = remaining.slice() + + offset = offset + newPosition + position = 1 + + Buf.ByteBuffer.Shared(buf) + } + + private def startInitialArray: Boolean = + parser.currentToken() == JsonToken.START_ARRAY && depth == 1 + + private def startObjectArray: Boolean = + depth == 2 && (parser.currentToken() == JsonToken.START_ARRAY || + parser.currentToken() == JsonToken.START_OBJECT) + + private def endObjectArray: Boolean = + depth == 1 && (parser.currentToken() == JsonToken.END_ARRAY || + parser.currentToken() == JsonToken.END_OBJECT) + + private def updateOpenBrace(): Unit = { + if (parser.currentToken() == JsonToken.START_ARRAY || + parser.currentToken() == JsonToken.START_OBJECT) + depth += 1 + else if (parser.currentToken() == JsonToken.END_ARRAY || + parser.currentToken() == JsonToken.END_OBJECT) + depth -= 1 + } + + private def endInitialArray: Boolean = + depth == 0 && parser.currentToken() == JsonToken.END_ARRAY + + private def assertState(): Unit = { + if (depth == -1) { + throw new IllegalStateException("End of the JSON object (`]`) already found") + } + } + + // expose for tests + private[json] def copiedByteBuffer: ByteBuffer = synchronized { remaining.duplicate() } + private[json] def getParsingDepth: Int = synchronized { depth } +} diff --git a/jackson/src/main/scala/com/twitter/finatra/json/internal/streaming/JsonArrayChunker.scala b/jackson/src/main/scala/com/twitter/finatra/json/internal/streaming/JsonArrayChunker.scala deleted file mode 100644 index c02fc5bd79..0000000000 --- a/jackson/src/main/scala/com/twitter/finatra/json/internal/streaming/JsonArrayChunker.scala +++ /dev/null @@ -1,107 +0,0 @@ -package com.twitter.finatra.json.internal.streaming - -import com.twitter.finatra.json.internal.streaming.ParsingState._ -import com.twitter.inject.Logging -import com.twitter.inject.conversions.buf._ -import com.twitter.io.Buf -import java.nio.ByteBuffer -import scala.collection.mutable.ArrayBuffer - -// General logic copied from: -// https://github.com/netty/netty/blob/master/codec/src/main/java/io/netty/handler/codec/json/JsonObjectDecoder.java -private[finatra] class JsonArrayChunker extends Logging { - - private[finatra] var parsingState: ParsingState = Normal - private[finatra] var done = false - private[finatra] var openBraces = 0 - private[finatra] var position = 0 - private var byteBuffer = ByteBuffer.allocate(0) - - /* Public */ - - def decode(inputBuf: Buf): Seq[Buf] = { - assertDecode(inputBuf) - byteBuffer = ByteBufferUtils.append(byteBuffer, inputBuf, position) - - val result = ArrayBuffer[Buf]() - - while (byteBuffer.hasRemaining) { - ByteBufferUtils.debugBuffer(byteBuffer) - val currByte = byteBuffer.get - position = byteBuffer.position - - if (!arrayFound && currByte == '[' && openBraces == 0) { - debug("ArrayFound. Openbraces = 1") - parsingState = InsideArray - openBraces += 1 - byteBuffer = byteBuffer.slice() - position = 0 - } else if (!arrayFound && Character.isWhitespace(currByte.toChar)) { - debug("Skip space") - } else { - decodeByteAndUpdateState(currByte, byteBuffer) - if (!insideString && (openBraces == 1 && currByte == ',' || openBraces == 0 && currByte == ']')) { - result += extractBuf() - - if (currByte == ']') { - debug("Done") - done = true - } - } - } - } - - result - } - - /* Private */ - - private def decodeByteAndUpdateState(c: Byte, in: ByteBuffer): Unit = { - debug("decode '" + c.toChar + "'") - if ((c == '{' || c == '[') && !insideString) { - openBraces += 1 - debug("openBraces = " + openBraces) - } else if ((c == '}' || c == ']') && !insideString) { - openBraces -= 1 - debug("openBraces = " + openBraces) - } else if (c == '"') { - // start of a new JSON string. It's necessary to detect strings as they may - // also contain braces/brackets and that could lead to incorrect results. - if (!insideString) { - debug("State = InsideString") - parsingState = InsideString - } - // If the double quote wasn't escaped then this is the end of a string. - else if (in.get(in.position() - 2) != '\\') { - debug("State = Parsing") - parsingState = Normal - } - } - } - - //TODO: Optimize - private def extractBuf(): Buf = { - val copy = byteBuffer.duplicate() - copy.position(0) - copy.limit(byteBuffer.position() - 1) - val copyBuf = Buf.ByteBuffer.Shared(copy) - - byteBuffer = byteBuffer.slice() - position = 0 - debug("Extract result " + copyBuf.utf8str) - copyBuf - } - - private def assertDecode(inputBuf: Buf): Unit = { - debug("Decode called with \"" + inputBuf.utf8str + "\"") - if (done) { - throw new scala.Exception("End array already found") - } - } - - private def insideString = parsingState == InsideString - - private def arrayFound = parsingState == InsideArray - - private[finatra] def copiedByteBuffer = byteBuffer.duplicate() -} diff --git a/jackson/src/main/scala/com/twitter/finatra/json/internal/streaming/JsonStreamParser.scala b/jackson/src/main/scala/com/twitter/finatra/json/internal/streaming/JsonStreamParser.scala index bba1e461f8..f527254dda 100644 --- a/jackson/src/main/scala/com/twitter/finatra/json/internal/streaming/JsonStreamParser.scala +++ b/jackson/src/main/scala/com/twitter/finatra/json/internal/streaming/JsonStreamParser.scala @@ -14,11 +14,15 @@ private[finatra] class JsonStreamParser @Inject()(mapper: FinatraObjectMapper) { } def parseArray[T: Manifest](bufs: AsyncStream[Buf]): AsyncStream[T] = { - val jsonDecoder = new JsonArrayChunker() - for { - buf <- bufs - jsonArrayDelimitedBuf <- AsyncStream.fromSeq(jsonDecoder.decode(buf)) - parsedElem = mapper.parse[T](jsonArrayDelimitedBuf) - } yield parsedElem + Reader.toAsyncStream(parseJson(Reader.fromAsyncStream(bufs))) + } + + def parseJson[T: Manifest](reader: Reader[Buf]): Reader[T] = { + val asyncJsonParser = new AsyncJsonParser + reader.flatMap { buf => + val bufs: Seq[Buf] = asyncJsonParser.feedAndParse(buf) + val values: Seq[T] = bufs.map(mapper.parse[T]) + Reader.fromSeq(values) + } } } diff --git a/jackson/src/main/scala/com/twitter/finatra/json/internal/streaming/ParsingState.scala b/jackson/src/main/scala/com/twitter/finatra/json/internal/streaming/ParsingState.scala deleted file mode 100644 index ea2a7f6462..0000000000 --- a/jackson/src/main/scala/com/twitter/finatra/json/internal/streaming/ParsingState.scala +++ /dev/null @@ -1,13 +0,0 @@ -package com.twitter.finatra.json.internal.streaming - -private[finatra] sealed trait ParsingState - -private[finatra] object ParsingState { - - case object Normal extends ParsingState - - case object InsideString extends ParsingState - - case object InsideArray extends ParsingState - -} \ No newline at end of file diff --git a/jackson/src/main/scala/com/twitter/finatra/validation/ErrorCode.scala b/jackson/src/main/scala/com/twitter/finatra/validation/ErrorCode.scala index c57a366409..e6d1f3e428 100644 --- a/jackson/src/main/scala/com/twitter/finatra/validation/ErrorCode.scala +++ b/jackson/src/main/scala/com/twitter/finatra/validation/ErrorCode.scala @@ -25,4 +25,6 @@ object ErrorCode { case class ValueOutOfRange(value: Number, min: Long, max: Long) extends ErrorCode case class ValueTooLarge(maxValue: Long, value: Number) extends ErrorCode case class ValueTooSmall(minValue: Long, value: Number) extends ErrorCode + case class PatternNotMatched(value: String, regex: String) extends ErrorCode + case class PatternSyntaxError(message: String, regex: String) extends ErrorCode } diff --git a/jackson/src/main/scala/com/twitter/finatra/validation/package.scala b/jackson/src/main/scala/com/twitter/finatra/validation/package.scala index 713bac2037..0346fd9f04 100644 --- a/jackson/src/main/scala/com/twitter/finatra/validation/package.scala +++ b/jackson/src/main/scala/com/twitter/finatra/validation/package.scala @@ -21,4 +21,5 @@ package object validation { type TimeGranularity = com.twitter.finatra.json.internal.caseclass.validation.validators.TimeGranularityInternal @param type UUID = com.twitter.finatra.json.internal.caseclass.validation.validators.UUIDInternal @param + type Pattern = com.twitter.finatra.json.internal.caseclass.validation.validators.PatternInternal @param } diff --git a/jackson/src/test/scala/com/twitter/finatra/json/tests/FinatraObjectMapperTest.scala b/jackson/src/test/scala/com/twitter/finatra/json/tests/FinatraObjectMapperTest.scala index 087fc9bed1..c235795f8e 100644 --- a/jackson/src/test/scala/com/twitter/finatra/json/tests/FinatraObjectMapperTest.scala +++ b/jackson/src/test/scala/com/twitter/finatra/json/tests/FinatraObjectMapperTest.scala @@ -25,6 +25,7 @@ import com.twitter.finatra.json.tests.internal.Obj.{ } import com.twitter.finatra.json.tests.internal.TypeAndCompanion.NestedCaseClassInCompanion import com.twitter.finatra.json.tests.internal._ +import com.twitter.finatra.json.tests.internal.caseclass.jackson.Aum import com.twitter.finatra.json.tests.internal.internal.{ SimplePersonInPackageObject, SimplePersonInPackageObjectWithoutConstructorParams @@ -50,6 +51,19 @@ class FinatraObjectMapperTest extends Test with Logging { private[this] val injector = TestInjector(FinatraJacksonModule).create + test("JsonProperty#annotation inheritance") { + val aumJson = """{"i":1,"j":"J"}""" + val aum = parse[Aum](aumJson) + aum should equal(Aum(1, "J")) + mapper.writeValueAsString(Aum(1, "J")) should equal(aumJson) + + val testCaseClassJson = """{"fedoras":["felt","straw"],"oldness":27}""" + val testCaseClass = parse[CaseClassTraitImpl](testCaseClassJson) + testCaseClass should equal(CaseClassTraitImpl(Seq("felt", "straw"), 27)) + mapper.writeValueAsString(CaseClassTraitImpl(Seq("felt", "straw"), 27)) should equal( + testCaseClassJson) + } + test("simple tests#parse super simple") { val foo = parse[SimplePerson]("""{"name": "Steve"}""") foo should equal(SimplePerson("Steve")) @@ -356,9 +370,11 @@ class FinatraObjectMapperTest extends Test with Logging { } test("Jodatime#invalid DateTime") { - assertJsonParse[CaseClassWithDateTime]("""{ + assertJsonParse[CaseClassWithDateTime]( + """{ "date_time" : "" - }""", withErrors = Seq("""date_time: field cannot be empty""")) + }""", + withErrors = Seq("""date_time: field cannot be empty""")) } test("Jodatime#invalid DateTime's") { @@ -1089,15 +1105,19 @@ class FinatraObjectMapperTest extends Test with Logging { } test("case class with boolean as string") { - assertJsonParse[CaseClassWithBoolean](""" { + assertJsonParse[CaseClassWithBoolean]( + """ { "foo": "bar" - }""", withErrors = Seq("foo: 'bar' is not a valid Boolean")) + }""", + withErrors = Seq("foo: 'bar' is not a valid Boolean")) } test("case class with boolean number as string") { - assertJsonParse[CaseClassWithBoolean](""" { + assertJsonParse[CaseClassWithBoolean]( + """ { "foo": "1" - }""", withErrors = Seq("foo: '1' is not a valid Boolean")) + }""", + withErrors = Seq("foo: '1' is not a valid Boolean")) } val msgHiJsonStr = """{"msg":"hi"}""" diff --git a/jackson/src/test/scala/com/twitter/finatra/json/tests/internal/caseclass/jackson/CaseClassFieldTest.scala b/jackson/src/test/scala/com/twitter/finatra/json/tests/internal/caseclass/jackson/CaseClassFieldTest.scala index 4a932e4c94..26d631d5b0 100644 --- a/jackson/src/test/scala/com/twitter/finatra/json/tests/internal/caseclass/jackson/CaseClassFieldTest.scala +++ b/jackson/src/test/scala/com/twitter/finatra/json/tests/internal/caseclass/jackson/CaseClassFieldTest.scala @@ -1,13 +1,18 @@ package com.twitter.finatra.json.tests.internal.caseclass.jackson +import com.fasterxml.jackson.annotation.{JsonIgnore, JsonProperty} import com.fasterxml.jackson.databind.PropertyNamingStrategy import com.fasterxml.jackson.databind.`type`.TypeFactory +import com.fasterxml.jackson.databind.annotation.JsonDeserialize +import com.fasterxml.jackson.databind.deser.std.NumberDeserializers.BigDecimalDeserializer import com.twitter.finatra.json.internal.caseclass.jackson.CaseClassField import com.twitter.finatra.json.tests.internal.{ WithEmptyJsonProperty, WithNonemptyJsonProperty, WithoutJsonPropertyAnnotation } +import com.twitter.finatra.request.{Header, QueryParam} +import com.twitter.finatra.validation.NotEmpty import com.twitter.inject.Test class CaseClassFieldTest extends Test { @@ -44,4 +49,194 @@ class CaseClassFieldTest extends Test { fields.length should equal(1) fields.head.name should equal("bar") } + + test("CaseClassField.createFields sees inherited JsonProperty annotation") { + val fields = CaseClassField.createFields( + classOf[Aum], + PropertyNamingStrategy.LOWER_CAMEL_CASE, + TypeFactory.defaultInstance + ) + + fields.length should equal(2) + + val iField = fields.head + iField.name should equal("i") + iField.annotations.size should equal(1) + iField.annotations.head.annotationType() should be(classOf[JsonProperty]) + + val jField = fields.last + jField.name should equal("j") + jField.annotations.size should equal(1) + jField.annotations.head.annotationType() should be(classOf[JsonProperty]) + } + + test("CaseClassField.createFields sees inherited JsonProperty annotation 2") { + val fields = CaseClassField.createFields( + classOf[FooBar], + PropertyNamingStrategy.LOWER_CAMEL_CASE, + TypeFactory.defaultInstance + ) + + fields.length should equal(1) + + val helloField = fields.head + helloField.name should equal("helloWorld") + helloField.annotations.size should equal(2) + helloField.annotations.head.annotationType() should be(classOf[JsonProperty]) + helloField.annotations.exists(_.annotationType() == classOf[Header]) should be(true) + helloField.annotations.last.asInstanceOf[Header].value() should be("accept") // from Bar + } + + test("CaseClassField.createFields sees inherited JsonProperty annotation 3") { + val fields = CaseClassField.createFields( + classOf[TestTraitImpl], + PropertyNamingStrategy.LOWER_CAMEL_CASE, + TypeFactory.defaultInstance + ) + + /* + in trait: + --------- + @JsonProperty("oldness") + def age: Int + @NotEmpty + def name: String + + case class constructor: + ----------------------- + @JsonProperty("ageness") age: Int, // should override inherited annotation from trait + @Header name: String, // should have two annotations, one from trait and one here + @QueryParam dateTime: DateTime, + @JsonProperty foo: String, // empty JsonProperty should default to field name + @JsonDeserialize(contentAs = classOf[BigDecimal], using = classOf[BigDecimalDeserializer]) + double: BigDecimal, + @JsonIgnore ignoreMe: String + */ + fields.length should equal(6) + + val fieldMap: Map[String, CaseClassField] = + fields.map(field => field.name -> field).toMap + + val ageField = fieldMap("ageness") + ageField.annotations.size should equal(1) + ageField.annotations.exists(_.annotationType() == classOf[JsonProperty]) should be(true) + ageField.annotations.head.asInstanceOf[JsonProperty].value() should equal("ageness") + + val nameField = fieldMap("name") + nameField.annotations.size should equal(2) + nameField.annotations.exists(_.annotationType() == classOf[NotEmpty]) should be(true) + nameField.annotations.exists(_.annotationType() == classOf[Header]) should be(true) + + val dateTimeField = fieldMap("dateTime") + dateTimeField.annotations.size should equal(1) + dateTimeField.annotations.exists(_.annotationType() == classOf[QueryParam]) should be(true) + + val fooField = fieldMap("foo") + fooField.annotations.size should equal(1) + fooField.annotations.exists(_.annotationType() == classOf[JsonProperty]) should be(true) + fooField.annotations.head.asInstanceOf[JsonProperty].value() should equal("") + + val doubleField = fieldMap("double") + doubleField.annotations.size should equal(1) + doubleField.annotations.exists(_.annotationType() == classOf[JsonDeserialize]) should be(true) + doubleField.annotations.head.asInstanceOf[JsonDeserialize].contentAs() should be( + classOf[BigDecimal]) + doubleField.annotations.head.asInstanceOf[JsonDeserialize].using() should be( + classOf[BigDecimalDeserializer]) + doubleField.deserializer should not be None + + val ignoreMeField = fieldMap("ignoreMe") + ignoreMeField.annotations.size should equal(1) + ignoreMeField.annotations.exists(_.annotationType() == classOf[JsonIgnore]) should be(true) + } + + test("CaseClassField.createFields sees inherited JsonProperty annotation 4") { + val fields = CaseClassField.createFields( + classOf[FooBaz], + PropertyNamingStrategy.LOWER_CAMEL_CASE, + TypeFactory.defaultInstance + ) + + fields.length should equal(1) + val helloField: CaseClassField = fields.head + helloField.annotations.size should equal(2) + helloField.annotations.exists(_.annotationType() == classOf[JsonProperty]) should be(true) + helloField.annotations.head.asInstanceOf[JsonProperty].value() should equal("goodbyeWorld") // from Baz + + helloField.annotations.exists(_.annotationType() == classOf[Header]) should be(true) + helloField.annotations.last.asInstanceOf[Header].value() should be("accept") // from Bar + } + + test("CaseClassField.createFields sees inherited JsonProperty annotation 5") { + val fields = CaseClassField.createFields( + classOf[FooBarBaz], + PropertyNamingStrategy.LOWER_CAMEL_CASE, + TypeFactory.defaultInstance + ) + + fields.length should equal(1) + val helloField: CaseClassField = fields.head + helloField.annotations.size should equal(2) + helloField.annotations.exists(_.annotationType() == classOf[JsonProperty]) should be(true) + helloField.annotations.head.asInstanceOf[JsonProperty].value() should equal("goodbye") // from BarBaz + + helloField.annotations.exists(_.annotationType() == classOf[Header]) should be(true) + helloField.annotations.last.asInstanceOf[Header].value() should be("accept") // from Bar + } + + test("CaseClassField.createFields sees inherited JsonProperty annotation 6") { + val fields = CaseClassField.createFields( + classOf[File], + PropertyNamingStrategy.LOWER_CAMEL_CASE, + TypeFactory.defaultInstance + ) + + fields.length should equal(1) + val uriField: CaseClassField = fields.head + uriField.annotations.size should equal(1) + uriField.annotations.exists(_.annotationType() == classOf[JsonProperty]) should be(true) + uriField.annotations.head.asInstanceOf[JsonProperty].value() should equal("file") + } + + test("CaseClassField.createFields sees inherited JsonProperty annotation 7") { + val fields = CaseClassField.createFields( + classOf[Folder], + PropertyNamingStrategy.LOWER_CAMEL_CASE, + TypeFactory.defaultInstance + ) + + fields.length should equal(1) + val uriField: CaseClassField = fields.head + uriField.annotations.size should equal(1) + uriField.annotations.exists(_.annotationType() == classOf[JsonProperty]) should be(true) + uriField.annotations.head.asInstanceOf[JsonProperty].value() should equal("folder") + } + + test("CaseClassField.createFields sees inherited JsonProperty annotation 8") { + val fields = CaseClassField.createFields( + classOf[LoadableFile], + PropertyNamingStrategy.LOWER_CAMEL_CASE, + TypeFactory.defaultInstance + ) + + fields.length should equal(1) + val uriField: CaseClassField = fields.head + uriField.annotations.size should equal(1) + uriField.annotations.exists(_.annotationType() == classOf[JsonProperty]) should be(true) + uriField.annotations.head.asInstanceOf[JsonProperty].value() should equal("file") + } + + test("CaseClassField.createFields sees inherited JsonProperty annotation 9") { + val fields = CaseClassField.createFields( + classOf[LoadableFolder], + PropertyNamingStrategy.LOWER_CAMEL_CASE, + TypeFactory.defaultInstance + ) + + fields.length should equal(1) + val uriField: CaseClassField = fields.head + uriField.annotations.size should equal(1) + uriField.annotations.exists(_.annotationType() == classOf[JsonProperty]) should be(true) + uriField.annotations.head.asInstanceOf[JsonProperty].value() should equal("folder") + } } diff --git a/jackson/src/test/scala/com/twitter/finatra/json/tests/internal/caseclass/jackson/caseclasses.scala b/jackson/src/test/scala/com/twitter/finatra/json/tests/internal/caseclass/jackson/caseclasses.scala new file mode 100644 index 0000000000..c3e72bb7d2 --- /dev/null +++ b/jackson/src/test/scala/com/twitter/finatra/json/tests/internal/caseclass/jackson/caseclasses.scala @@ -0,0 +1,72 @@ +package com.twitter.finatra.json.tests.internal.caseclass.jackson + +import com.fasterxml.jackson.annotation.{JsonIgnore, JsonProperty} +import com.fasterxml.jackson.databind.annotation.JsonDeserialize +import com.fasterxml.jackson.databind.deser.std.NumberDeserializers.BigDecimalDeserializer +import com.twitter.finatra.request.{Header, QueryParam} +import com.twitter.finatra.response.JsonCamelCase +import com.twitter.finatra.validation.NotEmpty +import org.joda.time.DateTime + +/* Note: the decoder automatically changes "_i" to "i" for de/serialization: + * See CaseClassField#jsonNameForField */ +trait Aumly { @JsonProperty("i") def _i: Int; @JsonProperty("j") def _j: String } +case class Aum(_i: Int, _j: String) extends Aumly + +trait Bar { + @JsonProperty("helloWorld") @Header("accept") + def hello: String +} +case class FooBar(hello: String) extends Bar + +trait Baz extends Bar { + @JsonProperty("goodbyeWorld") + def hello: String +} +case class FooBaz(hello: String) extends Baz + +trait BarBaz { + @JsonProperty("goodbye") + def hello: String +} +case class FooBarBaz(hello: String) extends BarBaz with Bar // will end up with BarBaz @JsonProperty value as trait linearization is "right-to-left" + +trait Loadable { + @JsonProperty("url") + def uri: String +} +abstract class Resource { + @JsonProperty("resource") + def uri: String +} +case class File(@JsonProperty("file") uri : String) extends Resource +case class Folder(@JsonProperty("folder") uri : String) extends Resource + +abstract class LoadableResource extends Loadable { + @JsonProperty("resource") + override def uri: String +} +case class LoadableFile(@JsonProperty("file") uri : String) extends LoadableResource +case class LoadableFolder(@JsonProperty("folder") uri : String) extends LoadableResource + +trait TestTrait { + @JsonProperty("oldness") + def age: Int + @NotEmpty + def name: String +} +@JsonCamelCase +case class TestTraitImpl( + @JsonProperty("ageness") age: Int,// should override inherited annotation from trait + @Header name: String, // should have two annotations, one from trait and one here + @QueryParam dateTime: DateTime, + @JsonProperty foo: String, + @JsonDeserialize(contentAs = classOf[BigDecimal], using = classOf[BigDecimalDeserializer]) + double: BigDecimal, + @JsonIgnore ignoreMe: String +) extends TestTrait { + + lazy val testFoo: String = "foo" + lazy val testBar: String = "bar" +} + diff --git a/jackson/src/test/scala/com/twitter/finatra/json/tests/internal/caseclass/validation/validators/PatternValidatorTest.scala b/jackson/src/test/scala/com/twitter/finatra/json/tests/internal/caseclass/validation/validators/PatternValidatorTest.scala new file mode 100644 index 0000000000..19b3d02078 --- /dev/null +++ b/jackson/src/test/scala/com/twitter/finatra/json/tests/internal/caseclass/validation/validators/PatternValidatorTest.scala @@ -0,0 +1,75 @@ +package com.twitter.finatra.json.tests.internal.caseclass.validation.validators + +import com.twitter.finatra.json.internal.caseclass.validation.validators.PatternValidator +import com.twitter.finatra.validation.ValidationResult.{Invalid, Valid} +import com.twitter.finatra.validation.{ErrorCode, Pattern, ValidationResult, ValidatorTest} +import org.scalacheck.Gen +import org.scalatest.prop.GeneratorDrivenPropertyChecks + +case class NumberPatternExample(@Pattern(regexp = "[0-9]+") stringValue: String) +case class NumberPatternArrayExample(@Pattern(regexp = "[0-9]+") stringValue: Array[String]) +case class EmptyPatternExample(@Pattern(regexp = "") stringValue: String) +case class InvalidPatternExample(@Pattern(regexp = "([)") stringValue: String) + +class PatternValidatorTest extends ValidatorTest with GeneratorDrivenPropertyChecks { + + test("pass validation when regex matches for array type") { + val passValue = for { + size <- Gen.choose(10, 50) + } yield + Array.fill(size) { + Gen.choose(10, 100) + } + forAll(passValue) { value => + validate[NumberPatternArrayExample](value) should equal(Valid) + } + } + + test("pass validation when regex matches") { + validate[NumberPatternExample]("12345") should equal(Valid) + + } + + test("fail validation when regex not matches") { + validate[NumberPatternExample]("meros") should equal( + Invalid(errorMessage("meros", "[0-9]+"), ErrorCode.PatternNotMatched("meros", "[0-9]+")) + ) + } + + test("fail validation when regex not matches for a invalid value in array type") { + forAll(Traversable("invalid", "6666")) { value => + validate[NumberPatternArrayExample](value) should equal( + Invalid( + errorMessage(value.toString, "[0-9]+"), + ErrorCode.PatternNotMatched(value mkString ",", "[0-9]+")) + ) + } + } + + test("it should throw exception for invalid class type") { + the[IllegalArgumentException] thrownBy validate[NumberPatternArrayExample](new Object()) should have message + "Class [class java.lang.Object}] is not supported by class com.twitter.finatra.json.internal.caseclass.validation.validators.PatternValidator" + } + + test("pass validation when regex matches for traversable type") { + forAll(Traversable("1234", "6666")) { value => + validate[NumberPatternArrayExample](value) should equal(Valid) + } + } + + test("fail validation when regex is invalid") { + validate[InvalidPatternExample](value = "123") should equal( + Invalid("java.util.regex.PatternSyntaxException", + ErrorCode.PatternSyntaxError("Unclosed character class near index 2\n([)\n ^", "([)") + ) + ) + } + + private def validate[C: Manifest](value: Any): ValidationResult = { + super.validate(manifest[C].runtimeClass, "stringValue", classOf[Pattern], value) + } + + private def errorMessage(value: String, regex: String): String = { + PatternValidator.errorMessage(messageResolver, value, regex) + } +} diff --git a/jackson/src/test/scala/com/twitter/finatra/json/tests/internal/ExampleCaseClasses.scala b/jackson/src/test/scala/com/twitter/finatra/json/tests/internal/caseclasses.scala similarity index 97% rename from jackson/src/test/scala/com/twitter/finatra/json/tests/internal/ExampleCaseClasses.scala rename to jackson/src/test/scala/com/twitter/finatra/json/tests/internal/caseclasses.scala index 80e999e055..1eab52fbe1 100644 --- a/jackson/src/test/scala/com/twitter/finatra/json/tests/internal/ExampleCaseClasses.scala +++ b/jackson/src/test/scala/com/twitter/finatra/json/tests/internal/caseclasses.scala @@ -362,6 +362,19 @@ case class WithoutJsonPropertyAnnotation(foo: String) case class NamingStrategyJsonProperty(@JsonProperty longFieldName: String) +trait CaseClassTrait { + @JsonProperty("fedoras") + @Size(min = 1, max = 2) + def names: Seq[String] + + @Min(1L) + def age: Int +} +case class CaseClassTraitImpl( + names: Seq[String], + @JsonProperty("oldness") age: Int +) extends CaseClassTrait + package object internal { case class SimplePersonInPackageObject( // not recommended but used here for testing use case diff --git a/jackson/src/test/scala/com/twitter/finatra/json/tests/internal/streaming/AsyncJsonParserTest.scala b/jackson/src/test/scala/com/twitter/finatra/json/tests/internal/streaming/AsyncJsonParserTest.scala new file mode 100644 index 0000000000..1ace3e41f9 --- /dev/null +++ b/jackson/src/test/scala/com/twitter/finatra/json/tests/internal/streaming/AsyncJsonParserTest.scala @@ -0,0 +1,270 @@ +package com.twitter.finatra.json.tests.internal.streaming + +import com.fasterxml.jackson.core.JsonParseException +import com.fasterxml.jackson.databind.JsonNode +import com.twitter.finatra.json.{FinatraObjectMapper, JsonDiff} +import com.twitter.finatra.json.internal.streaming.AsyncJsonParser +import com.twitter.inject.Test +import com.twitter.io.Buf +import com.twitter.inject.conversions.buf._ +import org.scalacheck.{Gen, Shrink} +import org.scalatest.prop.GeneratorDrivenPropertyChecks + +class AsyncJsonParserTest extends Test with GeneratorDrivenPropertyChecks { + + test("decode") { + val parser = new AsyncJsonParser() + + assertParsing( + parser, + input = "[1", + output = Seq(), + remainder = "[1", + pos = 0, + depth = 1 + ) + + assertParsing( + parser, + input = ",2", + output = Seq("1"), + remainder = ",2", + pos = 0, + depth = 1 + ) + + assertParsing( + parser, + input = ",3", + output = Seq("2"), + remainder = ",3", + pos = 0, + depth = 1 + ) + + assertParsing( + parser, + input = "]", + output = Seq("3"), + remainder = "]", + pos = 0, + depth = -1 + ) + } + + test("decode with empty json") { + val jsonObj = "" + assertJsonParse(jsonObj, 0) + } + + test("decode with nested arrays") { + val jsonObj = """[1, 2, 3]""" + assertJsonParse(jsonObj, 1) + } + + test("decode with nested objects") { + val jsonObj = """ + { + "sub_object": { + "msg": "hi" + } + } + """ + + assertJsonParse(jsonObj, 1) + } + + test("decode with nested objects in array") { + val jsonObj = """ + { + "sub_object1": { + "nested_item": { + "msg": "hi" + } + } + }, + {"sub_object2": { + "msg": "hi" + } + } + """ + assertJsonParse(jsonObj, 2) + } + + test("invalid json field name") { + val jsonObj = """ + { + "sub_object1": { + {"this" : "shouldnot}: "work" + } + }, + {"sub_object2": { + "msg": "hi" + } + } + """ + intercept[JsonParseException] { + assertJsonParse(jsonObj, 2) + } + } + + test("miss use of curlies and squares ") { + val jsonObj = """ + [ + { + "msg": "hi" + ], + } + """ + intercept[JsonParseException] { + assertJsonParse(jsonObj, 1) + } + } + + test("decode json inside a string") { + val jsonObj = """{"foo": "bar"}""" + assertJsonParse(jsonObj, 1) + } + + test("Calling decode when already finished") { + val parser = new AsyncJsonParser() + parser.feedAndParse(Buf.Utf8("[]")) + intercept[Exception] { + parser.feedAndParse(Buf.Utf8("{}")) + } + } + + test("other Buf types") { + val parser = new AsyncJsonParser() + val jsonObj = """{"foo": "bar"}""" + val result = parser.feedAndParse( + Buf.ByteBuffer.Owned(java.nio.ByteBuffer + .wrap("[".getBytes("UTF-8") ++ jsonObj.getBytes("UTF-8") ++ "]".getBytes("UTF-8")))) + val mapper = FinatraObjectMapper.create() + val nodes = result.map(mapper.parse[JsonNode]) + nodes.size should be(1) + JsonDiff.jsonDiff(nodes.head, jsonObj) + } + + test("decode json split up over multiple chunks") { + + implicit val noShrink: Shrink[Seq[Buf]] = Shrink.shrinkAny + + forAll(genJsonObj) { json => + forAll(chunk(Buf.Utf8(json))) { bufs => + testChunks(json, bufs) + } + } + } + + private def genJsonObj: Gen[String] = { + implicit val noShrink: Shrink[Seq[Buf]] = Shrink.shrinkAny + val jsonObjs = """ + [ + { + "id": "John" + }, + { + "id": "Tom" + }, + { + "id": "An" + }, + { + "id": "Jean" + } + ]""" + + val nestedObjs = """ + [ + { + "sub_object1": { + "nested_item": { + "msg": "hi" + } + } + }, + {"sub_object2": { + "msg": "hi" + } + } + ]""" + + val nestedLists = """ + [ + [1, 2, 3], + [4, 5, 6], + [7, 8, [1, 2, 3]] + ]""" + + val listAndObjs = """ + [ + { + "id": "John" + }, + { + "id": "Tom" + }, + [1,2,3,4], + { + "id": "An" + } + ]""" + + Gen.oneOf(Seq(jsonObjs, nestedObjs, nestedLists, listAndObjs)) + } + + private def chunk(buf: Buf): Gen[Seq[Buf]] = + if (buf.isEmpty) Gen.const(Seq.empty[Buf]) + else { + Gen.choose(1, buf.length).flatMap { n => + val taken = math.min(buf.length, n) + val a = buf.slice(0, taken) + val b = buf.slice(taken, buf.length) + + chunk(b).map(rest => a +: rest) + } + } + + private def testChunks(jsonObj: String, chunks: Seq[Buf]): Unit = { + val mapper = FinatraObjectMapper.create() + val parser = new AsyncJsonParser() + + val nodes: Seq[JsonNode] = chunks.flatMap { buf => + val bufs: Seq[Buf] = parser.feedAndParse(buf) + bufs.map(mapper.parse[JsonNode]) + } + JsonDiff.jsonDiff(nodes, jsonObj) + } + + private def assertParsing( + parser: AsyncJsonParser, + input: String, + output: Seq[String], + remainder: String, + pos: Int, + depth: Int = 0 + ): Unit = { + + val result: Seq[Buf] = parser.feedAndParse(Buf.Utf8(input)) + result.map { _.utf8str } should equal(output) + + val copiedByteBuffer = parser.copiedByteBuffer.duplicate() + copiedByteBuffer.position(0) + val recvBuf = Buf.ByteBuffer.Shared(copiedByteBuffer) + + recvBuf.utf8str should equal(remainder) + parser.copiedByteBuffer.position() should equal(pos) + parser.getParsingDepth should equal(depth) + } + + private def assertJsonParse(jsonObj: String, size: Int): Unit = { + val mapper = FinatraObjectMapper.create() + + val parser = new AsyncJsonParser() + val result: Seq[Buf] = parser.feedAndParse(Buf.Utf8("[" + jsonObj + "]")) + val nodes: Seq[JsonNode] = result.map(mapper.parse[JsonNode]) + nodes.size should be(size) + if (size > 0) JsonDiff.jsonDiff(nodes.head, jsonObj) + } +} diff --git a/jackson/src/test/scala/com/twitter/finatra/json/tests/internal/streaming/JsonObjectDecoderTest.scala b/jackson/src/test/scala/com/twitter/finatra/json/tests/internal/streaming/JsonObjectDecoderTest.scala deleted file mode 100644 index a6c15af988..0000000000 --- a/jackson/src/test/scala/com/twitter/finatra/json/tests/internal/streaming/JsonObjectDecoderTest.scala +++ /dev/null @@ -1,101 +0,0 @@ -package com.twitter.finatra.json.tests.internal.streaming - -import com.fasterxml.jackson.databind.JsonNode -import com.twitter.inject.conversions.buf._ -import com.twitter.finatra.json.internal.streaming.{JsonArrayChunker, ParsingState} -import com.twitter.finatra.json.{FinatraObjectMapper, JsonDiff} -import com.twitter.finatra.json.internal.streaming.ParsingState._ -import com.twitter.inject.Test -import com.twitter.io.Buf - -class JsonObjectDecoderTest extends Test { - - test("decode") { - val decoder = new JsonArrayChunker() - - assertDecode( - decoder, - input = "[1", - output = Seq(), - remainder = "1", - pos = 1, - openBraces = 1, - parsingState = InsideArray - ) - - assertDecode(decoder, input = ",2", output = Seq("1"), remainder = "2", pos = 1) - - assertDecode(decoder, input = ",3", output = Seq("2"), remainder = "3", pos = 1) - - assertDecode( - decoder, - input = "]", - output = Seq("3"), - remainder = "", - pos = 0, - openBraces = 0, - done = true - ) - } - - val mapper = FinatraObjectMapper.create() - - test("decode with nested objects") { - val jsonObj = """ - { - "sub_object": { - "msg": "hi" - } - } - """ - - assertSingleJsonParse(jsonObj) - } - - test("decode json inside a string") { - val jsonObj = """{"foo": "bar"}""" - assertSingleJsonParse(jsonObj) - } - - test("Caling decode when already finished") { - val decoder = new JsonArrayChunker() - decoder.decode(Buf.Utf8("[]")) - intercept[Exception] { - decoder.decode(Buf.Utf8("{}")) - } - } - - private def assertDecode( - decoder: JsonArrayChunker, - input: String, - output: Seq[String], - remainder: String, - pos: Int, - openBraces: Int = 1, - parsingState: ParsingState = InsideArray, - done: Boolean = false - ): Unit = { - - val result = decoder.decode(Buf.Utf8(input)) - result map { _.utf8str } should equal(output) - - val copiedByteBuffer = decoder.copiedByteBuffer.duplicate() - copiedByteBuffer.position(0) - val recvBuf = Buf.ByteBuffer.Shared(copiedByteBuffer) - println("Result remainder: " + recvBuf.utf8str) - - recvBuf.utf8str should equal(remainder) - decoder.copiedByteBuffer.position() should equal(pos) - decoder.openBraces should equal(openBraces) - decoder.parsingState should equal(parsingState) - decoder.done should equal(done) - } - - def assertSingleJsonParse(jsonObj: String): Unit = { - val decoder = new JsonArrayChunker() - val result = decoder.decode(Buf.Utf8("[" + jsonObj + "]")) - val nodes = result map mapper.parse[JsonNode] - nodes.size should be(1) - JsonDiff.jsonDiff(nodes.head, jsonObj) - } -} diff --git a/jackson/src/test/scala/com/twitter/finatra/json/tests/internal/streaming/StreamingTest.scala b/jackson/src/test/scala/com/twitter/finatra/json/tests/internal/streaming/StreamingTest.scala index 9c248b78d7..9102b9864b 100644 --- a/jackson/src/test/scala/com/twitter/finatra/json/tests/internal/streaming/StreamingTest.scala +++ b/jackson/src/test/scala/com/twitter/finatra/json/tests/internal/streaming/StreamingTest.scala @@ -6,8 +6,9 @@ import com.twitter.finatra.json.FinatraObjectMapper import com.twitter.finatra.json.internal.streaming.JsonStreamParser import com.twitter.finatra.json.tests.internal.{CaseClassWithSeqBooleans, FooClass} import com.twitter.inject.Test -import com.twitter.io.Buf -import com.twitter.util.Await +import com.twitter.io.{Buf, Reader} +import com.twitter.util.{Await, Future} +import scala.collection.mutable.ArrayBuffer class StreamingTest extends Test { @@ -15,7 +16,7 @@ class StreamingTest extends Test { val jsonStr = "[1,2,3]" val expected123 = AsyncStream(1, 2, 3) - test("bufs to json") { + test("bufs to json - AsyncStream") { assertParsed( AsyncStream( Buf.Utf8(jsonStr.substring(0, 1)), @@ -27,7 +28,7 @@ class StreamingTest extends Test { ) } - test("bufs to json 2") { + test("bufs to json 2 - AsyncStream") { assertParsed( AsyncStream(Buf.Utf8("[1"), Buf.Utf8(",2"), Buf.Utf8(",3"), Buf.Utf8("]")), expectedInputStr = jsonStr, @@ -35,7 +36,7 @@ class StreamingTest extends Test { ) } - test("bufs to json 3") { + test("bufs to json 3 - AsyncStream") { assertParsed( AsyncStream(Buf.Utf8("[1"), Buf.Utf8(",2,3,44"), Buf.Utf8("4,5]")), expectedInputStr = "[1,2,3,444,5]", @@ -43,30 +44,47 @@ class StreamingTest extends Test { ) } - test("bufs to json with some bigger objects") { + test("bufs to json with some bigger objects - AsyncStream") { assertParsedSimpleObjects( - AsyncStream(Buf.Utf8("""[{ "id":"John" }"""), - Buf.Utf8(""",{ "id":"Tom" },{ "id":"An" },"""), - Buf.Utf8("""{ "id":"Jean" }]""")), + AsyncStream( + Buf.Utf8("""[{ "id":"John" }"""), + Buf.Utf8(""",{ "id":"Tom" },{ "id":"An" },"""), + Buf.Utf8("""{ "id":"Jean" }]""")), expectedInputStr = """[{ "id":"John" },{ "id":"Tom" },{ "id":"An" },{ "id":"Jean" }]""", expected = AsyncStream(FooClass("John"), FooClass("Tom"), FooClass("An"), FooClass("Jean")) ) } - test("bufs to json with some complex objects") { + test("bufs to json with some complex objects - AsyncStream") { assertParsedComplexObjects( - AsyncStream(Buf.Utf8("""[{ "foos":[true,false,true] }"""), - Buf.Utf8(""",{ "foos": [true,false,true] },"""), - Buf.Utf8("""{ "foos":[false] }]""")), - expectedInputStr = """[{ "foos":[true,false,true] },{ "foos": [true,false,true] },{ "foos":[false] }]""", - expected = AsyncStream(CaseClassWithSeqBooleans(Seq(true,false,true)), - CaseClassWithSeqBooleans(Seq(true,false,true)), CaseClassWithSeqBooleans(Seq(false))) + AsyncStream( + Buf.Utf8("""[{ "foos":[true,false,true] }"""), + Buf.Utf8(""",{ "foos": [true,false,true] },"""), + Buf.Utf8("""{ "foos":[false] }]""")), + expectedInputStr = + """[{ "foos":[true,false,true] },{ "foos": [true,false,true] },{ "foos":[false] }]""", + expected = AsyncStream( + CaseClassWithSeqBooleans(Seq(true, false, true)), + CaseClassWithSeqBooleans(Seq(true, false, true)), + CaseClassWithSeqBooleans(Seq(false))) ) } - test("bufs to json with some bigger objects and an escape character") { + test("bufs to json with some bigger objects and an escape character - AsyncStream") { assertParsedSimpleObjects( - AsyncStream(Buf.Utf8("""[{ "id":"John" }"""), + AsyncStream( + Buf.Utf8("""[{ "id":"John" }"""), + Buf.Utf8(""",{ "id":"\\Tom" },{ "id""""), + Buf.Utf8(""":"An" },{ "id":"Jean" }]""")), + expectedInputStr = """[{ "id":"John" },{ "id":"\\Tom" },{ "id":"An" },{ "id":"Jean" }]""", + expected = AsyncStream(FooClass("John"), FooClass("\\Tom"), FooClass("An"), FooClass("Jean")) + ) + } + + test("bufs to json with some bigger objects but with json object split over multiple bufs - AsyncStream") { + assertParsedSimpleObjects( + AsyncStream( + Buf.Utf8("""[{ "id":"John" }"""), Buf.Utf8(""",{ "id":"Tom" },{ "id""""), Buf.Utf8(""":"An" },{ "id":"Jean" }]""")), expectedInputStr = """[{ "id":"John" },{ "id":"Tom" },{ "id":"An" },{ "id":"Jean" }]""", @@ -74,17 +92,97 @@ class StreamingTest extends Test { ) } - test("bufs to json with some bigger objects but with json object split over 2 bufs") { + test("parse request - AsyncStream") { + val jsonStr = "[1,2]" + val request = Request(Method.Post, "/") + request.setChunked(true) + + val parser = new JsonStreamParser(FinatraObjectMapper.create()) + + request.writer.write(Buf.Utf8(jsonStr)) ensure { + request.writer.close() + } + + assertFutureValue(parser.parseArray[Int](request.reader).toSeq(), Seq(1, 2)) + } + + test("bufs to json - Reader") { + assertParsed( + Reader.fromSeq( + Seq( + Buf.Utf8(jsonStr.substring(0, 1)), + Buf.Utf8(jsonStr.substring(1, 4)), + Buf.Utf8(jsonStr.substring(4)))), + expected = Reader.fromSeq(Seq(1, 2, 3)) + ) + } + + test("bufs to json 2 - Reader") { + assertParsed( + Reader.fromSeq(Seq(Buf.Utf8("[1"), Buf.Utf8(",2"), Buf.Utf8(",3"), Buf.Utf8("]"))), + expected = Reader.fromSeq(Seq(1, 2, 3)) + ) + } + + test("bufs to json 3 - Reader") { + assertParsed( + Reader.fromSeq(Seq(Buf.Utf8("[1"), Buf.Utf8(",2,3,44"), Buf.Utf8("4,5]"))), + expected = Reader.fromSeq(Seq(1, 2, 3, 444, 5)) + ) + } + + test("bufs to json with some bigger objects - Reader") { assertParsedSimpleObjects( - AsyncStream(Buf.Utf8("""[{ "id":"John" }"""), + Reader.fromSeq( + Seq( + Buf.Utf8("""[{ "id":"John" }"""), + Buf.Utf8(""",{ "id":"Tom" },{ "id":"An" },"""), + Buf.Utf8("""{ "id":"Jean" }]"""))), + expected = + Reader.fromSeq(Seq(FooClass("John"), FooClass("Tom"), FooClass("An"), FooClass("Jean"))) + ) + } + + test("bufs to json with some complex objects - Reader") { + assertParsedComplexObjects( + Reader.fromSeq( + Seq( + Buf.Utf8("""[{ "foos":[true,false,true] }"""), + Buf.Utf8(""",{ "foos": [true,false,true] },"""), + Buf.Utf8("""{ "foos":[false] }]"""))), + expected = Reader.fromSeq( + Seq( + CaseClassWithSeqBooleans(Seq(true, false, true)), + CaseClassWithSeqBooleans(Seq(true, false, true)), + CaseClassWithSeqBooleans(Seq(false)))) + ) + } + + test("bufs to json with some bigger objects and an escape character - Reader") { + assertParsedSimpleObjects( + Reader.fromSeq( + Seq( + Buf.Utf8("""[{ "id":"John" }"""), + Buf.Utf8(""",{ "id":"\\Tom" },{ "id""""), + Buf.Utf8(""":"An" },{ "id":"Jean" }]"""))), + expected = + Reader.fromSeq(Seq(FooClass("John"), FooClass("\\Tom"), FooClass("An"), FooClass("Jean"))) + ) + } + + test("bufs to json with some bigger objects but with json object split over multiple bufs - Reader") { + assertParsedSimpleObjects( + Reader.fromSeq( + Seq( + Buf.Utf8("""[{ "id":"John" }"""), Buf.Utf8(""",{ "id":"Tom" },{ "id""""), - Buf.Utf8(""":"An" },{ "id":"Jean" }]""")), - expectedInputStr = """[{ "id":"John" },{ "id":"Tom" },{ "id":"An" },{ "id":"Jean" }]""", - expected = AsyncStream(FooClass("John"), FooClass("Tom"), FooClass("An"), FooClass("Jean")) + Buf.Utf8(""":"An" },{ "id":"Jean" }]"""))), + expected = + Reader.fromSeq(Seq(FooClass("John"), FooClass("Tom"), FooClass("An"), FooClass("Jean"))) ) } - test("parse request") { + test("parse request - Reader") { val jsonStr = "[1,2]" val request = Request(Method.Post, "/") request.setChunked(true) @@ -139,4 +237,36 @@ class StreamingTest extends Test { assertFuture(parser.parseArray[CaseClassWithSeqBooleans](bufs).toSeq(), expected.toSeq()) } + private def assertParsed(reader: Reader[Buf], expected: Reader[Int]): Unit = { + val parser = new JsonStreamParser(FinatraObjectMapper.create()) + assertReader(parser.parseJson[Int](reader), expected) + } + + private def assertParsedSimpleObjects(reader: Reader[Buf], expected: Reader[FooClass]): Unit = { + val parser = new JsonStreamParser(FinatraObjectMapper.create()) + assertReader(parser.parseJson[FooClass](reader), expected) + } + + private def assertParsedComplexObjects( + reader: Reader[Buf], + expected: Reader[CaseClassWithSeqBooleans] + ): Unit = { + + val parser = new JsonStreamParser(FinatraObjectMapper.create()) + assertReader(parser.parseJson[CaseClassWithSeqBooleans](reader), expected) + } + + protected def assertReader[A](r1: Reader[A], r2: Reader[A]): Unit = { + assertFuture(readAll(r1), readAll(r2)) + } + + private def readAll[A](r: Reader[A]): Future[List[A]] = { + def loop(left: ArrayBuffer[A]): Future[List[A]] = + r.read().flatMap { + case Some(right) => loop(left += right) + case _ => Future.value(left.toList) + } + + loop(ArrayBuffer.empty[A]) + } } diff --git a/jackson/src/test/scala/com/twitter/finatra/validation/ValidatorTest.scala b/jackson/src/test/scala/com/twitter/finatra/validation/ValidatorTest.scala index a21db2541d..0a0f447c5c 100644 --- a/jackson/src/test/scala/com/twitter/finatra/validation/ValidatorTest.scala +++ b/jackson/src/test/scala/com/twitter/finatra/validation/ValidatorTest.scala @@ -23,10 +23,14 @@ class ValidatorTest extends Test { } def getValidationAnnotations(clazz: Class[_], paramName: String): Seq[Annotation] = { + val constructorParams = parseConstructorParams(clazz) + val annotations = findAnnotations(clazz, constructorParams) + for { - (param, annotations) <- parseConstructorParams(clazz).zip(constructorAnnotations(clazz)) + param <- constructorParams + paramAnnotations = annotations(param.name) if param.name.equals(paramName) - annotation <- annotations + annotation <- paramAnnotations if validationManager.isValidationAnnotation(annotation) } yield annotation } @@ -40,7 +44,10 @@ class ValidatorTest extends Test { findAnnotation(annotationClass, annotations) } - def findAnnotation[A <: Annotation](annotationClass: Class[A], annotations: Seq[Annotation]): A = { + def findAnnotation[A <: Annotation]( + annotationClass: Class[A], + annotations: Seq[Annotation] + ): A = { AnnotationUtils.findAnnotation(annotationClass, annotations) match { case Some(annotation) => annotation.asInstanceOf[A] diff --git a/kafka-streams/PROJECT b/kafka-streams/PROJECT index c57fb17484..878b3f9f1e 100644 --- a/kafka-streams/PROJECT +++ b/kafka-streams/PROJECT @@ -1,7 +1,9 @@ owners: + - csl-team:ldap - messaging-group:ldap - scosenza - dbress - adams watchers: + - csl-team@twitter.com - ds-messaging@twitter.com diff --git a/kafka-streams/kafka-streams-prerestore/src/main/scala/com/twitter/finatra/streams/prerestore/PreRestoreState.scala b/kafka-streams/kafka-streams-prerestore/src/main/scala/com/twitter/finatra/kafkastreams/prerestore/PreRestoreState.scala similarity index 97% rename from kafka-streams/kafka-streams-prerestore/src/main/scala/com/twitter/finatra/streams/prerestore/PreRestoreState.scala rename to kafka-streams/kafka-streams-prerestore/src/main/scala/com/twitter/finatra/kafkastreams/prerestore/PreRestoreState.scala index bbcfcd64c8..820d461fd2 100644 --- a/kafka-streams/kafka-streams-prerestore/src/main/scala/com/twitter/finatra/streams/prerestore/PreRestoreState.scala +++ b/kafka-streams/kafka-streams-prerestore/src/main/scala/com/twitter/finatra/kafkastreams/prerestore/PreRestoreState.scala @@ -1,10 +1,10 @@ -package com.twitter.finatra.streams.prerestore +package com.twitter.finatra.kafkastreams.prerestore -import com.twitter.finatra.annotations.Experimental import com.twitter.conversions.DurationOps._ +import com.twitter.finatra.annotations.Experimental import com.twitter.finatra.kafkastreams.KafkaStreamsTwitterServer import com.twitter.finatra.kafkastreams.internal.utils.ReflectionUtils -import com.twitter.finatra.streams.partitioning.StaticPartitioning +import com.twitter.finatra.kafkastreams.partitioning.StaticPartitioning import java.util.Properties import java.util.concurrent.TimeUnit import java.util.concurrent.atomic.AtomicInteger diff --git a/kafka-streams/kafka-streams-prerestore/src/test/scala/com/twitter/finatra/kafkastreams/integration/wordcount/PreRestoreWordCountRocksDbServer.scala b/kafka-streams/kafka-streams-prerestore/src/test/scala/com/twitter/finatra/kafkastreams/integration/wordcount/PreRestoreWordCountRocksDbServer.scala index e7621ff58b..3b51af5001 100644 --- a/kafka-streams/kafka-streams-prerestore/src/test/scala/com/twitter/finatra/kafkastreams/integration/wordcount/PreRestoreWordCountRocksDbServer.scala +++ b/kafka-streams/kafka-streams-prerestore/src/test/scala/com/twitter/finatra/kafkastreams/integration/wordcount/PreRestoreWordCountRocksDbServer.scala @@ -2,7 +2,7 @@ package com.twitter.finatra.kafkastreams.integration.wordcount import com.twitter.finatra.kafka.serde.ScalaSerdes import com.twitter.finatra.kafkastreams.KafkaStreamsTwitterServer -import com.twitter.finatra.streams.prerestore.PreRestoreState +import com.twitter.finatra.kafkastreams.prerestore.PreRestoreState import org.apache.kafka.common.serialization.Serdes import org.apache.kafka.streams.StreamsBuilder import org.apache.kafka.streams.kstream.{Consumed, Materialized, Produced, Serialized} diff --git a/kafka-streams/kafka-streams-queryable-thrift-client/src/main/java/BUILD b/kafka-streams/kafka-streams-queryable-thrift-client/src/main/java/BUILD index 2f44d8bcd7..ef1977867f 100644 --- a/kafka-streams/kafka-streams-queryable-thrift-client/src/main/java/BUILD +++ b/kafka-streams/kafka-streams-queryable-thrift-client/src/main/java/BUILD @@ -6,3 +6,10 @@ java_library( exports = [ ], ) + +# TODO: Remove this and references to it, +# when a fix for https://github.com/pantsbuild/pants/issues/7200 has landed. +files( + name = "pants-workaround", + sources = rglobs("*.java"), +) diff --git a/kafka-streams/kafka-streams-queryable-thrift-client/src/main/scala/BUILD b/kafka-streams/kafka-streams-queryable-thrift-client/src/main/scala/BUILD index c9e26577fa..edf9ef5564 100644 --- a/kafka-streams/kafka-streams-queryable-thrift-client/src/main/scala/BUILD +++ b/kafka-streams/kafka-streams-queryable-thrift-client/src/main/scala/BUILD @@ -14,6 +14,7 @@ scala_library( "3rdparty/jvm/com/twitter/bijection:core", "finagle/finagle-serversets", "finatra/inject/inject-thrift-client", + "finatra/kafka-streams/kafka-streams-queryable-thrift-client/src/main/java:pants-workaround", ], exports = [ "3rdparty/jvm/com/twitter/bijection:core", diff --git a/kafka-streams/kafka-streams-queryable-thrift/src/main/scala/com/twitter/finatra/streams/queryable/thrift/QueryableState.scala b/kafka-streams/kafka-streams-queryable-thrift/src/main/scala/com/twitter/finatra/streams/queryable/thrift/QueryableState.scala index 7141ea270a..d23891cc9e 100644 --- a/kafka-streams/kafka-streams-queryable-thrift/src/main/scala/com/twitter/finatra/streams/queryable/thrift/QueryableState.scala +++ b/kafka-streams/kafka-streams-queryable-thrift/src/main/scala/com/twitter/finatra/streams/queryable/thrift/QueryableState.scala @@ -1,12 +1,8 @@ package com.twitter.finatra.streams.queryable.thrift import com.twitter.app.Flag -import com.twitter.finatra.streams.partitioning.StaticPartitioning -import com.twitter.finatra.streams.query.{ - QueryableFinatraCompositeWindowStore, - QueryableFinatraKeyValueStore, - QueryableFinatraWindowStore -} +import com.twitter.finatra.kafkastreams.partitioning.StaticPartitioning +import com.twitter.finatra.kafkastreams.query.{QueryableFinatraCompositeWindowStore, QueryableFinatraKeyValueStore, QueryableFinatraWindowStore} import com.twitter.util.Duration import org.apache.kafka.common.serialization.Serde @@ -40,14 +36,6 @@ trait QueryableState extends StaticPartitioning { currentShardId = currentShard()) } - @deprecated("Use queryableFinatraWindowStore without a windowSize", "1/7/2019") - protected def queryableFinatraWindowStore[K, V]( - storeName: String, - primaryKeySerde: Serde[K] - ): QueryableFinatraWindowStore[K, V] = { - queryableFinatraWindowStore(storeName, null, primaryKeySerde) - } - protected def queryableFinatraCompositeWindowStore[PK, SK, V]( storeName: String, windowSize: Duration, diff --git a/kafka-streams/kafka-streams-static-partitioning/src/main/scala/com/twitter/finatra/streams/partitioning/StaticPartitioning.scala b/kafka-streams/kafka-streams-static-partitioning/src/main/scala/com/twitter/finatra/kafkastreams/partitioning/StaticPartitioning.scala similarity index 80% rename from kafka-streams/kafka-streams-static-partitioning/src/main/scala/com/twitter/finatra/streams/partitioning/StaticPartitioning.scala rename to kafka-streams/kafka-streams-static-partitioning/src/main/scala/com/twitter/finatra/kafkastreams/partitioning/StaticPartitioning.scala index 76544b0636..a195451c3e 100644 --- a/kafka-streams/kafka-streams-static-partitioning/src/main/scala/com/twitter/finatra/streams/partitioning/StaticPartitioning.scala +++ b/kafka-streams/kafka-streams-static-partitioning/src/main/scala/com/twitter/finatra/kafkastreams/partitioning/StaticPartitioning.scala @@ -1,8 +1,8 @@ -package com.twitter.finatra.streams.partitioning +package com.twitter.finatra.kafkastreams.partitioning import com.twitter.app.Flag import com.twitter.finatra.kafkastreams.KafkaStreamsTwitterServer -import com.twitter.finatra.streams.partitioning.internal.StaticPartitioningKafkaClientSupplierSupplier +import com.twitter.finatra.kafkastreams.partitioning.internal.StaticPartitioningKafkaClientSupplierSupplier import org.apache.kafka.streams.KafkaClientSupplier object StaticPartitioning { diff --git a/kafka-streams/kafka-streams-static-partitioning/src/main/scala/com/twitter/finatra/streams/partitioning/internal/ClientStateAndHostInfo.scala b/kafka-streams/kafka-streams-static-partitioning/src/main/scala/com/twitter/finatra/kafkastreams/partitioning/internal/ClientStateAndHostInfo.scala similarity index 86% rename from kafka-streams/kafka-streams-static-partitioning/src/main/scala/com/twitter/finatra/streams/partitioning/internal/ClientStateAndHostInfo.scala rename to kafka-streams/kafka-streams-static-partitioning/src/main/scala/com/twitter/finatra/kafkastreams/partitioning/internal/ClientStateAndHostInfo.scala index 2735b33d20..8f7897f173 100644 --- a/kafka-streams/kafka-streams-static-partitioning/src/main/scala/com/twitter/finatra/streams/partitioning/internal/ClientStateAndHostInfo.scala +++ b/kafka-streams/kafka-streams-static-partitioning/src/main/scala/com/twitter/finatra/kafkastreams/partitioning/internal/ClientStateAndHostInfo.scala @@ -1,4 +1,4 @@ -package com.twitter.finatra.streams.partitioning.internal +package com.twitter.finatra.kafkastreams.partitioning.internal import com.twitter.finatra.streams.queryable.thrift.domain.ServiceShardId import org.apache.kafka.streams.processor.internals.assignment.ClientState diff --git a/kafka-streams/kafka-streams-static-partitioning/src/main/scala/com/twitter/finatra/streams/partitioning/internal/StaticPartitioningKafkaClientSupplierSupplier.scala b/kafka-streams/kafka-streams-static-partitioning/src/main/scala/com/twitter/finatra/kafkastreams/partitioning/internal/StaticPartitioningKafkaClientSupplierSupplier.scala similarity index 91% rename from kafka-streams/kafka-streams-static-partitioning/src/main/scala/com/twitter/finatra/streams/partitioning/internal/StaticPartitioningKafkaClientSupplierSupplier.scala rename to kafka-streams/kafka-streams-static-partitioning/src/main/scala/com/twitter/finatra/kafkastreams/partitioning/internal/StaticPartitioningKafkaClientSupplierSupplier.scala index 74d01618f0..1bf5720d7a 100644 --- a/kafka-streams/kafka-streams-static-partitioning/src/main/scala/com/twitter/finatra/streams/partitioning/internal/StaticPartitioningKafkaClientSupplierSupplier.scala +++ b/kafka-streams/kafka-streams-static-partitioning/src/main/scala/com/twitter/finatra/kafkastreams/partitioning/internal/StaticPartitioningKafkaClientSupplierSupplier.scala @@ -1,4 +1,4 @@ -package com.twitter.finatra.streams.partitioning.internal +package com.twitter.finatra.kafkastreams.partitioning.internal import java.util import org.apache.kafka.clients.consumer.{Consumer, ConsumerConfig} diff --git a/kafka-streams/kafka-streams-static-partitioning/src/main/scala/com/twitter/finatra/streams/partitioning/internal/StaticPartitioningStreamAssignor.scala b/kafka-streams/kafka-streams-static-partitioning/src/main/scala/com/twitter/finatra/kafkastreams/partitioning/internal/StaticPartitioningStreamAssignor.scala similarity index 97% rename from kafka-streams/kafka-streams-static-partitioning/src/main/scala/com/twitter/finatra/streams/partitioning/internal/StaticPartitioningStreamAssignor.scala rename to kafka-streams/kafka-streams-static-partitioning/src/main/scala/com/twitter/finatra/kafkastreams/partitioning/internal/StaticPartitioningStreamAssignor.scala index 8da58e1a15..2b5f2de279 100644 --- a/kafka-streams/kafka-streams-static-partitioning/src/main/scala/com/twitter/finatra/streams/partitioning/internal/StaticPartitioningStreamAssignor.scala +++ b/kafka-streams/kafka-streams-static-partitioning/src/main/scala/com/twitter/finatra/kafkastreams/partitioning/internal/StaticPartitioningStreamAssignor.scala @@ -1,4 +1,4 @@ -package com.twitter.finatra.streams.partitioning.internal +package com.twitter.finatra.kafkastreams.partitioning.internal import com.twitter.finatra.streams.queryable.thrift.domain.ServiceShardId import com.twitter.finatra.streams.queryable.thrift.partitioning.StaticServiceShardPartitioner diff --git a/kafka-streams/kafka-streams-static-partitioning/src/main/scala/com/twitter/finatra/streams/partitioning/internal/StaticTaskAssignor.scala b/kafka-streams/kafka-streams-static-partitioning/src/main/scala/com/twitter/finatra/kafkastreams/partitioning/internal/StaticTaskAssignor.scala similarity index 97% rename from kafka-streams/kafka-streams-static-partitioning/src/main/scala/com/twitter/finatra/streams/partitioning/internal/StaticTaskAssignor.scala rename to kafka-streams/kafka-streams-static-partitioning/src/main/scala/com/twitter/finatra/kafkastreams/partitioning/internal/StaticTaskAssignor.scala index 0a9964e0d4..8d08f85a16 100644 --- a/kafka-streams/kafka-streams-static-partitioning/src/main/scala/com/twitter/finatra/streams/partitioning/internal/StaticTaskAssignor.scala +++ b/kafka-streams/kafka-streams-static-partitioning/src/main/scala/com/twitter/finatra/kafkastreams/partitioning/internal/StaticTaskAssignor.scala @@ -1,12 +1,8 @@ -package com.twitter.finatra.streams.partitioning.internal +package com.twitter.finatra.kafkastreams.partitioning.internal import com.twitter.finagle.stats.LoadedStatsReceiver -import com.twitter.finatra.streams.partitioning.StaticPartitioning -import com.twitter.finatra.streams.queryable.thrift.domain.{ - KafkaGroupId, - KafkaPartitionId, - ServiceShardId -} +import com.twitter.finatra.kafkastreams.partitioning.StaticPartitioning +import com.twitter.finatra.streams.queryable.thrift.domain.{KafkaGroupId, KafkaPartitionId, ServiceShardId} import com.twitter.finatra.streams.queryable.thrift.partitioning.ServiceShardPartitioner import com.twitter.inject.Logging import org.apache.kafka.streams.processor.TaskId diff --git a/kafka-streams/kafka-streams-static-partitioning/src/main/scala/com/twitter/finatra/streams/partitioning/internal/TaskAssignments.scala b/kafka-streams/kafka-streams-static-partitioning/src/main/scala/com/twitter/finatra/kafkastreams/partitioning/internal/TaskAssignments.scala similarity index 88% rename from kafka-streams/kafka-streams-static-partitioning/src/main/scala/com/twitter/finatra/streams/partitioning/internal/TaskAssignments.scala rename to kafka-streams/kafka-streams-static-partitioning/src/main/scala/com/twitter/finatra/kafkastreams/partitioning/internal/TaskAssignments.scala index 4ce2943d0d..9a48fc3ebc 100644 --- a/kafka-streams/kafka-streams-static-partitioning/src/main/scala/com/twitter/finatra/streams/partitioning/internal/TaskAssignments.scala +++ b/kafka-streams/kafka-streams-static-partitioning/src/main/scala/com/twitter/finatra/kafkastreams/partitioning/internal/TaskAssignments.scala @@ -1,4 +1,4 @@ -package com.twitter.finatra.streams.partitioning.internal +package com.twitter.finatra.kafkastreams.partitioning.internal import org.apache.kafka.streams.processor.TaskId import scala.collection.mutable.ArrayBuffer diff --git a/kafka-streams/kafka-streams-static-partitioning/src/test/scala/org/apache/kafka/streams/processor/internals/assignment/StaticTaskAssignorTest.scala b/kafka-streams/kafka-streams-static-partitioning/src/test/scala/org/apache/kafka/streams/processor/internals/assignment/StaticTaskAssignorTest.scala index 34c6ed1ea9..8e8499f7ea 100644 --- a/kafka-streams/kafka-streams-static-partitioning/src/test/scala/org/apache/kafka/streams/processor/internals/assignment/StaticTaskAssignorTest.scala +++ b/kafka-streams/kafka-streams-static-partitioning/src/test/scala/org/apache/kafka/streams/processor/internals/assignment/StaticTaskAssignorTest.scala @@ -1,10 +1,7 @@ package org.apache.kafka.streams.processor.internals.assignment -import com.twitter.finatra.streams.partitioning.StaticPartitioning -import com.twitter.finatra.streams.partitioning.internal.{ - ClientStateAndHostInfo, - StaticTaskAssignor -} +import com.twitter.finatra.kafkastreams.partitioning.StaticPartitioning +import com.twitter.finatra.kafkastreams.partitioning.internal.{ClientStateAndHostInfo, StaticTaskAssignor} import com.twitter.finatra.streams.queryable.thrift.partitioning.StaticServiceShardPartitioner import com.twitter.inject.Test import java.util diff --git a/kafka-streams/kafka-streams/src/main/java/BUILD b/kafka-streams/kafka-streams/src/main/java/BUILD index 0e8b8664b7..1027df6be8 100644 --- a/kafka-streams/kafka-streams/src/main/java/BUILD +++ b/kafka-streams/kafka-streams/src/main/java/BUILD @@ -1,8 +1,6 @@ java_library( sources = rglobs( "com/twitter/finatra/kafkastreams/*.java", - "com/twitter/finatra/streams/*.java", - "org/*.java", ), compiler_option_sets = {}, provides = artifact( @@ -11,10 +9,11 @@ java_library( repo = artifactory, ), dependencies = [ - "3rdparty/jvm/org/agrona", "3rdparty/jvm/org/apache/kafka:kafka-clients", "3rdparty/jvm/org/apache/kafka:kafka-streams", ], exports = [ + "3rdparty/jvm/org/apache/kafka:kafka-clients", + "3rdparty/jvm/org/apache/kafka:kafka-streams", ], ) diff --git a/kafka-streams/kafka-streams/src/main/java/org/apache/kafka/streams/state/internals/InMemoryKeyValueFlushingLoggedStore.java b/kafka-streams/kafka-streams/src/main/java/org/apache/kafka/streams/state/internals/InMemoryKeyValueFlushingLoggedStore.java deleted file mode 100644 index 597c473fb7..0000000000 --- a/kafka-streams/kafka-streams/src/main/java/org/apache/kafka/streams/state/internals/InMemoryKeyValueFlushingLoggedStore.java +++ /dev/null @@ -1,154 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.kafka.streams.state.internals; - -// SUPPRESS CHECKSTYLE:OFF LineLength -// SUPPRESS CHECKSTYLE:OFF ModifierOrder -// SUPPRESS CHECKSTYLE:OFF OperatorWrap -// SUPPRESS CHECKSTYLE:OFF HiddenField -// SUPPRESS CHECKSTYLE:OFF NeedBraces -// SUPPRESS CHECKSTYLE:OFF NestedForDepth -// SUPPRESS CHECKSTYLE:OFF JavadocStyle -// SUPPRESS CHECKSTYLE:OFF NestedForDepth - -import java.util.List; - -import org.apache.kafka.common.serialization.Serde; -import org.apache.kafka.streams.KeyValue; -import org.apache.kafka.streams.processor.ProcessorContext; -import org.apache.kafka.streams.processor.StateStore; -import org.apache.kafka.streams.processor.internals.ProcessorStateManager; -import org.apache.kafka.streams.state.KeyValueIterator; -import org.apache.kafka.streams.state.KeyValueStore; -import org.apache.kafka.streams.state.StateSerdes; - -//Note: This class is copied from Kafka Streams InMemoryKeyValueLoggedStore with the only changes commented with "Twitter Changed" -public class InMemoryKeyValueFlushingLoggedStore extends WrappedStateStore.AbstractStateStore implements KeyValueStore { - - private final KeyValueStore inner; - private final Serde keySerde; - private final Serde valueSerde; - - // Twitter Changed - //private StoreChangeLogger changeLogger; - private StoreChangeFlushingLogger changeLogger; - - InMemoryKeyValueFlushingLoggedStore(final KeyValueStore inner, Serde keySerde, Serde valueSerde) { - super(inner); - this.inner = inner; - this.keySerde = keySerde; - this.valueSerde = valueSerde; - } - - @Override - @SuppressWarnings("unchecked") - public void init(ProcessorContext context, StateStore root) { - inner.init(context, root); - - // construct the serde - StateSerdes serdes = new StateSerdes<>( - ProcessorStateManager.storeChangelogTopic(context.applicationId(), inner.name()), - keySerde == null ? (Serde) context.keySerde() : keySerde, - valueSerde == null ? (Serde) context.valueSerde() : valueSerde); - - // Twitter Changed - //this.changeLogger = new StoreChangeLogger<>(inner.name(), context, serdes); - this.changeLogger = new StoreChangeFlushingLogger<>(inner.name(), context, serdes); - - // if the inner store is an LRU cache, add the eviction listener to log removed record - if (inner instanceof MemoryLRUCache) { - ((MemoryLRUCache) inner).whenEldestRemoved(new MemoryNavigableLRUCache.EldestEntryRemovalListener() { - @Override - public void apply(K key, V value) { - removed(key); - } - }); - } - } - - @Override - public long approximateNumEntries() { - return inner.approximateNumEntries(); - } - - @Override - public V get(K key) { - return this.inner.get(key); - } - - @Override - public void put(K key, V value) { - this.inner.put(key, value); - - changeLogger.logChange(key, value); - } - - @Override - public V putIfAbsent(K key, V value) { - V originalValue = this.inner.putIfAbsent(key, value); - if (originalValue == null) { - changeLogger.logChange(key, value); - } - return originalValue; - } - - @Override - public void putAll(List> entries) { - this.inner.putAll(entries); - - for (KeyValue entry : entries) { - K key = entry.key; - changeLogger.logChange(key, entry.value); - } - } - - @Override - public V delete(K key) { - V value = this.inner.delete(key); - - removed(key); - - return value; - } - - /** - * Called when the underlying {@link #inner} {@link KeyValueStore} removes an entry in response to a call from this - * store. - * - * @param key the key for the entry that the inner store removed - */ - protected void removed(K key) { - changeLogger.logChange(key, null); - } - - @Override - public KeyValueIterator range(K from, K to) { - return this.inner.range(from, to); - } - - @Override - public KeyValueIterator all() { - return this.inner.all(); - } - - @Override - //Twitter added - public void flush() { - changeLogger.flush(); - super.flush(); - } -} diff --git a/kafka-streams/kafka-streams/src/main/java/org/apache/kafka/streams/state/internals/StoreChangeFlushingLogger.java b/kafka-streams/kafka-streams/src/main/java/org/apache/kafka/streams/state/internals/StoreChangeFlushingLogger.java deleted file mode 100644 index ff5153422a..0000000000 --- a/kafka-streams/kafka-streams/src/main/java/org/apache/kafka/streams/state/internals/StoreChangeFlushingLogger.java +++ /dev/null @@ -1,153 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.kafka.streams.state.internals; - -// SUPPRESS CHECKSTYLE:OFF LineLength -// SUPPRESS CHECKSTYLE:OFF ModifierOrder -// SUPPRESS CHECKSTYLE:OFF OperatorWrap -// SUPPRESS CHECKSTYLE:OFF HiddenField -// SUPPRESS CHECKSTYLE:OFF NeedBraces -// SUPPRESS CHECKSTYLE:OFF NestedForDepth -// SUPPRESS CHECKSTYLE:OFF JavadocStyle -// SUPPRESS CHECKSTYLE:OFF NestedForDepth -// SUPPRESS CHECKSTYLE:OFF ConstantName - -import java.util.function.BiConsumer; - -import org.agrona.collections.Hashing; -import org.agrona.collections.Object2ObjectHashMap; -import org.apache.kafka.common.serialization.Serializer; -import org.apache.kafka.streams.processor.ProcessorContext; -import org.apache.kafka.streams.processor.TaskId; -import org.apache.kafka.streams.processor.internals.ProcessorStateManager; -import org.apache.kafka.streams.processor.internals.RecordCollector; -import org.apache.kafka.streams.state.StateSerdes; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -/** - * Note that the use of array-typed keys is discouraged because they result in incorrect caching behavior. - * If you intend to work on byte arrays as key, for example, you may want to wrap them with the {@code Bytes} class, - * i.e. use {@code RocksDBStore} rather than {@code RocksDBStore}. - * - * @param - * @param - */ -//See FlushingStores for motivations of this class -//Note: This class is copied from Kafka Streams StoreChangeLogger with the only changes commented with "Twitter Changed" -// The modifications provide "flushing" functionality which flushes the latest records for a given key to the changelog -// after every Kafka commit (which triggers the flush method below) -class StoreChangeFlushingLogger { - - protected final StateSerdes serialization; - - private final String topic; - private final int partition; - private final ProcessorContext context; - private final RecordCollector collector; - - // Twitter Changed - private static final Logger log = LoggerFactory.getLogger(StoreChangeFlushingLogger.class); - private final TaskId taskId; - private final Serializer keySerializer; - private final Serializer valueSerializer; - private final Object2ObjectHashMap> newEntries = new Object2ObjectHashMap<>(100000, Hashing.DEFAULT_LOAD_FACTOR); - - StoreChangeFlushingLogger(String storeName, ProcessorContext context, StateSerdes serialization) { - this(storeName, context, context.taskId().partition, serialization); - } - - private StoreChangeFlushingLogger(String storeName, ProcessorContext context, int partition, StateSerdes serialization) { - this.topic = ProcessorStateManager.storeChangelogTopic(context.applicationId(), storeName); - this.context = context; - this.partition = partition; - this.serialization = serialization; - this.collector = ((RecordCollector.Supplier) context).recordCollector(); - - // Twitter Added - this.taskId = context.taskId(); - this.keySerializer = serialization.keySerializer(); - this.valueSerializer = serialization.valueSerializer(); - } - - void logChange(final K key, final V value) { - if (collector != null) { - // Twitter Added - newEntries.put(key, new ValueAndTimestamp<>(value, context.timestamp())); - } - } - - // Twitter Changed - /* - * logChange now saves new entries into a map, which collapses entries using the same key. When flush - * is called, we send the latest collapsed entries to the changelog topic. By buffering entries - * before flush is called, we avoid writing every log change to the changelog topic. - * Pros: Less messages to and from changelog. Less broker side compaction needed. Bursts of changelog messages are better batched and compressed. - * Cons: Changelog messages are written to the changelog topic in bursts. - */ - void flush() { - if (!newEntries.isEmpty()) { - newEntries.forEach(foreachConsumer); - log.info("Task " + taskId + " flushed " + newEntries.size() + " entries into " + topic + "." + partition); - newEntries.clear(); - } - } - - private final BiConsumer> foreachConsumer = new BiConsumer>() { - @Override - public final void accept(K key, ValueAndTimestamp valueAndTimestamp) { - // Sending null headers to changelog topics (KIP-244) - collector.send( - topic, - key, - valueAndTimestamp.value, - null, - partition, - valueAndTimestamp.timestamp, - keySerializer, - valueSerializer); - } - }; - - class ValueAndTimestamp { - public final V value; - public final Long timestamp; - - ValueAndTimestamp(V value, Long timestamp) { - this.value = value; - this.timestamp = timestamp; - } - - @Override - public boolean equals(Object o) { - if (this == o) return true; - if (o == null || getClass() != o.getClass()) return false; - - ValueAndTimestamp that = (ValueAndTimestamp) o; - - if (value != null ? !value.equals(that.value) : that.value != null) return false; - return timestamp != null ? timestamp.equals(that.timestamp) : that.timestamp == null; - } - - @Override - public int hashCode() { - int result = value != null ? value.hashCode() : 0; - result = 31 * result + (timestamp != null ? timestamp.hashCode() : 0); - return result; - } - } -} diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/KafkaStreamsTwitterServer.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/KafkaStreamsTwitterServer.scala index ce0916036e..e6f49d06db 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/KafkaStreamsTwitterServer.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/KafkaStreamsTwitterServer.scala @@ -4,20 +4,17 @@ import com.twitter.app.Flag import com.twitter.conversions.DurationOps._ import com.twitter.conversions.StorageUnitOps._ import com.twitter.finagle.stats.StatsReceiver -import com.twitter.finatra.kafka.interceptors.{ - InstanceMetadataProducerInterceptor, - MonitoringConsumerInterceptor, - PublishTimeProducerInterceptor -} +import com.twitter.finatra.kafka.interceptors.{InstanceMetadataProducerInterceptor, MonitoringConsumerInterceptor, PublishTimeProducerInterceptor} import com.twitter.finatra.kafka.stats.KafkaFinagleMetricsReporter import com.twitter.finatra.kafkastreams.config.{FinatraRocksDBConfig, KafkaStreamsConfig} import com.twitter.finatra.kafkastreams.domain.ProcessingGuarantee -import com.twitter.finatra.kafkastreams.internal.ScalaStreamsImplicits +import com.twitter.finatra.kafkastreams.internal.admin.AdminRoutes +import com.twitter.finatra.kafkastreams.internal.interceptors.KafkaStreamsMonitoringConsumerInterceptor import com.twitter.finatra.kafkastreams.internal.listeners.FinatraStateRestoreListener import com.twitter.finatra.kafkastreams.internal.serde.AvoidDefaultSerde import com.twitter.finatra.kafkastreams.internal.stats.KafkaStreamsFinagleMetricsReporter -import com.twitter.finatra.kafkastreams.utils.KafkaFlagUtils -import com.twitter.finatra.streams.interceptors.KafkaStreamsMonitoringConsumerInterceptor +import com.twitter.finatra.kafkastreams.internal.utils.KafkaFlagUtils +import com.twitter.finatra.kafkastreams.utils.ScalaStreamsImplicits import com.twitter.inject.server.TwitterServer import com.twitter.util.Duration import java.util.Properties @@ -27,13 +24,7 @@ import org.apache.kafka.clients.consumer.{ConsumerConfig, OffsetResetStrategy} import org.apache.kafka.common.metrics.Sensor.RecordingLevel import org.apache.kafka.streams.KafkaStreams.{State, StateListener} import org.apache.kafka.streams.processor.internals.DefaultKafkaClientSupplier -import org.apache.kafka.streams.{ - KafkaClientSupplier, - KafkaStreams, - StreamsBuilder, - StreamsConfig, - Topology -} +import org.apache.kafka.streams.{KafkaClientSupplier, KafkaStreams, StreamsBuilder, StreamsConfig, Topology} /** * A [[com.twitter.server.TwitterServer]] that supports configuring a KafkaStreams topology. @@ -146,6 +137,7 @@ abstract class KafkaStreamsTwitterServer super.postInjectorStartup() properties = createKafkaStreamsProperties() topology = createKafkaStreamsTopology() + addAdminRoutes(AdminRoutes(properties, topology)) } override protected def postWarmup(): Unit = { diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/config/DefaultTopicConfig.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/config/DefaultTopicConfig.scala similarity index 95% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/config/DefaultTopicConfig.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/config/DefaultTopicConfig.scala index bb4de18d22..e7c8eaac31 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/config/DefaultTopicConfig.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/config/DefaultTopicConfig.scala @@ -1,7 +1,7 @@ -package com.twitter.finatra.streams.config +package com.twitter.finatra.kafkastreams.config -import com.twitter.conversions.StorageUnitOps._ import com.twitter.conversions.DurationOps._ +import com.twitter.conversions.StorageUnitOps._ import java.util import org.apache.kafka.common.config.TopicConfig.{ CLEANUP_POLICY_COMPACT, diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/config/FinatraRocksDBConfig.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/config/FinatraRocksDBConfig.scala index 52b1adbc4b..9696f04322 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/config/FinatraRocksDBConfig.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/config/FinatraRocksDBConfig.scala @@ -11,6 +11,7 @@ import org.apache.kafka.streams.state.RocksDBConfigSetter import org.rocksdb.{ BlockBasedTableConfig, BloomFilter, + ColumnFamilyOptionsInterface, CompactionStyle, CompressionType, InfoLogLevel, @@ -25,12 +26,199 @@ import org.rocksdb.{ object FinatraRocksDBConfig { val RocksDbBlockCacheSizeConfig = "rocksdb.block.cache.size" + val RocksDbBlockCacheSizeConfigDefault: StorageUnit = 200.megabytes + val RocksDbBlockCacheSizeConfigDoc = + """Size of the rocksdb block cache per task. We recommend that this should be about 1/3 of + |your total memory budget. The remaining free memory can be left for the OS page cache""".stripMargin + + val RocksDbBlockCacheShardBitsConfig = "rocksdb.block.cache.shard.bits" + val RocksDbBlockCacheShardBitsConfigDefault: Int = 1 + val RocksDbBlockCacheShardBitsConfigDoc = + """Cache is is sharded 2^bits shards by hash of the key. Setting the value to -1 will + |cause auto determine the size with starting size of 512KB. Shard bits will not exceed 6. + |If mutex locking is frequent and database size is smaller then RAM, increasing this value + |will improve locking as more shards will be available. + """.stripMargin + val RocksDbLZ4Config = "rocksdb.lz4" + val RocksDbLZ4ConfigDefault: Boolean = false + val RocksDbLZ4ConfigDoc = + "Enable RocksDB LZ4 compression. (See https://github.com/facebook/rocksdb/wiki/Compression)" + val RocksDbEnableStatistics = "rocksdb.statistics" + val RocksDbEnableStatisticsDefault: Boolean = false + val RocksDbEnableStatisticsDoc = + """Enable RocksDB statistics. Note: RocksDB Statistics could add 5-10% degradation in performance + |(See https://github.com/facebook/rocksdb/wiki/Statistics)""".stripMargin + val RocksDbStatCollectionPeriodMs = "rocksdb.statistics.collection.period.ms" + val RocksDbStatCollectionPeriodMsDefault: Int = 60000 + val RocksDbStatCollectionPeriodMsDoc = "Set the period in milliseconds for stats collection." + + val RocksDbInfoLogLevel = "rocksdb.log.info.level" + val RocksDbInfoLogLevelDefault = "DEBUG_LEVEL" + val RocksDbInfoLogLevelDoc = + """Level of logging for rocksdb LOG file. + |DEBUG_LEVEL, INFO_LEVEL, WARN_LEVEL, ERROR_LEVEL, FATAL_LEVEL, HEADER_LEVEL""".stripMargin + + val RocksDbMaxLogFileSize = "rocksdb.log.max.file.size" + val RocksDbMaxLogFileSizeDefault: StorageUnit = 50.megabytes + val RocksDbMaxLogFileSizeDoc = + s"""Specify the maximal size of the info log file. If the log file is larger then + |"rocksdb.log.keep.file.num" a new log file will be created.""".stripMargin + + val RocksDbKeepLogFileNum = "rocksdb.log.keep.file.num" + val RocksDbKeepLogFileNumDefault: Int = 10 + val RocksDbKeepLogFileNumDoc = "Maximal info log files to be kept." + + val RocksDbCacheIndexAndFilterBlocks = "rocksdb.cache.index.and.filter.blocks" + val RocksDbCacheIndexAndFilterBlocksDefault: Boolean = true + val RocksDbCacheIndexAndFilterBlocksDoc = + """Store index and filter blocks into the block cache. This bounds the memory usage, + | which is desirable when running in a container. + |(See https://github.com/facebook/rocksdb/wiki/Memory-usage-in-RocksDB#indexes-and-filter-blocks)""".stripMargin + + val RocksDbCachePinL0IndexAndFilterBlocks = "rocksdb.cache.pin.l0.index.and.filter.blocks" + val RocksDbCachePinL0IndexAndFilterBlocksDefault: Boolean = true + val RocksDbCachePinL0IndexAndFilterBlocksDoc = + """Pin level-0 file's index and filter blocks in block cache, to avoid them from being evicted. + | This setting is generally recommended to be turned on along to minimize the negative + | performance impact resulted by turning on RocksDbCacheIndexAndFilterBlocks. + |(See https://github.com/facebook/rocksdb/wiki/Block-Cache#caching-index-and-filter-blocks)""".stripMargin + + val RocksDbTableConfigBlockSize = "rocksdb.tableconfig.block.size" + val RocksDbTableConfigBlockSizeDefault: StorageUnit = (16 * 1024).bytes + val RocksDbTableConfigBlockSizeDoc = + s"""Approximate size of user data packed per block. This is the uncompressed size and on disk + |size will differ due to compression. Increasing block_size decreases memory usage and space + |amplification, but increases read amplification.""".stripMargin + + val RocksDbTableConfigBoomFilterKeyBits = "rocksdb.tableconfig.bloomfilter.key.bits" + val RocksDbTableConfigBoomFilterKeyBitsDefault: Int = 10 + val RocksDbTableConfigBoomFilterKeyBitsDoc = + """ + |Bits per key in bloom filter. A bits_per_key if 10, yields a filter with ~ 1% false positive + |rate. + |(See https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide#bloom-filters)""".stripMargin + + val RocksDbTableConfigBoomFilterMode = "rocksdb.tableconfig.bloomfilter.mode" + val RocksDbTableConfigBoomFilterModeDefault: Boolean = true + val RocksDbTableConfigBoomFilterModeDoc = + s"""Toggle the mode of the bloom filer between Block-based filter (true) and Full filter (false). + |Block-based filter is a filter for each block where Full filter is a filter per file. + |If multiple keys are contained in the same file, the Block-based filter will serve best. + |If keys are same among database files then Full filter is best.""".stripMargin + + val RocksDbDatabaseWriteBufferSize = "rocksdb.db.write.buffer.size" + val RocksDbDatabaseWriteBufferSizeDefault: StorageUnit = 0.bytes + val RocksDbDatabaseWriteBufferSizeDoc = + """Data stored in memtables across all column families before writing to disk. Disabled by + |specifying a 0 value, can be enabled by setting positive value in bytes. This value can be + |used to control the total memtable sizes.""".stripMargin + + val RocksDbWriteBufferSize = "rocksdb.write.buffer.size" + val RocksDbWriteBufferSizeDefault: StorageUnit = 1.gigabyte + val RocksDbWriteBufferSizeDoc = + """Data stored in memory (stored in unsorted log on disk) before writing tto sorted on-disk + |file. Larger values will increase performance, especially on bulk loads up to + |max_write_buffer_number write buffers available. This value can be used to adjust the control + |of memory usage. Larger write buffers will cause longer recovery on file open.""".stripMargin + + val RocksDbMinWriteBufferNumberToMerge = "rocksdb.min.write.buffer.num.merge" + val RocksDbMinWriteBufferNumberToMergeDefault: Int = 1 + val RocksDbMinWriteBufferNumberToMergeDoc = + """Minimum number of write buffers that will be merged together before flushing to storage. + |Setting of 1 will cause L0 flushed as individual files and increase read amplification + |as all files will be scanned.""".stripMargin + + val RocksDbMaxWriteBufferNumber = "rocksdb.max.write.buffer.num" + val RocksDbMaxWriteBufferNumberDefault: Int = 2 + val RocksDbMaxWriteBufferNumberDoc = + """Maximum number of write buffers that will be stored in memory. While 1 buffer is flushed to disk + |other buffers can be written.""".stripMargin + + val RocksDbBytesPerSync = "rocksdb.bytes.per.sync" + val RocksDbBytesPerSyncDefault: StorageUnit = 1048576.bytes + val RocksDbBytesPerSyncDoc = + "Setting for OS to sync files to disk in the background while they are written." + + val RocksDbMaxBackgroundCompactions = "rocksdb.max.background.compactions" + val RocksDbMaxBackgroundCompactionsDefault: Int = 4 + val RocksDbMaxBackgroundCompactionsDoc = + """Maximum background compactions, increased values will fully utilize CPU and storage for + |compaction routines. If stats indication higher latency due to compaction, this value could + |be adjusted. + |(https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide#parallelism-options)""".stripMargin + + val RocksDbMaxBackgroundFlushes = "rocksdb.max.background.flushes" + val RocksDbMaxBackgroundFlushesDefault: Int = 2 + val RocksDbMaxBackgroundFlushesDoc = + """Maximum number of concurrent background flushes. + |(https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide#parallelism-options) + """.stripMargin + + val RocksDbIncreaseParallelism = "rocksdb.parallelism" + def RocksDbIncreaseParallelismDefault(): Int = numProcs().toInt + val RocksDbIncreaseParallelismDoc = + """Increases the total number of threads used for flushes and compaction. If rocks seems to be + |an indication of bottleneck, this is a value you want to increase for addressing that.""".stripMargin + + val RocksDbInplaceUpdateSupport = "rocksdb.inplace.update.support" + val RocksDbInplaceUpdateSupportDefault: Boolean = true + val RocksDbInplaceUpdateSupportDoc = + """Enables thread safe updates in place. If true point-in-time consistency using snapshot/iterator + |will not be possible. Set this to true if not using snapshot iterators, otherwise false.""".stripMargin + + val RocksDbAllowConcurrentMemtableWrite = "rocksdb.allow.concurrent.memtable.write" + val RocksDbAllowConcurrentMemtableWriteDefault: Boolean = false + val RocksDbAllowConcurrentMemtableWriteDoc = + """Set true if multiple writers to modify memtables in parallel. This flag is not compatible + |with inplace update support or filter deletes, default should be false unless memtable used + |supports it.""".stripMargin + + val RocksDbEnableWriteThreadAdaptiveYield = "rocksdb.enable.write.thread.adaptive.yield" + val RocksDbEnableWriteThreadAdaptiveYieldDefault: Boolean = false + val RocksDbEnableWriteThreadAdaptiveYieldDoc = + """Set true to enable thread synchronizing with write batch group leader. Concurrent workloads + |can be improved by setting to true.""".stripMargin + + val RocksDbCompactionStyle = "rocksdb.compaction.style" + val RocksDbCompactionStyleDefault = "UNIVERSAL" + val RocksDbCompactionStyleDoc = + """Set compaction style for database. + |UNIVERSAL, LEVEL, FIFO.""".stripMargin + + val RocksDbCompactionStyleOptimize = "rocksdb.compaction.style.optimize" + val RocksDbCompactionStyleOptimizeDefault = true + val RocksDbCompactionStyleOptimizeDoc = + s"""Heavy workloads and big datasets are not the default mode of operation for rocksdb databases. + |Enabling optimization will use rocksdb internal configuration for a range of values calculated. + |The values calculated are based on the flag value "rocksdb.compaction.style.memtable.budget" + |for memory given to optimize performance. Generally this should be true but means other settings + |values might be different from values specified on the commandline. + |(See https://github.com/facebook/rocksdb/blob/master/options/options.cc)""".stripMargin + + val RocksDbMaxBytesForLevelBase = "rocksdb.max.bytes.for.level.base" + val RocksDbMaxBytesForLevelBaseDefault: StorageUnit = 1.gigabyte + val RocksDbMaxBytesForLevelBaseDoc = + """Total size of level 1, should be about the same size as level 0. Lowering this value + |can help control memory usage.""".stripMargin - // BlockCache to be shared by all RocksDB instances created on this instance. Note: That a single Kafka Streams instance may get multiple tasks assigned to it - // and each stateful task will have a separate RocksDB instance created. This cache will be shared across all the tasks. + val RocksDbLevelCompactionDynamicLevelBytes = "rocksdb.level.compaction.dynamic.level.bytes" + val RocksDbLevelCompactionDynamicLevelBytesDefault: Boolean = true + val RocksDbLevelCompactionDynamicLevelBytesDoc = + """If true, enables rockdb to pick target size for each level dynamically.""".stripMargin + + val RocksDbCompactionStyleMemtableBudget = "rocksdb.compaction.style.memtable.budget" + val RocksDbCompactionStyleMemtableBudgetDefault: StorageUnit = + ColumnFamilyOptionsInterface.DEFAULT_COMPACTION_MEMTABLE_MEMORY_BUDGET.bytes + val RocksDbCompactionStyleMemtableBudgetDoc = + s"""Memory budget in bytes used when "rocksdb.compaction.style.optimize" is true.""" + + // BlockCache to be shared by all RocksDB instances created on this instance. + // Note: That a single Kafka Streams instance may get multiple tasks assigned to it + // and each stateful task will have a separate RocksDB instance created. + // This cache will be shared across all the tasks. // See: https://github.com/facebook/rocksdb/wiki/Block-Cache private var SharedBlockCache: LRUCache = _ @@ -41,6 +229,9 @@ object FinatraRocksDBConfig { } } +/** + * Maintains the RocksDB configuration used by Kafka Streams. + */ class FinatraRocksDBConfig extends RocksDBConfigSetter with Logging { //See https://github.com/facebook/rocksdb/wiki/Setup-Options-and-Basic-Tuning#other-general-options @@ -49,62 +240,254 @@ class FinatraRocksDBConfig extends RocksDBConfigSetter with Logging { options: Options, configs: util.Map[String, AnyRef] ): Unit = { + setTableConfiguration(options, configs) + setWriteBufferConfiguration(options, configs) + setOperatingSystemProcessConfiguration(options, configs) + setDatabaseConcurrency(options, configs) + setCompactionConfiguration(options, configs) + setCompression(options, configs) + setInformationLoggingLevel(options, configs) + setStatisticsOptions(options, configs) + } + + private def setWriteBufferConfiguration( + options: Options, + configs: util.Map[String, AnyRef] + ): Unit = { + val dbWriteBufferSize = getBytesOrDefault( + configs, + FinatraRocksDBConfig.RocksDbDatabaseWriteBufferSize, + FinatraRocksDBConfig.RocksDbDatabaseWriteBufferSizeDefault) + + val writeBufferSize = getBytesOrDefault( + configs, + FinatraRocksDBConfig.RocksDbWriteBufferSize, + FinatraRocksDBConfig.RocksDbWriteBufferSizeDefault) + + val minWriteBufferNumberToMerge = getIntOrDefault( + configs, + FinatraRocksDBConfig.RocksDbMinWriteBufferNumberToMerge, + FinatraRocksDBConfig.RocksDbMinWriteBufferNumberToMergeDefault) + + val maxWriteBufferNumber = getIntOrDefault( + configs, + FinatraRocksDBConfig.RocksDbMaxWriteBufferNumber, + FinatraRocksDBConfig.RocksDbMaxWriteBufferNumberDefault) + + options + .setDbWriteBufferSize(dbWriteBufferSize) + .setWriteBufferSize(writeBufferSize) + .setMinWriteBufferNumberToMerge(minWriteBufferNumberToMerge) + .setMaxWriteBufferNumber(maxWriteBufferNumber) + } + + private def setTableConfiguration(options: Options, configs: util.Map[String, AnyRef]): Unit = { if (FinatraRocksDBConfig.SharedBlockCache == null) { val blockCacheSize = - getBytesOrDefault(configs, FinatraRocksDBConfig.RocksDbBlockCacheSizeConfig, 100.megabytes) - val numShardBits = 1 //TODO: Make configurable so this can be increased for multi-threaded queryable state access + getBytesOrDefault( + configs, + FinatraRocksDBConfig.RocksDbBlockCacheSizeConfig, + FinatraRocksDBConfig.RocksDbBlockCacheSizeConfigDefault) + val numShardBits = getIntOrDefault( + configs, + FinatraRocksDBConfig.RocksDbBlockCacheShardBitsConfig, + FinatraRocksDBConfig.RocksDbBlockCacheShardBitsConfigDefault) FinatraRocksDBConfig.SharedBlockCache = new LRUCache(blockCacheSize, numShardBits) } val tableConfig = new BlockBasedTableConfig - tableConfig.setBlockSize(16 * 1024) + + val blockSize = getBytesOrDefault( + configs, + FinatraRocksDBConfig.RocksDbTableConfigBlockSize, + FinatraRocksDBConfig.RocksDbTableConfigBlockSizeDefault) + + val bitsPerKey = getIntOrDefault( + configs, + FinatraRocksDBConfig.RocksDbTableConfigBoomFilterKeyBits, + FinatraRocksDBConfig.RocksDbTableConfigBoomFilterKeyBitsDefault) + + val useBlockBasedMode = getBooleanOrDefault( + configs, + FinatraRocksDBConfig.RocksDbTableConfigBoomFilterMode, + FinatraRocksDBConfig.RocksDbTableConfigBoomFilterModeDefault) + + val cacheIndexAndFilterBlocks = getBooleanOrDefault( + configs, + FinatraRocksDBConfig.RocksDbCacheIndexAndFilterBlocks, + FinatraRocksDBConfig.RocksDbCacheIndexAndFilterBlocksDefault) + + val cachePinL0IndexAndFilterBlocks = getBooleanOrDefault( + configs, + FinatraRocksDBConfig.RocksDbCachePinL0IndexAndFilterBlocks, + FinatraRocksDBConfig.RocksDbCachePinL0IndexAndFilterBlocksDefault) + + tableConfig.setBlockSize(blockSize) tableConfig.setBlockCache(FinatraRocksDBConfig.SharedBlockCache) - tableConfig.setFilter(new BloomFilter(10)) + tableConfig.setFilter(new BloomFilter(bitsPerKey, useBlockBasedMode)) + tableConfig.setCacheIndexAndFilterBlocks(cacheIndexAndFilterBlocks) + tableConfig.setPinL0FilterAndIndexBlocksInCache(cachePinL0IndexAndFilterBlocks) + options .setTableFormatConfig(tableConfig) + } - options - .setDbWriteBufferSize(0) - .setWriteBufferSize(1.gigabyte.inBytes) //TODO: Make configurable with default value equal to RocksDB default (which is much lower than 1 GB!) - .setMinWriteBufferNumberToMerge(1) - .setMaxWriteBufferNumber(2) + private def setOperatingSystemProcessConfiguration( + options: Options, + configs: util.Map[String, AnyRef] + ): Unit = { + + val bytesPerSync = getBytesOrDefault( + configs, + FinatraRocksDBConfig.RocksDbBytesPerSync, + FinatraRocksDBConfig.RocksDbBytesPerSyncDefault) + + val maxBackgroundCompactions = getIntOrDefault( + configs, + FinatraRocksDBConfig.RocksDbMaxBackgroundCompactions, + FinatraRocksDBConfig.RocksDbMaxBackgroundCompactionsDefault) + + val maxBackgroundFlushes = getIntOrDefault( + configs, + FinatraRocksDBConfig.RocksDbMaxBackgroundFlushes, + FinatraRocksDBConfig.RocksDbMaxBackgroundFlushesDefault) + + val increaseParallelism = getIntOrDefault( + configs, + FinatraRocksDBConfig.RocksDbIncreaseParallelism, + FinatraRocksDBConfig.RocksDbIncreaseParallelismDefault()) options - .setBytesPerSync(1048576) //See: https://github.com/facebook/rocksdb/wiki/Setup-Options-and-Basic-Tuning#other-general-options - .setMaxBackgroundCompactions(4) - .setMaxBackgroundFlushes(2) - .setIncreaseParallelism(Math.max(numProcs().toInt, 2)) - - /* From the docs: "Allows thread-safe inplace updates. If this is true, there is no way to - achieve point-in-time consistency using snapshot or iterator (assuming concurrent updates). - Hence iterator and multi-get will return results which are not consistent as of any point-in-time." */ + .setBytesPerSync(bytesPerSync) + .setMaxBackgroundCompactions(maxBackgroundCompactions) + .setMaxBackgroundFlushes(maxBackgroundFlushes) + .setIncreaseParallelism(Math.max(increaseParallelism, 2)) + } + + private def setDatabaseConcurrency(options: Options, configs: util.Map[String, AnyRef]): Unit = { + val inplaceUpdateSupport = getBooleanOrDefault( + configs, + FinatraRocksDBConfig.RocksDbInplaceUpdateSupport, + FinatraRocksDBConfig.RocksDbInplaceUpdateSupportDefault) + + val allowConcurrentMemtableWrite = getBooleanOrDefault( + configs, + FinatraRocksDBConfig.RocksDbAllowConcurrentMemtableWrite, + FinatraRocksDBConfig.RocksDbAllowConcurrentMemtableWriteDefault) + + val enableWriteThreadAdaptiveYield = getBooleanOrDefault( + configs, + FinatraRocksDBConfig.RocksDbEnableWriteThreadAdaptiveYield, + FinatraRocksDBConfig.RocksDbEnableWriteThreadAdaptiveYieldDefault) + options - .setInplaceUpdateSupport(true) //We set to true since we never have concurrent updates - .setAllowConcurrentMemtableWrite(false) - .setEnableWriteThreadAdaptiveYield(false) + .setInplaceUpdateSupport(inplaceUpdateSupport) + .setAllowConcurrentMemtableWrite(allowConcurrentMemtableWrite) + .setEnableWriteThreadAdaptiveYield(enableWriteThreadAdaptiveYield) + } + + private def setCompactionConfiguration( + options: Options, + configs: util.Map[String, AnyRef] + ): Unit = { + val compactionStyle = CompactionStyle.valueOf(getStringOrDefault( + configs, + FinatraRocksDBConfig.RocksDbCompactionStyle, + FinatraRocksDBConfig.RocksDbCompactionStyleDefault).toUpperCase) + + val compactionStyleOptimize = getBooleanOrDefault( + configs, + FinatraRocksDBConfig.RocksDbCompactionStyleOptimize, + FinatraRocksDBConfig.RocksDbCompactionStyleOptimizeDefault) + + val maxBytesForLevelBase = getBytesOrDefault( + configs, + FinatraRocksDBConfig.RocksDbMaxBytesForLevelBase, + FinatraRocksDBConfig.RocksDbMaxBytesForLevelBaseDefault) + + val levelCompactionDynamicLevelBytes = getBooleanOrDefault( + configs, + FinatraRocksDBConfig.RocksDbLevelCompactionDynamicLevelBytes, + FinatraRocksDBConfig.RocksDbLevelCompactionDynamicLevelBytesDefault) + + val optimizeWithMemtableMemoryBudget = getBytesOrDefault( + configs, + FinatraRocksDBConfig.RocksDbCompactionStyleMemtableBudget, + FinatraRocksDBConfig.RocksDbCompactionStyleMemtableBudgetDefault) options - .setCompactionStyle(CompactionStyle.UNIVERSAL) - .setMaxBytesForLevelBase(1.gigabyte.inBytes) - .setLevelCompactionDynamicLevelBytes(true) - .optimizeUniversalStyleCompaction() + .setCompactionStyle(compactionStyle) + .setMaxBytesForLevelBase(maxBytesForLevelBase) + .setLevelCompactionDynamicLevelBytes(levelCompactionDynamicLevelBytes) - if (configs.get(FinatraRocksDBConfig.RocksDbLZ4Config) == "true") { + compactionStyle match { + case CompactionStyle.UNIVERSAL if compactionStyleOptimize => + options + .optimizeUniversalStyleCompaction(optimizeWithMemtableMemoryBudget) + case CompactionStyle.LEVEL if compactionStyleOptimize => + options + .optimizeLevelStyleCompaction(optimizeWithMemtableMemoryBudget) + case _ => + } + } + + private def setCompression(options: Options, configs: util.Map[String, AnyRef]): Unit = { + val lz4Config = getBooleanOrDefault( + configs, + FinatraRocksDBConfig.RocksDbLZ4Config, + FinatraRocksDBConfig.RocksDbLZ4ConfigDefault + ) + + if (lz4Config) { options.setCompressionType(CompressionType.LZ4_COMPRESSION) } + } + + private def setInformationLoggingLevel( + options: Options, + configs: util.Map[String, AnyRef] + ): Unit = { + val infoLogLevel = InfoLogLevel.valueOf(getStringOrDefault( + configs, + FinatraRocksDBConfig.RocksDbInfoLogLevel, + FinatraRocksDBConfig.RocksDbInfoLogLevelDefault).toUpperCase) options - .setInfoLogLevel(InfoLogLevel.DEBUG_LEVEL) + .setInfoLogLevel(infoLogLevel) + + } + + private def setStatisticsOptions(options: Options, configs: util.Map[String, AnyRef]): Unit = { + val maxLogFileSize = getBytesOrDefault( + configs, + FinatraRocksDBConfig.RocksDbMaxLogFileSize, + FinatraRocksDBConfig.RocksDbMaxLogFileSizeDefault) + + options.setMaxLogFileSize(maxLogFileSize) + + val keepLogFileNum = getIntOrDefault( + configs, + FinatraRocksDBConfig.RocksDbKeepLogFileNum, + FinatraRocksDBConfig.RocksDbKeepLogFileNumDefault) + options.setKeepLogFileNum(keepLogFileNum) + + val enableStatistics = getBooleanOrDefault( + configs, + FinatraRocksDBConfig.RocksDbEnableStatistics, + FinatraRocksDBConfig.RocksDbEnableStatisticsDefault) - if (configs.get(FinatraRocksDBConfig.RocksDbEnableStatistics) == "true") { + if (enableStatistics) { val statistics = new Statistics val statsCallback = new RocksDBStatsCallback(FinatraRocksDBConfig.globalStatsReceiver) val statsCollectorInput = new StatsCollectorInput(statistics, statsCallback) val statsCollector = new StatisticsCollector( util.Arrays.asList(statsCollectorInput), - getIntOrDefault(configs, FinatraRocksDBConfig.RocksDbStatCollectionPeriodMs, 60000) - ) + getIntOrDefault( + configs, + FinatraRocksDBConfig.RocksDbStatCollectionPeriodMs, + FinatraRocksDBConfig.RocksDbStatCollectionPeriodMsDefault)) + statsCollector.start() statistics.setStatsLevel(StatsLevel.ALL) @@ -135,4 +518,29 @@ class FinatraRocksDBConfig extends RocksDBConfigSetter with Logging { default } } + + private def getStringOrDefault( + configs: util.Map[String, AnyRef], + key: String, default: String + ): String = { + val valueString = configs.get(key) + if (valueString != null) { + valueString.toString + } else { + default + } + } + + private def getBooleanOrDefault( + configs: util.Map[String, AnyRef], + key: String, + default: Boolean + ): Boolean = { + val valueString = configs.get(key) + if (valueString != null) { + valueString.toString.toBoolean + } else { + default + } + } } diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/flags/FinatraTransformerFlags.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/config/FinatraTransformerFlags.scala similarity index 89% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/flags/FinatraTransformerFlags.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/config/FinatraTransformerFlags.scala index f0dbd2fee6..2c505878bf 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/flags/FinatraTransformerFlags.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/config/FinatraTransformerFlags.scala @@ -1,8 +1,8 @@ -package com.twitter.finatra.streams.flags +package com.twitter.finatra.kafkastreams.config import com.twitter.conversions.DurationOps._ import com.twitter.finatra.kafkastreams.KafkaStreamsTwitterServer -import com.twitter.finatra.streams.flags.FinatraTransformerFlags._ +import com.twitter.finatra.kafkastreams.config.FinatraTransformerFlags._ object FinatraTransformerFlags { val AutoWatermarkInterval = "finatra.streams.watermarks.auto.interval" diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/config/RocksDbFlags.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/config/RocksDbFlags.scala new file mode 100644 index 0000000000..54a0decc14 --- /dev/null +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/config/RocksDbFlags.scala @@ -0,0 +1,211 @@ +package com.twitter.finatra.kafkastreams.config + +import com.twitter.app.Flag +import com.twitter.inject.server.TwitterServer +import com.twitter.util.StorageUnit + +trait RocksDbFlags extends TwitterServer { + + protected val rocksDbCountsStoreBlockCacheSize: Flag[StorageUnit] = + flag( + name = FinatraRocksDBConfig.RocksDbBlockCacheSizeConfig, + default = FinatraRocksDBConfig.RocksDbBlockCacheSizeConfigDefault, + help = FinatraRocksDBConfig.RocksDbBlockCacheSizeConfigDoc + ) + + protected val rocksDbBlockCacheShardBitsConfig: Flag[Int] = + flag( + name = FinatraRocksDBConfig.RocksDbBlockCacheShardBitsConfig, + default = FinatraRocksDBConfig.RocksDbBlockCacheShardBitsConfigDefault, + help = FinatraRocksDBConfig.RocksDbBlockCacheShardBitsConfigDoc + ) + + protected val rocksDbEnableStatistics: Flag[Boolean] = + flag( + name = FinatraRocksDBConfig.RocksDbEnableStatistics, + default = FinatraRocksDBConfig.RocksDbEnableStatisticsDefault, + help = FinatraRocksDBConfig.RocksDbEnableStatisticsDoc + ) + + protected val rocksDbStatCollectionPeriodMs: Flag[Int] = + flag( + name = FinatraRocksDBConfig.RocksDbStatCollectionPeriodMs, + default = FinatraRocksDBConfig.RocksDbStatCollectionPeriodMsDefault, + help = FinatraRocksDBConfig.RocksDbStatCollectionPeriodMsDoc + ) + + protected val rocksDbEnableLZ4: Flag[Boolean] = + flag( + name = FinatraRocksDBConfig.RocksDbLZ4Config, + default = FinatraRocksDBConfig.RocksDbLZ4ConfigDefault, + help = FinatraRocksDBConfig.RocksDbLZ4ConfigDoc + ) + + protected val rocksDbInfoLogLevel: Flag[String] = + flag( + name = FinatraRocksDBConfig.RocksDbInfoLogLevel, + default = FinatraRocksDBConfig.RocksDbInfoLogLevelDefault, + help = FinatraRocksDBConfig.RocksDbInfoLogLevelDoc + ) + + protected val rocksDbMaxLogFileSize: Flag[StorageUnit] = + flag( + name = FinatraRocksDBConfig.RocksDbMaxLogFileSize, + default = FinatraRocksDBConfig.RocksDbMaxLogFileSizeDefault, + help = FinatraRocksDBConfig.RocksDbMaxLogFileSizeDoc + ) + + protected val rocksDbKeepLogFileNum: Flag[Int] = + flag( + name = FinatraRocksDBConfig.RocksDbKeepLogFileNum, + default = FinatraRocksDBConfig.RocksDbKeepLogFileNumDefault, + help = FinatraRocksDBConfig.RocksDbKeepLogFileNumDoc + ) + + protected val rocksDbCacheIndexAndFilterBlocks: Flag[Boolean] = + flag( + name = FinatraRocksDBConfig.RocksDbCacheIndexAndFilterBlocks, + default = FinatraRocksDBConfig.RocksDbCacheIndexAndFilterBlocksDefault, + help = FinatraRocksDBConfig.RocksDbCacheIndexAndFilterBlocksDoc + ) + + protected val rocksDbCachePinL0IndexAndFilterBlocks: Flag[Boolean] = + flag( + name = FinatraRocksDBConfig.RocksDbCachePinL0IndexAndFilterBlocks, + default = FinatraRocksDBConfig.RocksDbCachePinL0IndexAndFilterBlocksDefault, + help = FinatraRocksDBConfig.RocksDbCachePinL0IndexAndFilterBlocksDoc + ) + + protected val rocksDbTableConfigBlockSize: Flag[StorageUnit] = + flag( + name = FinatraRocksDBConfig.RocksDbTableConfigBlockSize, + default = FinatraRocksDBConfig.RocksDbTableConfigBlockSizeDefault, + help = FinatraRocksDBConfig.RocksDbTableConfigBlockSizeDoc + ) + + protected val rocksDbTableConfigBoomFilterKeyBits: Flag[Int] = + flag( + name = FinatraRocksDBConfig.RocksDbTableConfigBoomFilterKeyBits, + default = FinatraRocksDBConfig.RocksDbTableConfigBoomFilterKeyBitsDefault, + help = FinatraRocksDBConfig.RocksDbTableConfigBoomFilterKeyBitsDoc + ) + + protected val rocksDbTableConfigBoomFilterMode: Flag[Boolean] = + flag( + name = FinatraRocksDBConfig.RocksDbTableConfigBoomFilterMode, + default = FinatraRocksDBConfig.RocksDbTableConfigBoomFilterModeDefault, + help = FinatraRocksDBConfig.RocksDbTableConfigBoomFilterModeDoc + ) + + protected val rocksDbDatabaseWriteBufferSize: Flag[StorageUnit] = + flag( + name = FinatraRocksDBConfig.RocksDbDatabaseWriteBufferSize, + default = FinatraRocksDBConfig.RocksDbDatabaseWriteBufferSizeDefault, + help = FinatraRocksDBConfig.RocksDbDatabaseWriteBufferSizeDoc + ) + + protected val rocksDbWriteBufferSize: Flag[StorageUnit] = + flag( + name = FinatraRocksDBConfig.RocksDbWriteBufferSize, + default = FinatraRocksDBConfig.RocksDbWriteBufferSizeDefault, + help = FinatraRocksDBConfig.RocksDbWriteBufferSizeDoc + ) + + protected val rocksDbMinWriteBufferNumberToMerge: Flag[Int] = + flag( + name = FinatraRocksDBConfig.RocksDbMinWriteBufferNumberToMerge, + default = FinatraRocksDBConfig.RocksDbMinWriteBufferNumberToMergeDefault, + help = FinatraRocksDBConfig.RocksDbMinWriteBufferNumberToMergeDoc + ) + + protected val rocksDbMaxWriteBufferNumber: Flag[Int] = + flag( + name = FinatraRocksDBConfig.RocksDbMaxWriteBufferNumber, + default = FinatraRocksDBConfig.RocksDbMaxWriteBufferNumberDefault, + help = FinatraRocksDBConfig.RocksDbMaxWriteBufferNumberDoc + ) + + protected val rocksDbBytesPerSync: Flag[StorageUnit] = + flag( + name = FinatraRocksDBConfig.RocksDbBytesPerSync, + default = FinatraRocksDBConfig.RocksDbBytesPerSyncDefault, + help = FinatraRocksDBConfig.RocksDbBytesPerSyncDoc + ) + + protected val rocksDbMaxBackgroundCompactions: Flag[Int] = + flag( + name = FinatraRocksDBConfig.RocksDbMaxBackgroundCompactions, + default = FinatraRocksDBConfig.RocksDbMaxBackgroundCompactionsDefault, + help = FinatraRocksDBConfig.RocksDbMaxBackgroundCompactionsDoc + ) + + protected val rocksDbMaxBackgroundFlushes: Flag[Int] = + flag( + name = FinatraRocksDBConfig.RocksDbMaxBackgroundFlushes, + default = FinatraRocksDBConfig.RocksDbMaxBackgroundFlushesDefault, + help = FinatraRocksDBConfig.RocksDbMaxBackgroundFlushesDoc + ) + + protected val rocksDbIncreaseParallelism: Flag[Int] = + flag( + name = FinatraRocksDBConfig.RocksDbIncreaseParallelism, + default = FinatraRocksDBConfig.RocksDbIncreaseParallelismDefault(), + help = FinatraRocksDBConfig.RocksDbIncreaseParallelismDoc + ) + + protected val rocksDbInplaceUpdateSupport: Flag[Boolean] = + flag( + name = FinatraRocksDBConfig.RocksDbInplaceUpdateSupport, + default = FinatraRocksDBConfig.RocksDbInplaceUpdateSupportDefault, + help = FinatraRocksDBConfig.RocksDbInplaceUpdateSupportDoc + ) + + protected val rocksDbAllowConcurrentMemtableWrite: Flag[Boolean] = + flag( + name = FinatraRocksDBConfig.RocksDbAllowConcurrentMemtableWrite, + default = FinatraRocksDBConfig.RocksDbAllowConcurrentMemtableWriteDefault, + help = FinatraRocksDBConfig.RocksDbAllowConcurrentMemtableWriteDoc + ) + + protected val rocksDbEnableWriteThreadAdaptiveYield: Flag[Boolean] = + flag( + name = FinatraRocksDBConfig.RocksDbEnableWriteThreadAdaptiveYield, + default = FinatraRocksDBConfig.RocksDbEnableWriteThreadAdaptiveYieldDefault, + help = FinatraRocksDBConfig.RocksDbEnableWriteThreadAdaptiveYieldDoc + ) + + protected val rocksDbCompactionStyle: Flag[String] = + flag( + name = FinatraRocksDBConfig.RocksDbCompactionStyle, + default = FinatraRocksDBConfig.RocksDbCompactionStyleDefault, + help = FinatraRocksDBConfig.RocksDbCompactionStyleDoc + ) + + protected val rocksDbCompactionStyleOptimize: Flag[Boolean] = + flag( + name = FinatraRocksDBConfig.RocksDbCompactionStyleOptimize, + default = FinatraRocksDBConfig.RocksDbCompactionStyleOptimizeDefault, + help = FinatraRocksDBConfig.RocksDbCompactionStyleOptimizeDoc + ) + + protected val rocksDbMaxBytesForLevelBase: Flag[StorageUnit] = + flag( + name = FinatraRocksDBConfig.RocksDbMaxBytesForLevelBase, + default = FinatraRocksDBConfig.RocksDbMaxBytesForLevelBaseDefault, + help = FinatraRocksDBConfig.RocksDbMaxBytesForLevelBaseDoc + ) + + protected val rocksDbLevelCompactionDynamicLevelBytes: Flag[Boolean] = + flag( + name = FinatraRocksDBConfig.RocksDbLevelCompactionDynamicLevelBytes, + default = FinatraRocksDBConfig.RocksDbLevelCompactionDynamicLevelBytesDefault, + help = FinatraRocksDBConfig.RocksDbLevelCompactionDynamicLevelBytesDoc + ) + + protected val rocksDbCompactionStyleMemtableBudget: Flag[StorageUnit] = + flag( + name = FinatraRocksDBConfig.RocksDbCompactionStyleMemtableBudget, + default = FinatraRocksDBConfig.RocksDbCompactionStyleMemtableBudgetDefault, + help = FinatraRocksDBConfig.RocksDbCompactionStyleMemtableBudgetDoc + ) +} diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/dsl/FinatraDslSampling.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/dsl/FinatraDslSampling.scala index 5f210f2160..4bc00a8cb1 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/dsl/FinatraDslSampling.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/dsl/FinatraDslSampling.scala @@ -3,13 +3,10 @@ package com.twitter.finatra.kafkastreams.dsl import com.twitter.finagle.stats.StatsReceiver import com.twitter.finatra.kafka.serde.ScalaSerdes import com.twitter.finatra.kafkastreams.KafkaStreamsTwitterServer -import com.twitter.finatra.kafkastreams.internal.utils.sampling.{ - IndexedSampleKeySerde, - ReservoirSamplingTransformer -} -import com.twitter.finatra.streams.config.DefaultTopicConfig -import com.twitter.finatra.streams.flags.FinatraTransformerFlags -import com.twitter.finatra.streams.transformer.{FinatraTransformer, SamplingUtils} +import com.twitter.finatra.kafkastreams.config.{DefaultTopicConfig, FinatraTransformerFlags} +import com.twitter.finatra.kafkastreams.internal.utils.sampling.{IndexedSampleKeySerde, ReservoirSamplingTransformer} +import com.twitter.finatra.kafkastreams.transformer.FinatraTransformer +import com.twitter.finatra.kafkastreams.transformer.utils.SamplingUtils import com.twitter.inject.Logging import com.twitter.util.Duration import org.apache.kafka.common.serialization.Serde diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/dsl/FinatraDslWindowedAggregations.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/dsl/FinatraDslWindowedAggregations.scala index 7c8f9d4887..819c56589f 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/dsl/FinatraDslWindowedAggregations.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/dsl/FinatraDslWindowedAggregations.scala @@ -1,25 +1,19 @@ package com.twitter.finatra.kafkastreams.dsl import com.twitter.app.Flag -import com.twitter.conversions.storage._ -import com.twitter.conversions.time._ +import com.twitter.conversions.StorageUnitOps._ +import com.twitter.conversions.DurationOps._ import com.twitter.finagle.stats.StatsReceiver import com.twitter.finatra.kafka.serde.ScalaSerdes -import com.twitter.finatra.kafkastreams.internal.ScalaStreamsImplicits -import com.twitter.finatra.kafkastreams.processors.FlushingAwareServer -import com.twitter.finatra.streams.flags.FinatraTransformerFlags -import com.twitter.finatra.streams.transformer.domain._ -import com.twitter.finatra.streams.transformer.{AggregatorTransformer, FinatraTransformer} +import com.twitter.finatra.kafkastreams.config.FinatraTransformerFlags +import com.twitter.finatra.kafkastreams.flushing.FlushingAwareServer +import com.twitter.finatra.kafkastreams.transformer.FinatraTransformer +import com.twitter.finatra.kafkastreams.transformer.aggregation.{AggregatorTransformer, FixedTimeWindowedSerde, TimeWindowed, WindowedValue} +import com.twitter.finatra.kafkastreams.transformer.domain.Time +import com.twitter.finatra.kafkastreams.utils.ScalaStreamsImplicits import com.twitter.inject.Logging import com.twitter.util.Duration -import org.apache.kafka.common.config.TopicConfig.{ - CLEANUP_POLICY_COMPACT, - CLEANUP_POLICY_CONFIG, - CLEANUP_POLICY_DELETE, - DELETE_RETENTION_MS_CONFIG, - RETENTION_MS_CONFIG, - SEGMENT_BYTES_CONFIG -} +import org.apache.kafka.common.config.TopicConfig.{CLEANUP_POLICY_COMPACT, CLEANUP_POLICY_CONFIG, CLEANUP_POLICY_DELETE, DELETE_RETENTION_MS_CONFIG, RETENTION_MS_CONFIG, SEGMENT_BYTES_CONFIG} import org.apache.kafka.common.serialization.Serde import org.apache.kafka.streams.scala.kstream.{KStream => KStreamS} import org.apache.kafka.streams.state.Stores @@ -73,10 +67,10 @@ trait FinatraDslWindowedAggregations * A Window is closed after event time passes the end of a TimeWindow + allowedLateness. * * After a window is closed, if emitOnClose=true it is forwarded out of this transformer with a - * [[WindowedValue.resultState]] of [[com.twitter.finatra.streams.transformer.domain.WindowClosed]] + * [[WindowedValue.windowResultType]] of [[WindowClosed]] * * If a record arrives after a window is closed it is immediately forwarded out of this - * transformer with a [[WindowedValue.resultState]] of [[com.twitter.finatra.streams.transformer.domain.Restatement]] + * transformer with a [[WindowedValue.windowResultType]] of [[Restatement]] * * @param stateStore the name of the StateStore used to maintain the counts. * @param windowSize splits the stream of data into buckets of data of windowSize, @@ -111,7 +105,7 @@ trait FinatraDslWindowedAggregations aggregateSerde: Serde[Aggregate], initializer: () => Aggregate, aggregator: ((K, V), Aggregate) => Aggregate, - windowStart: (Time, K, V) => Long = null, + windowStart: (Time, K, V) => Time = null, emitOnClose: Boolean = true, emitUpdatedEntriesOnCommit: Boolean = false, windowSizeRetentionMultiplier: Int = 2 @@ -174,10 +168,10 @@ trait FinatraDslWindowedAggregations * A Window is closed after event time passes the end of a TimeWindow + allowedLateness. * * After a window is closed, if emitOnClose=true it is forwarded out of this transformer with a - * [[WindowedValue.resultState]] of [[com.twitter.finatra.streams.transformer.domain.WindowClosed]] + * [[WindowedValue.windowResultType]] of [[WindowClosed]] * * If a record arrives after a window is closed it is immediately forwarded out of this - * transformer with a [[WindowedValue.resultState]] of [[com.twitter.finatra.streams.transformer.domain.Restatement]] + * transformer with a [[WindowedValue.windowResultType]] of [[Restatement]] * * @param stateStore the name of the StateStore used to maintain the counts. * @param windowSize splits the stream of data into buckets of data of windowSize, @@ -192,6 +186,7 @@ trait FinatraDslWindowedAggregations * Streams commit interval. Emitted entries will have a * WindowResultType set to WindowOpen. * @param windowSizeRetentionMultiplier A multiplier on top of the windowSize to ensure data is not deleted from the changelog prematurely. Allows for clock drift. Default is 2 + * * @return a stream of Keys for a particular timewindow, and the sum of the values for that key * within a particular timewindow. */ @@ -242,10 +237,10 @@ trait FinatraDslWindowedAggregations * A Window is closed after event time passes the end of a TimeWindow + allowedLateness. * * After a window is closed, if emitOnClose=true it is forwarded out of this transformer with a - * [[WindowedValue.resultState]] of [[com.twitter.finatra.streams.transformer.domain.WindowClosed]] + * [[WindowedValue.windowResultType]] of [[WindowClosed]] * * If a record arrives after a window is closed it is immediately forwarded out of this - * transformer with a [[WindowedValue.resultState]] of [[com.twitter.finatra.streams.transformer.domain.Restatement]] + * transformer with a [[WindowedValue.windowResultType]] of [[Restatement]] * * @param stateStore the name of the StateStore used to maintain the counts. * @param windowSize splits the stream of data into buckets of data of windowSize, @@ -260,6 +255,7 @@ trait FinatraDslWindowedAggregations * Streams commit interval. Emitted entries will have a * WindowResultType set to WindowOpen. * @param windowSizeRetentionMultiplier A multiplier on top of the windowSize to ensure data is not deleted from the changelog prematurely. Allows for clock drift. Default is 2 + * * @return a stream of Keys for a particular timewindow, and the sum of the values for that key * within a particular timewindow. */ @@ -290,7 +286,7 @@ trait FinatraDslWindowedAggregations windowStart = { case (time, key, timeWindowedCount) => assert(timeWindowedCount.sizeMillis == windowSizeMillis) - timeWindowedCount.startMs + timeWindowedCount.start }, emitOnClose = emitOnClose, emitUpdatedEntriesOnCommit = emitUpdatedEntriesOnCommit, diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/processors/internal/AsyncFlushing.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/flushing/AsyncFlushing.scala similarity index 91% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/processors/internal/AsyncFlushing.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/flushing/AsyncFlushing.scala index 1b5596c11f..f36646b5b7 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/processors/internal/AsyncFlushing.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/flushing/AsyncFlushing.scala @@ -1,8 +1,8 @@ -package com.twitter.finatra.kafkastreams.processors.internal +package com.twitter.finatra.kafkastreams.flushing import com.twitter.finagle.stats.StatsReceiver -import com.twitter.finatra.kafkastreams.processors.MessageTimestamp -import com.twitter.finatra.streams.transformer.internal.{OnClose, OnInit} +import com.twitter.finatra.kafkastreams.transformer.lifecycle.{OnClose, OnInit} +import com.twitter.finatra.kafkastreams.utils.MessageTimestamp import com.twitter.util.{Await, Duration, Future, Return, Throw} import java.util.concurrent.Semaphore diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/processors/AsyncProcessor.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/flushing/AsyncProcessor.scala similarity index 81% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/processors/AsyncProcessor.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/flushing/AsyncProcessor.scala index b70bd549ee..9a0adafeb4 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/processors/AsyncProcessor.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/flushing/AsyncProcessor.scala @@ -1,7 +1,7 @@ -package com.twitter.finatra.kafkastreams.processors +package com.twitter.finatra.kafkastreams.flushing import com.twitter.finagle.stats.StatsReceiver -import com.twitter.finatra.kafkastreams.processors.internal.AsyncFlushing +import com.twitter.finatra.kafkastreams.utils.MessageTimestamp import com.twitter.util.{Duration, Future} abstract class AsyncProcessor[K, V]( @@ -9,7 +9,7 @@ abstract class AsyncProcessor[K, V]( override val maxOutstandingFuturesPerTask: Int, override val commitInterval: Duration, override val flushTimeout: Duration) - extends FlushingProcessor[K, V] + extends FlushingProcessor[K, V] with AsyncFlushing[K, V, Unit, Unit] { protected def processAsync(key: K, value: V, timestamp: MessageTimestamp): Future[Unit] diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/processors/AsyncTransformer.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/flushing/AsyncTransformer.scala similarity index 95% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/processors/AsyncTransformer.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/flushing/AsyncTransformer.scala index b6a5d5bb6d..eb8bc26b9b 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/processors/AsyncTransformer.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/flushing/AsyncTransformer.scala @@ -1,17 +1,11 @@ -package com.twitter.finatra.kafkastreams.processors +package com.twitter.finatra.kafkastreams.flushing import com.twitter.finagle.stats.StatsReceiver import com.twitter.finatra.kafkastreams.internal.utils.ProcessorContextLogging -import com.twitter.finatra.kafkastreams.processors.internal.AsyncFlushing +import com.twitter.finatra.kafkastreams.utils.MessageTimestamp import com.twitter.util.{Duration, Future} import java.util.concurrent.ConcurrentHashMap -import org.apache.kafka.streams.processor.{ - Cancellable, - ProcessorContext, - PunctuationType, - Punctuator, - To -} +import org.apache.kafka.streams.processor.{Cancellable, ProcessorContext, PunctuationType, Punctuator, To} /** * The AsyncTransformer trait allows async futures to be used to emit records downstreams diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/processors/internal/Flushing.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/flushing/Flushing.scala similarity index 86% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/processors/internal/Flushing.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/flushing/Flushing.scala index 2063b122cf..e91c278216 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/processors/internal/Flushing.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/flushing/Flushing.scala @@ -1,19 +1,17 @@ -package com.twitter.finatra.kafkastreams.processors.internal +package com.twitter.finatra.kafkastreams.flushing import com.twitter.finatra.kafkastreams.internal.utils.ProcessorContextLogging -import com.twitter.finatra.streams.transformer.internal.{OnClose, OnInit} +import com.twitter.finatra.kafkastreams.transformer.lifecycle.{OnClose, OnFlush, OnInit} import com.twitter.util.Duration import org.apache.kafka.streams.StreamsConfig import org.apache.kafka.streams.processor.{Cancellable, PunctuationType, Punctuator} -trait Flushing extends OnInit with OnClose with ProcessorContextLogging { +trait Flushing extends OnInit with OnClose with OnFlush with ProcessorContextLogging { @volatile private var commitPunctuatorCancellable: Cancellable = _ protected def commitInterval: Duration - protected def onFlush(): Unit = {} - //TODO: Create and use frameworkOnInit for framework use override def onInit(): Unit = { super.onInit() diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/processors/FlushingAwareServer.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/flushing/FlushingAwareServer.scala similarity index 86% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/processors/FlushingAwareServer.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/flushing/FlushingAwareServer.scala index cae93745fc..7fba9fc1fc 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/processors/FlushingAwareServer.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/flushing/FlushingAwareServer.scala @@ -1,4 +1,4 @@ -package com.twitter.finatra.kafkastreams.processors +package com.twitter.finatra.kafkastreams.flushing import com.twitter.finatra.kafkastreams.KafkaStreamsTwitterServer import com.twitter.finatra.kafkastreams.config.KafkaStreamsConfig @@ -7,7 +7,7 @@ import com.twitter.util.Duration /** * FlushingAwareServer must be mixed in to servers that rely on manually controlling when a flush/commit occurs. * As such, this trait will be needed when using the following classes, FlushingProcessor, FlushingTransformer, - * AsyncProcessor, AsyncTransformer, FinatraTransformer, and FinatraTransformerV2 + * AsyncProcessor, AsyncTransformer, and FinatraTransformer * * This trait sets 'kafka.commit.interval' to 'Duration.Top' to disable the normal Kafka Streams commit process. * As such the only commits that will occur are triggered manually, thus allowing us to control when flush/commit diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/processors/FlushingProcessor.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/flushing/FlushingProcessor.scala similarity index 69% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/processors/FlushingProcessor.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/flushing/FlushingProcessor.scala index 61401a715e..e587dc4594 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/processors/FlushingProcessor.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/flushing/FlushingProcessor.scala @@ -1,12 +1,11 @@ -package com.twitter.finatra.kafkastreams.processors +package com.twitter.finatra.kafkastreams.flushing import com.twitter.finatra.kafkastreams.internal.utils.ProcessorContextLogging -import com.twitter.finatra.kafkastreams.processors.internal.Flushing -import com.twitter.finatra.streams.transformer.internal.OnInit +import com.twitter.finatra.kafkastreams.transformer.lifecycle.OnInit import org.apache.kafka.streams.processor._ trait FlushingProcessor[K, V] - extends AbstractProcessor[K, V] + extends AbstractProcessor[K, V] with OnInit with Flushing with ProcessorContextLogging { diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/processors/FlushingTransformer.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/flushing/FlushingTransformer.scala similarity index 54% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/processors/FlushingTransformer.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/flushing/FlushingTransformer.scala index 274244d555..b01ac3c39c 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/processors/FlushingTransformer.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/flushing/FlushingTransformer.scala @@ -1,6 +1,5 @@ -package com.twitter.finatra.kafkastreams.processors +package com.twitter.finatra.kafkastreams.flushing -import com.twitter.finatra.kafkastreams.processors.internal.Flushing import org.apache.kafka.streams.kstream.Transformer trait FlushingTransformer[K, V, K1, V1] extends Transformer[K, V, (K1, V1)] with Flushing diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/admin/AdminRoutes.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/admin/AdminRoutes.scala new file mode 100644 index 0000000000..0602693ce3 --- /dev/null +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/admin/AdminRoutes.scala @@ -0,0 +1,25 @@ +package com.twitter.finatra.kafkastreams.internal.admin + +import com.twitter.server.AdminHttpServer +import com.twitter.server.AdminHttpServer.Route +import java.util.Properties +import org.apache.kafka.streams.Topology + +private[kafkastreams] object AdminRoutes { + def apply(properties: Properties, topology: Topology): Seq[Route] = { + // Kafka Properties + Seq(AdminHttpServer.mkRoute( + path = "/admin/kafka/streams/properties", + handler = KafkaStreamsPropertiesHandler(properties), + alias = "kafkaStreamsProperties", + group = Some("Kafka"), + includeInIndex = true), + // Kafka Topology + AdminHttpServer.mkRoute( + path = "/admin/kafka/streams/topology", + handler = KafkaStreamsTopologyHandler(topology), + alias = "kafkaStreamsTopology", + group = Some("Kafka"), + includeInIndex = true)) + } +} diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/admin/KafkaStreamsPropertiesHandler.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/admin/KafkaStreamsPropertiesHandler.scala new file mode 100644 index 0000000000..19ae0ac4ab --- /dev/null +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/admin/KafkaStreamsPropertiesHandler.scala @@ -0,0 +1,28 @@ +package com.twitter.finatra.kafkastreams.internal.admin + +import com.twitter.finagle.Service +import com.twitter.finagle.http._ +import com.twitter.util.Future +import java.util.Properties +import scala.collection.JavaConversions._ + +private[kafkastreams] object KafkaStreamsPropertiesHandler { + /** + * Create a service function that extracts the key/value of kafka properties and formats it in + * HTML. + * @param properties Kafka Properties + * @return HTML formatted properties + */ + def apply(properties: Properties): Service[Request, Response] = { + new Service[Request, Response] { + override def apply(request: Request): Future[Response] = { + val response = Response(Version.Http11, Status.Ok) + response.setContentType(MediaType.Html) + val sortedProperties = properties.propertyNames().map { property => + s"$property=${properties.get(property)}" + }.toSeq.sorted.mkString("\n") + ResponseWriter(response)(_.print(s"
$sortedProperties
")) + } + } + } +} diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/admin/KafkaStreamsTopologyHandler.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/admin/KafkaStreamsTopologyHandler.scala new file mode 100644 index 0000000000..aa6a25f45b --- /dev/null +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/admin/KafkaStreamsTopologyHandler.scala @@ -0,0 +1,29 @@ +package com.twitter.finatra.kafkastreams.internal.admin + +import com.twitter.finagle.Service +import com.twitter.finagle.http._ +import com.twitter.util.Future +import org.apache.kafka.streams.Topology + +private[kafkastreams] object KafkaStreamsTopologyHandler { + /** + * Create a service function that prints the kafka topology and formats it in HTML. + * @param topology Kafka Topology + * @return HTML formatted properties + */ + def apply(topology: Topology): Service[Request, Response] = { + new Service[Request, Response] { + override def apply(request: Request): Future[Response] = { + val response = Response(Version.Http11, Status.Ok) + response.setContentType(MediaType.Html) + val describeHtml = + s""" + |
+            |${topology.describe().toString.trim()}
+            |
+ """.stripMargin + ResponseWriter(response)(_.print(describeHtml)) + } + } + } +} diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/admin/ResponseWriter.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/admin/ResponseWriter.scala new file mode 100644 index 0000000000..6195f050c7 --- /dev/null +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/admin/ResponseWriter.scala @@ -0,0 +1,17 @@ +package com.twitter.finatra.kafkastreams.internal.admin + +import com.twitter.finagle.http._ +import com.twitter.util.Future +import java.io.{PrintWriter, StringWriter} + +private[kafkastreams] object ResponseWriter { + def apply(response: Response)(printer: PrintWriter => Unit): Future[Response] = { + val writer = new StringWriter() + val printWriter = new PrintWriter(writer) + printer(printWriter) + response.write(writer.getBuffer.toString) + printWriter.close() + writer.close() + Future.value(response) + } +} diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/interceptors/KafkaStreamsMonitoringConsumerInterceptor.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/interceptors/KafkaStreamsMonitoringConsumerInterceptor.scala similarity index 81% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/interceptors/KafkaStreamsMonitoringConsumerInterceptor.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/interceptors/KafkaStreamsMonitoringConsumerInterceptor.scala index 0250e8cac2..1378494e40 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/interceptors/KafkaStreamsMonitoringConsumerInterceptor.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/interceptors/KafkaStreamsMonitoringConsumerInterceptor.scala @@ -1,4 +1,4 @@ -package com.twitter.finatra.streams.interceptors +package com.twitter.finatra.kafkastreams.internal.interceptors import com.twitter.finatra.kafka.interceptors.MonitoringConsumerInterceptor @@ -9,7 +9,7 @@ import com.twitter.finatra.kafka.interceptors.MonitoringConsumerInterceptor * Note: Since this interceptor is Kafka Streams aware, it will not calculate stats when reading changelog topics to restore * state, since this has been shown to be a hot-spot during restoration of large amounts of state. */ -class KafkaStreamsMonitoringConsumerInterceptor extends MonitoringConsumerInterceptor { +private[kafkastreams] class KafkaStreamsMonitoringConsumerInterceptor extends MonitoringConsumerInterceptor { /** * Determines if this interceptor should be enabled given the consumer client id diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/listeners/FinatraStateRestoreListener.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/listeners/FinatraStateRestoreListener.scala index b52d2a2308..d28e973b6d 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/listeners/FinatraStateRestoreListener.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/listeners/FinatraStateRestoreListener.scala @@ -4,18 +4,31 @@ import com.twitter.finagle.stats.StatsReceiver import com.twitter.util.logging.Logging import org.apache.kafka.common.TopicPartition import org.apache.kafka.streams.processor.StateRestoreListener +import org.joda.time.DateTimeUtils -class FinatraStateRestoreListener( - statsReceiver: StatsReceiver) //TODO: Add stats for restoration (e.g. total time) +/** + * A [[StateRestoreListener]] that emits logs and metrics relating to state restoration. + * + * @param statsReceiver A StatsReceiver used for metric tracking. + */ +private[kafkastreams] class FinatraStateRestoreListener(statsReceiver: StatsReceiver) extends StateRestoreListener with Logging { + private val scopedStatReceiver = statsReceiver.scope("finatra_state_restore_listener") + private val totalRestoreTime = + scopedStatReceiver.addGauge("restore_time_elapsed_ms")(restoreTimeElapsedMs) + + private var restoreTimestampStartMs: Option[Long] = None + private var restoreTimestampEndMs: Option[Long] = None + override def onRestoreStart( topicPartition: TopicPartition, storeName: String, startingOffset: Long, endingOffset: Long ): Unit = { + restoreTimestampStartMs = Some(DateTimeUtils.currentTimeMillis) val upToRecords = endingOffset - startingOffset info( s"${storeAndPartition(storeName, topicPartition)} start restoring up to $upToRecords records from $startingOffset to $endingOffset" @@ -36,12 +49,20 @@ class FinatraStateRestoreListener( storeName: String, totalRestored: Long ): Unit = { + restoreTimestampEndMs = Some(DateTimeUtils.currentTimeMillis) info( - s"${storeAndPartition(storeName, topicPartition)} finished restoring $totalRestored records" + s"${storeAndPartition(storeName, topicPartition)} finished restoring $totalRestored records in $restoreTimeElapsedMs ms" ) } - private def storeAndPartition(storeName: String, topicPartition: TopicPartition) = { + private def storeAndPartition(storeName: String, topicPartition: TopicPartition): String = { s"$storeName topic ${topicPartition.topic}_${topicPartition.partition}" } + + private def restoreTimeElapsedMs: Long = { + val currentTimestampMs = DateTimeUtils.currentTimeMillis + restoreTimestampEndMs.getOrElse(currentTimestampMs) - restoreTimestampStartMs.getOrElse( + currentTimestampMs + ) + } } diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/serde/AvoidDefaultSerde.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/serde/AvoidDefaultSerde.scala index dfdef92963..4ecd66efcc 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/serde/AvoidDefaultSerde.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/serde/AvoidDefaultSerde.scala @@ -3,7 +3,7 @@ package com.twitter.finatra.kafkastreams.internal.serde import java.util import org.apache.kafka.common.serialization.{Deserializer, Serde, Serializer} -class AvoidDefaultSerde extends Serde[Object] { +private[kafkastreams] class AvoidDefaultSerde extends Serde[Object] { private val exceptionErrorStr = "should be avoided as they are error prone and often result in confusing error messages. " + "Instead, explicitly specify your serdes. See https://kafka.apache.org/10/documentation/streams/developer-guide/datatypes.html#overriding-default-serdes" diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/stats/KafkaStreamsFinagleMetricsReporter.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/stats/KafkaStreamsFinagleMetricsReporter.scala index ca758ac3ad..eed1c2d83c 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/stats/KafkaStreamsFinagleMetricsReporter.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/stats/KafkaStreamsFinagleMetricsReporter.scala @@ -4,8 +4,8 @@ import com.twitter.finatra.kafka.stats.KafkaFinagleMetricsReporter import java.util import org.apache.kafka.clients.CommonClientConfigs import org.apache.kafka.common.MetricName -import org.apache.kafka.common.metrics.Sensor.RecordingLevel import org.apache.kafka.common.metrics.KafkaMetric +import org.apache.kafka.common.metrics.Sensor.RecordingLevel object KafkaStreamsFinagleMetricsReporter { @@ -108,7 +108,7 @@ object KafkaStreamsFinagleMetricsReporter { * Kafka-Streams specific MetricsReporter which adds some additional logic on top of the metrics * reporter used for Kafka consumers and producers */ -class KafkaStreamsFinagleMetricsReporter extends KafkaFinagleMetricsReporter { +private[kafkastreams] class KafkaStreamsFinagleMetricsReporter extends KafkaFinagleMetricsReporter { private var includeProcessorNodeId = false private var includeGlobalTableMetrics = false diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/stats/RocksDBStatsCallback.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/stats/RocksDBStatsCallback.scala index 311df2e538..02600a7f31 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/stats/RocksDBStatsCallback.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/stats/RocksDBStatsCallback.scala @@ -18,7 +18,7 @@ import scala.collection.mutable.{Map => MutableMap} * https://github.com/facebook/rocksdb/wiki/Statistics * https://github.com/facebook/rocksdb/blob/master/include/rocksdb/statistics.h */ -class RocksDBStatsCallback(statsReceiver: StatsReceiver) +private[kafkastreams] class RocksDBStatsCallback(statsReceiver: StatsReceiver) extends StatisticsCollectorCallback with Logging { diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/utils/FinatraDslV2Implicits.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/utils/FinatraDslV2Implicits.scala deleted file mode 100644 index 0bc4ea2550..0000000000 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/utils/FinatraDslV2Implicits.scala +++ /dev/null @@ -1,254 +0,0 @@ -package com.twitter.finatra.kafkastreams.internal.utils - -import com.twitter.app.Flag -import com.twitter.finagle.stats.StatsReceiver -import com.twitter.finatra.kafka.serde.ScalaSerdes -import com.twitter.finatra.kafkastreams.internal.ScalaStreamsImplicits -import com.twitter.finatra.streams.config.DefaultTopicConfig -import com.twitter.finatra.streams.transformer.domain.{ - CompositeKey, - FixedTimeWindowedSerde, - TimeWindowed, - WindowedValue -} -import com.twitter.finatra.streams.transformer.{ - CompositeSumAggregator, - FinatraTransformer, - SumAggregator -} -import com.twitter.inject.Logging -import com.twitter.util.Duration -import org.apache.kafka.common.serialization.Serde -import org.apache.kafka.streams.kstream.Transformer -import org.apache.kafka.streams.scala.kstream.{KStream => KStreamS} -import org.apache.kafka.streams.state.Stores -import org.apache.kafka.streams.{KafkaStreams, StreamsBuilder} -import scala.reflect.ClassTag - -@deprecated("Use FinatraDslWindowedAggregations", "1/7/2019") -trait FinatraDslV2Implicits extends ScalaStreamsImplicits { - - protected def kafkaStreams: KafkaStreams - - protected def streamsStatsReceiver: StatsReceiver - - protected def kafkaStreamsBuilder: StreamsBuilder - - protected def commitInterval: Flag[Duration] - - /* ---------------------------------------- */ - implicit class FinatraKStream[K: ClassTag](inner: KStreamS[K, Int]) extends Logging { - - /** - * For each unique key, sum the values in the stream that occurred within a given time window - * and store those values in a StateStore named stateStore. - * - * A TimeWindow is a tumbling window of fixed length defined by the windowSize parameter. - * - * A Window is closed after event time passes the end of a TimeWindow + allowedLateness. - * - * After a window is closed it is forwarded out of this transformer with a - * [[WindowedValue.resultState]] of [[com.twitter.finatra.streams.transformer.domain.WindowClosed]] - * - * If a record arrives after a window is closed it is immediately forwarded out of this - * transformer with a [[WindowedValue.resultState]] of [[com.twitter.finatra.streams.transformer.domain.Restatement]] - * - * @param stateStore the name of the StateStore used to maintain the counts. - * @param windowSize splits the stream of data into buckets of data of windowSize, - * based on the timestamp of each message. - * @param allowedLateness allow messages that are upto this amount late to be added to the - * store, otherwise they are emitted as restatements. - * @param queryableAfterClose allow state to be queried upto this amount after the window is - * closed. - * @param keyRangeStart The minimum value that will be stored in the key based on binary sort order. - * @param keySerde Serde for the keys in the StateStore. - * @return a stream of Keys for a particular timewindow, and the sum of the values for that key - * within a particular timewindow. - */ - def sum( - stateStore: String, - windowSize: Duration, - allowedLateness: Duration, - queryableAfterClose: Duration, - keyRangeStart: K, - keySerde: Serde[K] - ): KStreamS[TimeWindowed[K], WindowedValue[Int]] = { - - kafkaStreamsBuilder.addStateStore( - Stores - .keyValueStoreBuilder( - Stores.persistentKeyValueStore(stateStore), - FixedTimeWindowedSerde(keySerde, windowSize), - ScalaSerdes.Int - ) - .withLoggingEnabled(DefaultTopicConfig.FinatraChangelogConfig) - ) - - //Note: The TimerKey is a WindowStartMs value used by MultiAttributeCountAggregator - val timerStore = FinatraTransformer.timerStore(s"$stateStore-TimerStore", ScalaSerdes.Long) - kafkaStreamsBuilder.addStateStore(timerStore) - - val transformerSupplier = () => - new SumAggregator[K, Int]( - commitInterval = commitInterval(), - keyRangeStart = keyRangeStart, - statsReceiver = streamsStatsReceiver, - stateStoreName = stateStore, - timerStoreName = timerStore.name(), - windowSize = windowSize, - allowedLateness = allowedLateness, - queryableAfterClose = queryableAfterClose, - countToAggregate = (key, count) => count, - windowStart = (messageTime, key, value) => - TimeWindowed.windowStart(messageTime, windowSize.inMillis) - ) - - inner.transform(transformerSupplier, stateStore, timerStore.name) - } - } - - /* ---------------------------------------- */ - implicit class FinatraKeyToWindowedValueStream[K, TimeWindowedType <: TimeWindowed[Int]]( - inner: KStreamS[K, TimeWindowedType]) - extends Logging { - - def sum( - stateStore: String, - allowedLateness: Duration, - queryableAfterClose: Duration, - emitOnClose: Boolean, - windowSize: Duration, - keyRangeStart: K, - keySerde: Serde[K] - ): KStreamS[TimeWindowed[K], WindowedValue[Int]] = { - kafkaStreamsBuilder.addStateStore( - Stores - .keyValueStoreBuilder( - Stores.persistentKeyValueStore(stateStore), - FixedTimeWindowedSerde(keySerde, windowSize), - ScalaSerdes.Int - ) - .withLoggingEnabled(DefaultTopicConfig.FinatraChangelogConfig) - ) - - //Note: The TimerKey is a WindowStartMs value used by MultiAttributeCountAggregator - val timerStore = FinatraTransformer.timerStore(s"$stateStore-TimerStore", ScalaSerdes.Long) - kafkaStreamsBuilder.addStateStore(timerStore) - - val transformerSupplier = ( - () => - new SumAggregator[K, TimeWindowed[Int]]( - commitInterval = commitInterval(), - keyRangeStart = keyRangeStart, - statsReceiver = streamsStatsReceiver, - stateStoreName = stateStore, - timerStoreName = timerStore.name(), - windowSize = windowSize, - allowedLateness = allowedLateness, - emitOnClose = emitOnClose, - queryableAfterClose = queryableAfterClose, - countToAggregate = (key, windowedValue) => windowedValue.value, - windowStart = (messageTime, key, windowedValue) => windowedValue.startMs - ) - ).asInstanceOf[() => Transformer[ - K, - TimeWindowedType, - (TimeWindowed[K], WindowedValue[Int])]] //Coerce TimeWindowed[Int] into TimeWindowedType :-/ - - inner - .transform(transformerSupplier, stateStore, timerStore.name) - } - } - - /* ---------------------------------------- */ - implicit class FinatraCompositeKeyKStream[CompositeKeyType <: CompositeKey[_, _]: ClassTag]( - inner: KStreamS[CompositeKeyType, Int]) - extends Logging { - - /** - * For each unique composite key, sum the values in the stream that occurred within a given time window. - * - * A composite key is a multi part key that can be efficiently range scanned using the - * primary key, or the primary key and the secondary key. - * - * A TimeWindow is a tumbling window of fixed length defined by the windowSize parameter. - * - * A Window is closed after event time passes the end of a TimeWindow + allowedLateness. - * - * After a window is closed it is forwarded out of this transformer with a - * [[WindowedValue.resultState]] of [[com.twitter.finatra.streams.transformer.domain.WindowClosed]] - * - * If a record arrives after a window is closed it is immediately forwarded out of this - * transformer with a [[WindowedValue.resultState]] of [[com.twitter.finatra.streams.transformer.domain.Restatement]] - * - * @param stateStore the name of the StateStore used to maintain the counts. - * @param windowSize splits the stream of data into buckets of data of windowSize, - * based on the timestamp of each message. - * @param allowedLateness allow messages that are upto this amount late to be added to the - * store, otherwise they are emitted as restatements. - * @param queryableAfterClose allow state to be queried upto this amount after the window is - * closed. - * @param emitOnClose whether or not to emit a record when the window is closed. - * @param compositeKeyRangeStart The minimum value that will be stored in the key based on binary sort order. - * @param compositeKeySerde serde for the composite key in the StateStore. - * @tparam PrimaryKey the type for the primary key - * @tparam SecondaryKey the type for the secondary key - * - * @return - */ - def compositeSum[PrimaryKey, SecondaryKey]( - stateStore: String, - windowSize: Duration, - allowedLateness: Duration, - queryableAfterClose: Duration, - emitOnClose: Boolean, - compositeKeyRangeStart: CompositeKey[PrimaryKey, SecondaryKey], - compositeKeySerde: Serde[CompositeKeyType] - ): KStreamS[TimeWindowed[PrimaryKey], WindowedValue[scala.collection.Map[SecondaryKey, Int]]] = { - - kafkaStreamsBuilder.addStateStore( - Stores - .keyValueStoreBuilder( - Stores.persistentKeyValueStore(stateStore), - FixedTimeWindowedSerde(compositeKeySerde, windowSize), - ScalaSerdes.Int - ) - .withLoggingEnabled(DefaultTopicConfig.FinatraChangelogConfig) - ) - - //Note: The TimerKey is a WindowStartMs value used by MultiAttributeCountAggregator - val timerStore = FinatraTransformer.timerStore(s"$stateStore-TimerStore", ScalaSerdes.Long) - kafkaStreamsBuilder.addStateStore(timerStore) - - val transformerSupplier = ( - () => - new CompositeSumAggregator[ - PrimaryKey, - SecondaryKey, - CompositeKey[ - PrimaryKey, - SecondaryKey - ]]( - commitInterval = commitInterval(), - compositeKeyRangeStart = compositeKeyRangeStart, - statsReceiver = streamsStatsReceiver, - stateStoreName = stateStore, - timerStoreName = timerStore.name(), - windowSize = windowSize, - allowedLateness = allowedLateness, - queryableAfterClose = queryableAfterClose, - emitOnClose = emitOnClose - ) - ).asInstanceOf[() => Transformer[ - CompositeKeyType, - Int, - (TimeWindowed[PrimaryKey], WindowedValue[scala.collection.Map[SecondaryKey, Int]]) - ]] //Coerce CompositeKey[PrimaryKey, SecondaryKey] into CompositeKeyType :-/ - - inner - .transform(transformerSupplier, stateStore, timerStore.name) - } - - } - -} diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/utils/KafkaFlagUtils.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/utils/KafkaFlagUtils.scala index f018923881..1140c1debd 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/utils/KafkaFlagUtils.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/utils/KafkaFlagUtils.scala @@ -1,9 +1,9 @@ -package com.twitter.finatra.kafkastreams.utils +package com.twitter.finatra.kafkastreams.internal.utils import com.twitter.app.{App, Flag, Flaggable} import org.apache.kafka.streams.StreamsConfig -trait KafkaFlagUtils extends App { +private[kafkastreams] trait KafkaFlagUtils extends App { def requiredKafkaFlag[T: Flaggable: Manifest](key: String, helpPrefix: String = ""): Flag[T] = { flag[T](name = "kafka." + key, help = helpPrefix + kafkaDocumentation(key)) diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/utils/ProcessorContextLogging.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/utils/ProcessorContextLogging.scala index b81032af9c..82f2c336ac 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/utils/ProcessorContextLogging.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/utils/ProcessorContextLogging.scala @@ -5,15 +5,11 @@ import org.apache.kafka.streams.processor.ProcessorContext import org.joda.time.DateTime import org.joda.time.format.ISODateTimeFormat -trait ProcessorContextLogging { +//TODO: Change viability to [kafkastreams] after deleting deprecated dependent code +private[finatra] trait ProcessorContextLogging { private val _logger = Logger(getClass) - @deprecated("Use error, warn, info, debug, or trace methods directly") - protected def logger: Logger = { - _logger - } - protected def processorContext: ProcessorContext final protected[this] def error(message: => Any): Unit = { diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/utils/ReflectionUtils.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/utils/ReflectionUtils.scala index 8b8d04dcd9..19b19b4079 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/utils/ReflectionUtils.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/utils/ReflectionUtils.scala @@ -2,7 +2,7 @@ package com.twitter.finatra.kafkastreams.internal.utils import java.lang.reflect.{Field, Modifier} -object ReflectionUtils { +private[kafkastreams] object ReflectionUtils { def getField(clazz: Class[_], fieldName: String): Field = { val field = clazz.getDeclaredField(fieldName) diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/utils/TopologyReflectionUtils.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/utils/TopologyReflectionUtils.scala index f73337c204..5aef2ea947 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/utils/TopologyReflectionUtils.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/utils/TopologyReflectionUtils.scala @@ -3,7 +3,7 @@ package com.twitter.finatra.kafkastreams.internal.utils import org.apache.kafka.streams.Topology import org.apache.kafka.streams.processor.internals.InternalTopologyBuilder -object TopologyReflectionUtils { +private[kafkastreams] object TopologyReflectionUtils { private val internalTopologyBuilderField = ReflectionUtils.getFinalField(classOf[Topology], "internalTopologyBuilder") diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/domain/IndexedSampleKey.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/utils/sampling/IndexedSampleKey.scala similarity index 91% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/domain/IndexedSampleKey.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/utils/sampling/IndexedSampleKey.scala index 71749c9346..17d57aebd0 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/domain/IndexedSampleKey.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/utils/sampling/IndexedSampleKey.scala @@ -1,4 +1,4 @@ -package com.twitter.finatra.streams.transformer.domain +package com.twitter.finatra.kafkastreams.internal.utils.sampling object IndexedSampleKey { def rangeStart[SampleKey](sampleKey: SampleKey): IndexedSampleKey[SampleKey] = { diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/utils/sampling/IndexedSampleKeySerde.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/utils/sampling/IndexedSampleKeySerde.scala index 7d3e16bd72..ed6cd8a37f 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/utils/sampling/IndexedSampleKeySerde.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/utils/sampling/IndexedSampleKeySerde.scala @@ -2,11 +2,10 @@ package com.twitter.finatra.kafkastreams.internal.utils.sampling import com.google.common.primitives.Ints import com.twitter.finatra.kafka.serde.AbstractSerde -import com.twitter.finatra.streams.transformer.domain.IndexedSampleKey import java.nio.ByteBuffer import org.apache.kafka.common.serialization.Serde -object IndexedSampleKeySerde { +private[kafkastreams] object IndexedSampleKeySerde { /** * Indexed sample key adds one Integer to the bytes @@ -14,7 +13,7 @@ object IndexedSampleKeySerde { val IndexSize: Int = Ints.BYTES } -class IndexedSampleKeySerde[SampleKey](sampleKeySerde: Serde[SampleKey]) +private[kafkastreams] class IndexedSampleKeySerde[SampleKey](sampleKeySerde: Serde[SampleKey]) extends AbstractSerde[IndexedSampleKey[SampleKey]] { private val sampleKeySerializer = sampleKeySerde.serializer() diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/utils/sampling/ReservoirSamplingTransformer.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/utils/sampling/ReservoirSamplingTransformer.scala index 1b309590fa..4b193a99af 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/utils/sampling/ReservoirSamplingTransformer.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/utils/sampling/ReservoirSamplingTransformer.scala @@ -1,13 +1,9 @@ package com.twitter.finatra.kafkastreams.internal.utils.sampling import com.twitter.finagle.stats.StatsReceiver -import com.twitter.finatra.streams.transformer.domain.{ - Expire, - IndexedSampleKey, - Time, - TimerMetadata -} -import com.twitter.finatra.streams.transformer.{FinatraTransformerV2, PersistentTimers} +import com.twitter.finatra.kafkastreams.transformer.FinatraTransformer +import com.twitter.finatra.kafkastreams.transformer.domain.{Expire, Time, TimerMetadata} +import com.twitter.finatra.kafkastreams.transformer.stores.PersistentTimers import com.twitter.util.Duration import org.apache.kafka.streams.processor.PunctuationType import scala.reflect.ClassTag @@ -31,7 +27,7 @@ class ReservoirSamplingTransformer[ countStoreName: String, sampleStoreName: String, timerStoreName: String) - extends FinatraTransformerV2[Key, Value, SampleKey, SampleValue](statsReceiver = statsReceiver) + extends FinatraTransformer[Key, Value, SampleKey, SampleValue](statsReceiver = statsReceiver) with PersistentTimers { private val numExpiredCounter = statsReceiver.counter("numExpired") @@ -49,7 +45,7 @@ class ReservoirSamplingTransformer[ for (eTime <- expirationTime) { if (isFirstTimeSampleKeySeen(totalCount)) { - timerStore.addTimer(messageTime.plus(eTime), Expire, sampleKey) + timerStore.addTimer(messageTime + eTime, Expire, sampleKey) } } @@ -66,9 +62,7 @@ class ReservoirSamplingTransformer[ sampleStore .deleteRange( IndexedSampleKey.rangeStart(key), - IndexedSampleKey.rangeEnd(key), - maxDeletes = sampleSize - ) + IndexedSampleKey.rangeEnd(key)) numExpiredCounter.incr() } diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/partitioners/RoundRobinStreamPartitioner.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/partitioners/RoundRobinStreamPartitioner.scala deleted file mode 100644 index 8d0243b0cb..0000000000 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/partitioners/RoundRobinStreamPartitioner.scala +++ /dev/null @@ -1,21 +0,0 @@ -package com.twitter.finatra.kafkastreams.partitioners - -import org.apache.kafka.streams.processor.StreamPartitioner - -/** - * Partitions in a round robin fashion going from 0 to numPartitions -1 and wrapping around again. - * - * @tparam K the key on the stream - * @tparam V the value on the stream - */ -@deprecated("no longer supported", "1/7/2019") -class RoundRobinStreamPartitioner[K, V] extends StreamPartitioner[K, V] { - - private var nextPartitionId: Int = 0 - - override def partition(topic: String, key: K, value: V, numPartitions: Int): Integer = { - val partitionIdToReturn = nextPartitionId - nextPartitionId = (nextPartitionId + 1) % numPartitions - partitionIdToReturn - } -} diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/punctuators/AdvancedPunctuator.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/punctuators/AdvancedPunctuator.scala deleted file mode 100644 index 0914e8d24d..0000000000 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/punctuators/AdvancedPunctuator.scala +++ /dev/null @@ -1,28 +0,0 @@ -package com.twitter.finatra.kafkastreams.punctuators - -import org.apache.kafka.streams.processor.Punctuator - -/** - * A Punctuator that will only only call 'punctuateAdvanced' when the timestamp is greater than the last timestamp. - * - * *Note* if you extend this class you probably do not want to override 'punctuate' - */ -@deprecated("no longer supported", "1/7/2019") -trait AdvancedPunctuator extends Punctuator { - - private var lastPunctuateTimeMillis = Long.MinValue - - override def punctuate(timestampMillis: Long): Unit = { - if (timestampMillis > lastPunctuateTimeMillis) { - punctuateAdvanced(timestampMillis) - lastPunctuateTimeMillis = timestampMillis - } - } - - /** - * This will only be called if the timestamp is greater than the previous time - * - * @param timestampMillis the timestamp of the punctuate - */ - def punctuateAdvanced(timestampMillis: Long): Unit -} diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/query/QueryableFinatraCompositeWindowStore.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/query/QueryableFinatraCompositeWindowStore.scala similarity index 80% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/query/QueryableFinatraCompositeWindowStore.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/query/QueryableFinatraCompositeWindowStore.scala index ba1fd4a19b..68d2ab104d 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/query/QueryableFinatraCompositeWindowStore.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/query/QueryableFinatraCompositeWindowStore.scala @@ -1,14 +1,13 @@ -package com.twitter.finatra.streams.query - -import com.twitter.finatra.streams.converters.time._ +package com.twitter.finatra.kafkastreams.query + +import com.twitter.conversions.DurationOps._ +import com.twitter.finatra.kafkastreams.transformer.FinatraTransformer.{DateTimeMillis, WindowStartTime} +import com.twitter.finatra.kafkastreams.transformer.aggregation.TimeWindowed +import com.twitter.finatra.kafkastreams.transformer.domain.{CompositeKey, Time} +import com.twitter.finatra.kafkastreams.transformer.stores.internal.FinatraStoresGlobalManager +import com.twitter.finatra.kafkastreams.utils.time._ import com.twitter.finatra.streams.queryable.thrift.domain.ServiceShardId -import com.twitter.finatra.streams.queryable.thrift.partitioning.{ - KafkaPartitioner, - StaticServiceShardPartitioner -} -import com.twitter.finatra.streams.stores.internal.FinatraStoresGlobalManager -import com.twitter.finatra.streams.transformer.FinatraTransformer.{DateTimeMillis, WindowStartTime} -import com.twitter.finatra.streams.transformer.domain.{CompositeKey, Time, TimeWindowed} +import com.twitter.finatra.streams.queryable.thrift.partitioning.{KafkaPartitioner, StaticServiceShardPartitioner} import com.twitter.inject.Logging import com.twitter.util.Duration import org.apache.kafka.common.serialization.{Serde, Serializer} @@ -90,13 +89,13 @@ class QueryableFinatraCompositeWindowStore[PK, SK, V]( allowStaleReads: Boolean, resultMap: scala.collection.mutable.Map[Long, scala.collection.mutable.Map[SK, V]] ): Unit = { - trace(s"QueryWindow $startCompositeKey to $endCompositeKey ${windowStartTime.iso8601}") + trace(s"QueryWindow $startCompositeKey to $endCompositeKey ${windowStartTime.asInstanceOf[Long].iso8601}") //TODO: Use store.taskId to find exact store where the key is assigned for (store <- FinatraStoresGlobalManager.getWindowedCompositeStores[PK, SK, V](storeName)) { val iterator = store.range( - TimeWindowed.forSize(startMs = windowStartTime, windowSizeMillis, startCompositeKey), - TimeWindowed.forSize(startMs = windowStartTime, windowSizeMillis, endCompositeKey), + TimeWindowed.forSize(start = Time(windowStartTime), windowSize, startCompositeKey), + TimeWindowed.forSize(start = Time(windowStartTime), windowSize, endCompositeKey), allowStaleReads = allowStaleReads ) @@ -104,7 +103,7 @@ class QueryableFinatraCompositeWindowStore[PK, SK, V]( val entry = iterator.next() trace(s"$store\t$entry") val innerMap = - resultMap.getOrElseUpdate(entry.key.startMs, scala.collection.mutable.Map[SK, V]()) + resultMap.getOrElseUpdate(entry.key.start.millis, scala.collection.mutable.Map[SK, V]()) innerMap += (entry.key.value.secondary -> entry.value) } } @@ -116,9 +115,10 @@ class QueryableFinatraCompositeWindowStore[PK, SK, V]( windowSizeMillis: DateTimeMillis ): (DateTimeMillis, DateTimeMillis) = { val endWindowRange = endTime.getOrElse { - TimeWindowed.windowStart( - messageTime = Time(DateTimeUtils.currentTimeMillis), - sizeMs = windowSizeMillis) + defaultWindowMultiplier * windowSizeMillis + TimeWindowed + .windowStart(messageTime = Time(DateTimeUtils.currentTimeMillis), size = windowSize) + .+(windowSizeMillis.millis * defaultWindowMultiplier) + .millis } val startWindowRange = diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/query/QueryableFinatraKeyValueStore.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/query/QueryableFinatraKeyValueStore.scala similarity index 92% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/query/QueryableFinatraKeyValueStore.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/query/QueryableFinatraKeyValueStore.scala index 86ae4b92b3..b42b9961e6 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/query/QueryableFinatraKeyValueStore.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/query/QueryableFinatraKeyValueStore.scala @@ -1,12 +1,9 @@ -package com.twitter.finatra.streams.query +package com.twitter.finatra.kafkastreams.query +import com.twitter.finatra.kafkastreams.transformer.stores.FinatraKeyValueStore +import com.twitter.finatra.kafkastreams.transformer.stores.internal.FinatraStoresGlobalManager import com.twitter.finatra.streams.queryable.thrift.domain.ServiceShardId -import com.twitter.finatra.streams.queryable.thrift.partitioning.{ - KafkaPartitioner, - StaticServiceShardPartitioner -} -import com.twitter.finatra.streams.stores.FinatraKeyValueStore -import com.twitter.finatra.streams.stores.internal.FinatraStoresGlobalManager +import com.twitter.finatra.streams.queryable.thrift.partitioning.{KafkaPartitioner, StaticServiceShardPartitioner} import com.twitter.inject.Logging import java.util.NoSuchElementException import org.apache.kafka.common.serialization.Serde diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/query/QueryableFinatraWindowStore.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/query/QueryableFinatraWindowStore.scala similarity index 65% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/query/QueryableFinatraWindowStore.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/query/QueryableFinatraWindowStore.scala index f73e4eb9f6..acdd9e18b6 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/query/QueryableFinatraWindowStore.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/query/QueryableFinatraWindowStore.scala @@ -1,14 +1,12 @@ -package com.twitter.finatra.streams.query +package com.twitter.finatra.kafkastreams.query +import com.twitter.finatra.kafkastreams.transformer.FinatraTransformer.{DateTimeMillis, WindowStartTime} +import com.twitter.finatra.kafkastreams.transformer.aggregation.TimeWindowed +import com.twitter.finatra.kafkastreams.transformer.domain.Time +import com.twitter.finatra.kafkastreams.transformer.stores.FinatraKeyValueStore +import com.twitter.finatra.kafkastreams.transformer.stores.internal.FinatraStoresGlobalManager import com.twitter.finatra.streams.queryable.thrift.domain.ServiceShardId -import com.twitter.finatra.streams.queryable.thrift.partitioning.{ - KafkaPartitioner, - StaticServiceShardPartitioner -} -import com.twitter.finatra.streams.stores.FinatraKeyValueStore -import com.twitter.finatra.streams.stores.internal.FinatraStoresGlobalManager -import com.twitter.finatra.streams.transformer.FinatraTransformer.{DateTimeMillis, WindowStartTime} -import com.twitter.finatra.streams.transformer.domain.{Time, TimeWindowed} +import com.twitter.finatra.streams.queryable.thrift.partitioning.{KafkaPartitioner, StaticServiceShardPartitioner} import com.twitter.inject.Logging import com.twitter.util.Duration import org.apache.kafka.common.serialization.{Serde, Serializer} @@ -31,7 +29,7 @@ class QueryableFinatraWindowStore[K, V]( private val currentServiceShardId = ServiceShardId(currentShardId) - private val windowSizeMillis = windowSize.inMillis + private val queryWindowSize = windowSize * defaultWindowMultiplier private val partitioner = new KafkaPartitioner( StaticServiceShardPartitioner(numShards = numShards), @@ -46,28 +44,29 @@ class QueryableFinatraWindowStore[K, V]( throwIfNonLocalKey(key, keySerializer) val endWindowRange = endTime.getOrElse( - TimeWindowed.windowStart( - messageTime = Time(DateTimeUtils.currentTimeMillis), - sizeMs = windowSizeMillis) + defaultWindowMultiplier * windowSizeMillis) + TimeWindowed + .windowStart(messageTime = Time(DateTimeUtils.currentTimeMillis), size = windowSize) + .+(queryWindowSize) + .millis) val startWindowRange = - startTime.getOrElse(endWindowRange - (defaultWindowMultiplier * windowSizeMillis)) + startTime.getOrElse(endWindowRange - queryWindowSize.inMillis) val windowedMap = new java.util.TreeMap[DateTimeMillis, V] - var currentWindowStart = startWindowRange - while (currentWindowStart <= endWindowRange) { - val windowedKey = TimeWindowed.forSize(currentWindowStart, windowSize.inMillis, key) + var currentWindowStart = Time(startWindowRange) + while (currentWindowStart.millis <= endWindowRange) { + val windowedKey = TimeWindowed.forSize(currentWindowStart, windowSize, key) //TODO: Use store.taskId to find exact store where the key is assigned for (store <- stores) { val result = store.get(windowedKey) if (result != null) { - windowedMap.put(currentWindowStart, result) + windowedMap.put(currentWindowStart.millis, result) } } - currentWindowStart = currentWindowStart + windowSizeMillis + currentWindowStart = currentWindowStart + windowSize } windowedMap.asScala.toMap diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/FinatraTransformerV2.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/FinatraTransformer.scala similarity index 70% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/FinatraTransformerV2.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/FinatraTransformer.scala index 4adf00758c..e77160d749 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/FinatraTransformerV2.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/FinatraTransformer.scala @@ -1,72 +1,79 @@ -package com.twitter.finatra.streams.transformer +package com.twitter.finatra.kafkastreams.transformer import com.google.common.annotations.Beta import com.twitter.finagle.stats.StatsReceiver import com.twitter.finatra.kafka.utils.ConfigUtils +import com.twitter.finatra.kafkastreams.config.{DefaultTopicConfig, FinatraTransformerFlags} import com.twitter.finatra.kafkastreams.internal.utils.ProcessorContextLogging -import com.twitter.finatra.streams.flags.FinatraTransformerFlags -import com.twitter.finatra.streams.stores.FinatraKeyValueStore -import com.twitter.finatra.streams.stores.internal.{ - FinatraKeyValueStoreImpl, - FinatraStoresGlobalManager -} -import com.twitter.finatra.streams.transformer.FinatraTransformer.TimerTime -import com.twitter.finatra.streams.transformer.domain.{Time, Watermark} -import com.twitter.finatra.streams.transformer.internal.{OnClose, OnInit} -import com.twitter.finatra.streams.transformer.watermarks.internal.WatermarkManager -import com.twitter.finatra.streams.transformer.watermarks.{ - DefaultWatermarkAssignor, - WatermarkAssignor -} +import com.twitter.finatra.kafkastreams.transformer.FinatraTransformer.TimerTime +import com.twitter.finatra.kafkastreams.transformer.domain.Time +import com.twitter.finatra.kafkastreams.transformer.lifecycle.{OnClose, OnFlush, OnInit, OnWatermark} +import com.twitter.finatra.kafkastreams.transformer.stores.FinatraKeyValueStore +import com.twitter.finatra.kafkastreams.transformer.stores.internal.{FinatraKeyValueStoreImpl, FinatraStoresGlobalManager, Timer} +import com.twitter.finatra.kafkastreams.transformer.watermarks.{DefaultWatermarkAssignor, Watermark, WatermarkAssignor, WatermarkManager} +import com.twitter.finatra.streams.transformer.internal.domain.TimerSerde import com.twitter.util.Duration +import org.apache.kafka.common.serialization.{Serde, Serdes} import org.apache.kafka.streams.kstream.Transformer -import org.apache.kafka.streams.processor.{ - Cancellable, - ProcessorContext, - PunctuationType, - Punctuator, - To -} +import org.apache.kafka.streams.processor.{Cancellable, ProcessorContext, PunctuationType, Punctuator, To} +import org.apache.kafka.streams.state.{KeyValueStore, StoreBuilder, Stores} import scala.collection.mutable import scala.reflect.ClassTag +object FinatraTransformer { + type TimerTime = Long + type WindowStartTime = Long + type DateTimeMillis = Long + + def timerStore[TimerKey]( + name: String, + timerKeySerde: Serde[TimerKey] + ): StoreBuilder[KeyValueStore[Timer[TimerKey], Array[Byte]]] = { + Stores + .keyValueStoreBuilder( + Stores.persistentKeyValueStore(name), + TimerSerde(timerKeySerde), + Serdes.ByteArray + ) + .withLoggingEnabled(DefaultTopicConfig.FinatraChangelogConfig) + } +} + /** * A KafkaStreams Transformer offering an upgraded API over the built in Transformer interface. * - * This Transformer differs from the built in Transformer interface by exposing an [onMesssage] + * This Transformer differs from the built in Transformer interface by exposing an [onMessage] * interface that is used to process incoming messages. Within [onMessage] you may use the * [forward] method to emit 0 or more records. * * This transformer also manages watermarks(see [WatermarkManager]), and extends [OnWatermark] which * allows you to track the passage of event time. * - * Note: In time, this class will replace the deprecated FinatraTransformer class - * * @tparam InputKey Type of the input keys * @tparam InputValue Type of the input values * @tparam OutputKey Type of the output keys * @tparam OutputValue Type of the output values */ @Beta -abstract class FinatraTransformerV2[InputKey, InputValue, OutputKey, OutputValue]( +abstract class FinatraTransformer[InputKey, InputValue, OutputKey, OutputValue]( statsReceiver: StatsReceiver, watermarkAssignor: WatermarkAssignor[InputKey, InputValue] = - new DefaultWatermarkAssignor[InputKey, InputValue]) - extends Transformer[InputKey, InputValue, (OutputKey, OutputValue)] + new DefaultWatermarkAssignor[InputKey, InputValue]) + extends Transformer[InputKey, InputValue, (OutputKey, OutputValue)] with OnInit with OnWatermark with OnClose + with OnFlush with ProcessorContextLogging { - protected[streams] val finatraKeyValueStoresMap: mutable.Map[String, FinatraKeyValueStore[_, _]] = + protected[kafkastreams] val finatraKeyValueStoresMap: mutable.Map[String, FinatraKeyValueStore[_, _]] = scala.collection.mutable.Map[String, FinatraKeyValueStore[_, _]]() - private var watermarkManager: WatermarkManager[InputKey, InputValue] = _ - /* Private Mutable */ @volatile private var _context: ProcessorContext = _ @volatile private var watermarkTimerCancellable: Cancellable = _ + @volatile private var watermarkManager: WatermarkManager[InputKey, InputValue] = _ /* Abstract */ @@ -88,6 +95,8 @@ abstract class FinatraTransformerV2[InputKey, InputValue, OutputKey, OutputValue _context = processorContext watermarkManager = new WatermarkManager[InputKey, InputValue]( + taskId = processorContext.taskId(), + transformerName = this.getClass.getSimpleName, onWatermark = this, watermarkAssignor = watermarkAssignor, emitWatermarkPerMessage = shouldEmitWatermarkPerMessage(_context)) @@ -112,6 +121,11 @@ abstract class FinatraTransformerV2[InputKey, InputValue, OutputKey, OutputValue onInit() } + override def onFlush(): Unit = { + super.onFlush() + watermarkManager.callOnWatermarkIfChanged() + } + override def onWatermark(watermark: Watermark): Unit = { trace(s"onWatermark $watermark") } @@ -122,8 +136,8 @@ abstract class FinatraTransformerV2[InputKey, InputValue, OutputKey, OutputValue can cause context.timestamp to be mutated to the forwarded message timestamp :-( */ val messageTime = Time(_context.timestamp()) - debug(s"onMessage $watermark MessageTime(${messageTime.millis.iso8601Millis}) $k -> $v") watermarkManager.onMessage(messageTime, _context.topic(), k, v) + debug(s"onMessage LastEmitted $watermark MessageTime $messageTime $k -> $v") onMessage(messageTime, k, v) null } @@ -175,7 +189,7 @@ abstract class FinatraTransformerV2[InputKey, InputValue, OutputKey, OutputValue _context.forward(key, value, To.all().withTimestamp(timestamp)) } - final protected def watermark: Watermark = { + final protected[finatra] def watermark: Watermark = { watermarkManager.watermark } diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/AggregatorTransformer.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/aggregation/AggregatorTransformer.scala similarity index 84% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/AggregatorTransformer.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/aggregation/AggregatorTransformer.scala index da63f1dc87..97a12f1406 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/AggregatorTransformer.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/aggregation/AggregatorTransformer.scala @@ -1,10 +1,11 @@ -package com.twitter.finatra.streams.transformer +package com.twitter.finatra.kafkastreams.transformer.aggregation import com.twitter.finagle.stats.StatsReceiver import com.twitter.finatra.kafka.serde.ScalaSerdes -import com.twitter.finatra.streams.stores.CachingFinatraKeyValueStore -import com.twitter.finatra.streams.transformer.FinatraTransformer.WindowStartTime -import com.twitter.finatra.streams.transformer.domain._ +import com.twitter.finatra.kafkastreams.transformer.FinatraTransformer +import com.twitter.finatra.kafkastreams.transformer.FinatraTransformer.WindowStartTime +import com.twitter.finatra.kafkastreams.transformer.domain._ +import com.twitter.finatra.kafkastreams.transformer.stores.{CachingFinatraKeyValueStore, CachingKeyValueStores, PersistentTimers} import com.twitter.util.Duration import it.unimi.dsi.fastutil.longs.LongOpenHashSet import org.apache.kafka.streams.processor.PunctuationType @@ -19,10 +20,10 @@ import org.apache.kafka.streams.state.KeyValueIterator * A Window is closed after event time passes the end of a TimeWindow + allowedLateness. * * After a window is closed, if emitOnClose=true it is forwarded out of this transformer with a - * [[WindowedValue.resultState]] of [[com.twitter.finatra.streams.transformer.domain.WindowClosed]] + * [[WindowedValue.windowResultType]] of [[WindowClosed]] * * If a record arrives after a window is closed it is immediately forwarded out of this - * transformer with a [[WindowedValue.resultState]] of [[com.twitter.finatra.streams.transformer.domain.Restatement]] + * transformer with a [[WindowedValue.windowResultType]] of [[Restatement]] * * @param statsReceiver The StatsReceiver for collecting stats * @param stateStoreName the name of the StateStore used to maintain the counts. @@ -39,6 +40,7 @@ import org.apache.kafka.streams.state.KeyValueIterator * @param emitUpdatedEntriesOnCommit Emit messages for each updated entry in the window on the Kafka * Streams commit interval. Emitted entries will have a * WindowResultType set to WindowOpen. + * * @return a stream of Keys for a particular timewindow, and the aggregations of the values for that * key within a particular timewindow. */ @@ -50,12 +52,12 @@ class AggregatorTransformer[K, V, Aggregate]( allowedLateness: Duration, initializer: () => Aggregate, aggregator: ((K, V), Aggregate) => Aggregate, - customWindowStart: (Time, K, V) => Long, + customWindowStart: (Time, K, V) => Time, emitOnClose: Boolean = false, queryableAfterClose: Duration, emitUpdatedEntriesOnCommit: Boolean, val commitInterval: Duration) - extends FinatraTransformerV2[K, V, TimeWindowed[K], WindowedValue[Aggregate]](statsReceiver) + extends FinatraTransformer[K, V, TimeWindowed[K], WindowedValue[Aggregate]](statsReceiver) with CachingKeyValueStores[K, V, TimeWindowed[K], WindowedValue[Aggregate]] with PersistentTimers { @@ -89,14 +91,14 @@ class AggregatorTransformer[K, V, Aggregate]( override def onMessage(time: Time, key: K, value: V): Unit = { val windowedKey = TimeWindowed.forSize( - startMs = windowStart(time, key, value), - sizeMs = windowSizeMillis, + start = windowStart(time, key, value), + size = windowSize, value = key) - if (windowedKey.isLate(allowedLatenessMillis, watermark)) { + if (windowedKey.isLate(allowedLateness, watermark)) { restatement(time, key, value, windowedKey) } else { - addWindowTimersIfNew(windowedKey.startMs) + addWindowTimersIfNew(windowedKey.start.millis) val currentAggregateValue = stateStore.getOrDefault(windowedKey, initializer()) stateStore.put(windowedKey, aggregator((key, value), currentAggregateValue)) @@ -124,14 +126,14 @@ class AggregatorTransformer[K, V, Aggregate]( val existing = stateStore.get(timeWindowedKey) forward( key = timeWindowedKey, - value = WindowedValue(resultState = WindowOpen, value = existing), + value = WindowedValue(windowResultType = WindowOpen, value = existing), timestamp = forwardTime) } } private def restatement(time: Time, key: K, value: V, windowedKey: TimeWindowed[K]): Unit = { val windowedValue = - WindowedValue(resultState = Restatement, value = aggregator((key, value), initializer())) + WindowedValue(windowResultType = Restatement, value = aggregator((key, value), initializer())) forward(key = windowedKey, value = windowedValue, timestamp = forwardTime) @@ -165,10 +167,10 @@ class AggregatorTransformer[K, V, Aggregate]( ): Unit = { while (windowIterator.hasNext) { val entry = windowIterator.next() - assert(entry.key.startMs == windowStartTime) + assert(entry.key.start.millis == windowStartTime) forward( key = entry.key, - value = WindowedValue(resultState = WindowClosed, value = entry.value), + value = WindowedValue(windowResultType = WindowClosed, value = entry.value), timestamp = forwardTime) } @@ -192,11 +194,11 @@ class AggregatorTransformer[K, V, Aggregate]( longSerializer.serialize("", windowStartMs) } - private def windowStart(time: Time, key: K, value: V): Long = { + private def windowStart(time: Time, key: K, value: V): Time = { if (customWindowStart != null) { customWindowStart(time, key, value) } else { - TimeWindowed.windowStart(time, windowSizeMillis) + TimeWindowed.windowStart(time, windowSize) } } diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/domain/FixedTimeWindowedSerde.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/aggregation/FixedTimeWindowedSerde.scala similarity index 79% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/domain/FixedTimeWindowedSerde.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/aggregation/FixedTimeWindowedSerde.scala index 6573effffb..6ab890dd66 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/domain/FixedTimeWindowedSerde.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/aggregation/FixedTimeWindowedSerde.scala @@ -1,6 +1,7 @@ -package com.twitter.finatra.streams.transformer.domain +package com.twitter.finatra.kafkastreams.transformer.aggregation import com.twitter.finatra.kafka.serde.AbstractSerde +import com.twitter.finatra.kafkastreams.transformer.domain.Time import com.twitter.util.Duration import java.nio.ByteBuffer import org.apache.kafka.common.serialization.Serde @@ -37,20 +38,24 @@ class FixedTimeWindowedSerde[K](val inner: Serde[K], windowSize: Duration) bb.get(keyBytes) val endMs = startMs + windowSizeMillis - TimeWindowed(startMs = startMs, endMs = endMs, innerDeserializer.deserialize(topic, keyBytes)) + TimeWindowed( + start = Time(startMs), + end = Time(endMs), + innerDeserializer.deserialize(topic, keyBytes) + ) } final override def serialize(timeWindowedKey: TimeWindowed[K]): Array[Byte] = { assert( - timeWindowedKey.startMs + windowSizeMillis == timeWindowedKey.endMs, - s"TimeWindowed element being serialized has end time which is not consistent with the FixedTimeWindowedSerde window size of $windowSize. ${timeWindowedKey.startMs + windowSizeMillis} != ${timeWindowedKey.endMs}" + timeWindowedKey.start + windowSize == timeWindowedKey.end, + s"TimeWindowed element being serialized has end time which is not consistent with the FixedTimeWindowedSerde window size of $windowSize. ${timeWindowedKey.start + windowSize} != ${timeWindowedKey.end}" ) val keyBytes = innerSerializer.serialize(topic, timeWindowedKey.value) val windowAndKeyBytesSize = new Array[Byte](WindowStartTimeSizeBytes + keyBytes.length) val bb = ByteBuffer.wrap(windowAndKeyBytesSize) - bb.putLong(timeWindowedKey.startMs) + bb.putLong(timeWindowedKey.start.millis) bb.put(keyBytes) bb.array() } diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/aggregation/TimeWindowed.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/aggregation/TimeWindowed.scala new file mode 100644 index 0000000000..17f2ef53c5 --- /dev/null +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/aggregation/TimeWindowed.scala @@ -0,0 +1,88 @@ +package com.twitter.finatra.kafkastreams.transformer.aggregation + +import com.twitter.finatra.kafkastreams.transformer.domain.Time +import com.twitter.finatra.kafkastreams.transformer.watermarks.Watermark +import com.twitter.util.Duration +import org.joda.time.{DateTime, DateTimeConstants} + +object TimeWindowed { + + def forSize[V](start: Time, size: Duration, value: V): TimeWindowed[V] = { + TimeWindowed(start, start + size, value) + } + + def forSizeFromMessageTime[V](messageTime: Time, size: Duration, value: V): TimeWindowed[V] = { + val startWindow = windowStart(messageTime, size) + TimeWindowed(startWindow, startWindow + size, value) + } + + def hourly[V](start: Time, value: V): TimeWindowed[V] = { + TimeWindowed(start, Time(start.millis + DateTimeConstants.MILLIS_PER_HOUR), value) + } + + def windowStart(messageTime: Time, size: Duration): Time = { + Time((messageTime.millis / size.inMillis) * size.inMillis) + } +} + +/** + * A time windowed value specified by a start and end time + * @param start the start time of the window (inclusive) + * @param end the end time of the window (exclusive) + */ +case class TimeWindowed[V](start: Time, end: Time, value: V) { + + /** + * Determine if this windowed value is late given the allowedLateness configuration and the + * current watermark + * + * @param allowedLateness the configured amount of allowed lateness specified in milliseconds + * @param watermark a watermark used to determine if this windowed value is late + * @return If the windowed value is late + */ + def isLate(allowedLateness: Duration, watermark: Watermark): Boolean = { + watermark.timeMillis > end.millis + allowedLateness.inMillis + } + + /** + * Determine the start of the next fixed window interval + */ + def nextInterval(time: Time, duration: Duration): Time = { + val intervalStart = Time(math.max(start.millis, time.millis)) + Time.nextInterval(intervalStart, duration) + } + + /** + * Map the time windowed value into another value occurring in the same window + */ + def map[KK](f: V => KK): TimeWindowed[KK] = { + copy(value = f(value)) + } + + /** + * The size of this windowed value in milliseconds + */ + def sizeMillis: Long = end.millis - start.millis + + final override val hashCode: Int = { + var result = value.hashCode() + result = 31 * result + (start.millis ^ (start.millis >>> 32)).toInt + result = 31 * result + (end.millis ^ (end.millis >>> 32)).toInt + result + } + + final override def equals(obj: scala.Any): Boolean = { + obj match { + case other: TimeWindowed[V] => + start == other.start && + end == other.end && + value == other.value + case _ => + false + } + } + + override def toString: String = { + s"TimeWindowed(${new DateTime(start.millis)}-${new DateTime(end.millis)}-$value)" + } +} diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/domain/WindowValueResult.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/aggregation/WindowValueResult.scala similarity index 93% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/domain/WindowValueResult.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/aggregation/WindowValueResult.scala index 8b99291499..f2a68294d2 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/domain/WindowValueResult.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/aggregation/WindowValueResult.scala @@ -1,4 +1,4 @@ -package com.twitter.finatra.streams.transformer.domain +package com.twitter.finatra.kafkastreams.transformer.aggregation object WindowResultType { def apply(value: Byte): WindowResultType = { diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/aggregation/WindowedValue.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/aggregation/WindowedValue.scala new file mode 100644 index 0000000000..70fe2ea788 --- /dev/null +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/aggregation/WindowedValue.scala @@ -0,0 +1,3 @@ +package com.twitter.finatra.kafkastreams.transformer.aggregation + +case class WindowedValue[V](windowResultType: WindowResultType, value: V) diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/domain/WindowedValueSerde.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/aggregation/WindowedValueSerde.scala similarity index 87% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/domain/WindowedValueSerde.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/aggregation/WindowedValueSerde.scala index 5ea0aeacea..9a1cc7ae28 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/domain/WindowedValueSerde.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/aggregation/WindowedValueSerde.scala @@ -1,4 +1,4 @@ -package com.twitter.finatra.streams.transformer.domain +package com.twitter.finatra.kafkastreams.transformer.aggregation import com.twitter.finatra.kafka.serde.AbstractSerde import java.nio.ByteBuffer @@ -27,7 +27,7 @@ class WindowedValueSerde[V](inner: Serde[V]) extends AbstractSerde[WindowedValue System.arraycopy(bytes, 1, valueBytes, 0, valueBytes.length) val value = innerDeserializer.deserialize(topic, valueBytes) - WindowedValue(resultState = resultState, value = value) + WindowedValue(windowResultType = resultState, value = value) } override def serialize(windowedValue: WindowedValue[V]): Array[Byte] = { @@ -35,7 +35,7 @@ class WindowedValueSerde[V](inner: Serde[V]) extends AbstractSerde[WindowedValue val resultTypeAndValueBytes = new Array[Byte](1 + valueBytes.size) val bb = ByteBuffer.wrap(resultTypeAndValueBytes) - bb.put(windowedValue.resultState.value) + bb.put(windowedValue.windowResultType.value) bb.put(valueBytes) resultTypeAndValueBytes } diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/domain/CompositeKey.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/domain/CompositeKey.scala similarity index 52% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/domain/CompositeKey.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/domain/CompositeKey.scala index e948f0ed74..b022e97d9c 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/domain/CompositeKey.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/domain/CompositeKey.scala @@ -1,4 +1,4 @@ -package com.twitter.finatra.streams.transformer.domain +package com.twitter.finatra.kafkastreams.transformer.domain trait CompositeKey[P, S] { def primary: P diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/domain/Time.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/domain/Time.scala new file mode 100644 index 0000000000..f22ab679af --- /dev/null +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/domain/Time.scala @@ -0,0 +1,77 @@ +package com.twitter.finatra.kafkastreams.transformer.domain + +import com.twitter.finatra.kafkastreams.transformer.aggregation.TimeWindowed +import com.twitter.finatra.kafkastreams.utils.time._ +import com.twitter.util.Duration +import org.joda.time.{DateTime, DateTimeConstants} + +object Time { + /** + * Construct a [[Time]] from a [[DateTime]]. + */ + def create(datetime: DateTime): Time = { + new Time(datetime.getMillis) + } + + /** + * Finds the next interval occurrence of a [[Duration]] from a [[Time]]. + * + * @param time A point in time. + * @param duration The duration of an interval. + */ + def nextInterval(time: Time, duration: Duration): Time = { + val durationMillis = duration.inMillis + val currentNumIntervals = time.millis / durationMillis + Time((currentNumIntervals + 1) * durationMillis) + } +} + +/** + * A Value Class representing a point in time. + * + * @param millis A millisecond timestamp. + */ +case class Time(millis: Long) extends AnyVal { + + /** + * Adds a [[Time]] to the [[Time]]. + */ + final def +(time: Time): Time = { + new Time(millis + time.millis) + } + + /** + * Adds a [[Duration]] to the [[Time]]. + */ + final def +(duration: Duration): Time = { + new Time(millis + duration.inMillis) + } + + /** + * Rounds down [[Time]] to the nearest hour. + */ + final def hour: Time = { + roundDown(DateTimeConstants.MILLIS_PER_HOUR) + } + + /** + * Rounds down ''millis'' to the nearest multiple of ''milliseconds''. + */ + final def roundDown(milliseconds: Long): Time = { + val unitsSinceEpoch = millis / milliseconds + Time(unitsSinceEpoch * milliseconds) + } + + /** + * @return An hourly window from derived from the [[Time]]. + */ + final def hourlyWindowed[K](key: K): TimeWindowed[K] = { + val start = hour + val end = Time(start.millis + DateTimeConstants.MILLIS_PER_HOUR) + TimeWindowed(start, end, key) + } + + override def toString: String = { + s"Time(${millis.iso8601Millis})" + } +} diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/domain/TimerMetadata.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/domain/TimerMetadata.scala similarity index 89% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/domain/TimerMetadata.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/domain/TimerMetadata.scala index 901cdb9cb9..c1db7ae9c1 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/domain/TimerMetadata.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/domain/TimerMetadata.scala @@ -1,4 +1,4 @@ -package com.twitter.finatra.streams.transformer.domain +package com.twitter.finatra.kafkastreams.transformer.domain object TimerMetadata { def apply(value: Byte): TimerMetadata = { @@ -13,7 +13,7 @@ object TimerMetadata { /** * Metadata used to convey the purpose of a - * [[com.twitter.finatra.streams.transformer.internal.domain.Timer]]. + * [[Timer]]. * * [[TimerMetadata]] represents the following Timer actions: [[EmitEarly]], [[Close]], [[Expire]] */ diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/internal/ProcessorContextUtils.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/internal/ProcessorContextUtils.scala similarity index 93% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/internal/ProcessorContextUtils.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/internal/ProcessorContextUtils.scala index d13c2fe90a..6332678fe2 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/internal/ProcessorContextUtils.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/internal/ProcessorContextUtils.scala @@ -1,4 +1,4 @@ -package com.twitter.finatra.streams.transformer.internal +package com.twitter.finatra.kafkastreams.transformer.internal import com.twitter.finatra.kafkastreams.internal.utils.ReflectionUtils import java.lang.reflect.Field diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/lifecycle/OnClose.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/lifecycle/OnClose.scala new file mode 100644 index 0000000000..640a8c5c79 --- /dev/null +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/lifecycle/OnClose.scala @@ -0,0 +1,5 @@ +package com.twitter.finatra.kafkastreams.transformer.lifecycle + +trait OnClose { + protected def onClose(): Unit = {} +} diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/lifecycle/OnFlush.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/lifecycle/OnFlush.scala new file mode 100644 index 0000000000..dee9a53ea4 --- /dev/null +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/lifecycle/OnFlush.scala @@ -0,0 +1,10 @@ +package com.twitter.finatra.kafkastreams.transformer.lifecycle + +trait OnFlush { + + /** + * Callback method for when you should flush any cached data. + * This method is typically called prior to a Kafka commit + */ + protected def onFlush(): Unit = {} +} diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/lifecycle/OnInit.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/lifecycle/OnInit.scala new file mode 100644 index 0000000000..f639256443 --- /dev/null +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/lifecycle/OnInit.scala @@ -0,0 +1,5 @@ +package com.twitter.finatra.kafkastreams.transformer.lifecycle + +trait OnInit { + protected def onInit(): Unit = {} +} diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/lifecycle/OnWatermark.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/lifecycle/OnWatermark.scala new file mode 100644 index 0000000000..b2800d03a0 --- /dev/null +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/lifecycle/OnWatermark.scala @@ -0,0 +1,7 @@ +package com.twitter.finatra.kafkastreams.transformer.lifecycle + +import com.twitter.finatra.kafkastreams.transformer.watermarks.Watermark + +trait OnWatermark { + def onWatermark(watermark: Watermark): Unit +} diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/stores/CachingFinatraKeyValueStore.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/stores/CachingFinatraKeyValueStore.scala similarity index 88% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/stores/CachingFinatraKeyValueStore.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/stores/CachingFinatraKeyValueStore.scala index eafd7cba16..8e2543ec8c 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/stores/CachingFinatraKeyValueStore.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/stores/CachingFinatraKeyValueStore.scala @@ -1,4 +1,4 @@ -package com.twitter.finatra.streams.stores +package com.twitter.finatra.kafkastreams.transformer.stores /** * A FinatraKeyValueStore with a callback that fires when an entry is flushed into the underlying store diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/CachingKeyValueStores.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/stores/CachingKeyValueStores.scala similarity index 76% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/CachingKeyValueStores.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/stores/CachingKeyValueStores.scala index 615bcb8a73..c2f61adfb1 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/CachingKeyValueStores.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/stores/CachingKeyValueStores.scala @@ -1,13 +1,8 @@ -package com.twitter.finatra.streams.transformer +package com.twitter.finatra.kafkastreams.transformer.stores import com.twitter.finagle.stats.StatsReceiver -import com.twitter.finatra.kafkastreams.processors.FlushingTransformer -import com.twitter.finatra.streams.stores.internal.{ - CachingFinatraKeyValueStoreImpl, - FinatraKeyValueStoreImpl, - FinatraStoresGlobalManager -} -import com.twitter.finatra.streams.stores.{CachingFinatraKeyValueStore, FinatraKeyValueStore} +import com.twitter.finatra.kafkastreams.flushing.FlushingTransformer +import com.twitter.finatra.kafkastreams.transformer.stores.internal.{CachingFinatraKeyValueStoreImpl, FinatraKeyValueStoreImpl, FinatraStoresGlobalManager} import scala.collection.mutable import scala.reflect.ClassTag @@ -18,6 +13,7 @@ trait CachingKeyValueStores[K, V, K1, V1] extends FlushingTransformer[K, V, K1, protected def finatraKeyValueStoresMap: mutable.Map[String, FinatraKeyValueStore[_, _]] override def onFlush(): Unit = { + super.onFlush() finatraKeyValueStoresMap.values.foreach(_.flush()) } diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/stores/FinatraKeyValueStore.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/stores/FinatraKeyValueStore.scala similarity index 88% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/stores/FinatraKeyValueStore.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/stores/FinatraKeyValueStore.scala index 617b011451..8c001efc67 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/stores/FinatraKeyValueStore.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/stores/FinatraKeyValueStore.scala @@ -1,5 +1,6 @@ -package com.twitter.finatra.streams.stores -import com.twitter.finatra.streams.transformer.domain.TimerResult +package com.twitter.finatra.kafkastreams.transformer.stores + +import org.apache.kafka.streams.errors.InvalidStateStoreException import org.apache.kafka.streams.processor.TaskId import org.apache.kafka.streams.state.{KeyValueIterator, KeyValueStore} @@ -25,10 +26,18 @@ trait FinatraKeyValueStore[K, V] extends KeyValueStore[K, V] { * @throws NullPointerException If null is used for from or to. * @throws InvalidStateStoreException if the store is not initialized */ + @throws[InvalidStateStoreException] def range(from: K, to: K, allowStaleReads: Boolean): KeyValueIterator[K, V] - @deprecated("no longer supported", "1/7/2019") - def deleteRange(from: K, to: K, maxDeletes: Int = 25000): TimerResult[K] + /** + * Removes the database entries in the range ["from", "to"), i.e., + * including "from" and excluding "to". It is not an error if no keys exist + * in the range ["from", "to"). + * + * @throws InvalidStateStoreException if the store is not initialized + */ + @throws[InvalidStateStoreException] + def deleteRange(from: K, to: K): Unit /** * Delete the value from the store (if there is one) @@ -63,7 +72,10 @@ trait FinatraKeyValueStore[K, V] extends KeyValueStore[K, V] { * Note 2: If this RocksDB instance is configured in "prefix seek mode", than fromBytes will be used as a "prefix" and the iteration will end when the prefix is no longer part of the next element. * Enabling "prefix seek mode" can be done by calling options.useFixedLengthPrefixExtractor. When enabled, prefix scans can take advantage of a prefix based bloom filter for better seek performance * See: https://github.com/facebook/rocksdb/wiki/Prefix-Seek-API-Changes + * + * @throws InvalidStateStoreException if the store is not initialized */ + @throws[InvalidStateStoreException] def range(fromBytes: Array[Byte]): KeyValueIterator[K, V] /** @@ -79,6 +91,7 @@ trait FinatraKeyValueStore[K, V] extends KeyValueStore[K, V] { * @throws NullPointerException If null is used for from or to. * @throws InvalidStateStoreException if the store is not initialized */ + @throws[InvalidStateStoreException] def range(fromBytesInclusive: Array[Byte], toBytesExclusive: Array[Byte]): KeyValueIterator[K, V] /** diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/PersistentTimerStore.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/stores/PersistentTimerStore.scala similarity index 79% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/PersistentTimerStore.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/stores/PersistentTimerStore.scala index 81d2dd5717..77fb36e801 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/PersistentTimerStore.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/stores/PersistentTimerStore.scala @@ -1,11 +1,13 @@ -package com.twitter.finatra.streams.transformer +package com.twitter.finatra.kafkastreams.transformer.stores import com.google.common.annotations.Beta -import com.twitter.finatra.streams.converters.time._ -import com.twitter.finatra.streams.stores.FinatraKeyValueStore -import com.twitter.finatra.streams.transformer.FinatraTransformer.TimerTime -import com.twitter.finatra.streams.transformer.domain.{Time, TimerMetadata, Watermark} -import com.twitter.finatra.streams.transformer.internal.domain.{Timer, TimerSerde} +import com.twitter.finatra.kafkastreams.transformer.FinatraTransformer.TimerTime +import com.twitter.finatra.kafkastreams.transformer.domain.{Time, TimerMetadata} +import com.twitter.finatra.kafkastreams.transformer.lifecycle.OnWatermark +import com.twitter.finatra.kafkastreams.transformer.stores.internal.Timer +import com.twitter.finatra.kafkastreams.transformer.watermarks.Watermark +import com.twitter.finatra.kafkastreams.utils.time._ +import com.twitter.finatra.streams.transformer.internal.domain.TimerSerde import com.twitter.inject.Logging import org.apache.kafka.streams.state.KeyValueIterator @@ -26,12 +28,12 @@ class PersistentTimerStore[TimerKey]( def onInit(): Unit = { setNextTimerTime(Long.MaxValue) - currentWatermark = Watermark(0) + currentWatermark = Watermark(0L) val iterator = timersStore.all() try { if (iterator.hasNext) { - setNextTimerTime(iterator.next.key.time) + setNextTimerTime(iterator.next.key.time.millis) } } finally { iterator.close() @@ -61,7 +63,7 @@ class PersistentTimerStore[TimerKey]( } else { debug(f"${"AddTimer:"}%-20s ${metadata.getClass.getSimpleName}%-12s Key $key Timer $time") timersStore.put( - Timer(time = time.millis, metadata = metadata, key = key), + Timer(time = time, metadata = metadata, key = key), Array.emptyByteArray) if (time.millis < nextTimerTime) { @@ -97,7 +99,7 @@ class PersistentTimerStore[TimerKey]( while (timerIterator.hasNext && !timerIteratorState.done) { currentTimer = timerIterator.next().key - if (watermark.timeMillis >= currentTimer.time) { + if (watermark.timeMillis >= currentTimer.time.millis) { fireAndDeleteTimer(currentTimer) numTimerFires += 1 if (numTimerFires >= maxTimerFiresPerWatermark) { @@ -109,11 +111,11 @@ class PersistentTimerStore[TimerKey]( } if (timerIteratorState == FoundTimerAfterWatermark) { - setNextTimerTime(currentTimer.time) + setNextTimerTime(currentTimer.time.millis) } else if (timerIteratorState == ExceededMaxTimers && timerIterator.hasNext) { - setNextTimerTime(timerIterator.next().key.time) + setNextTimerTime(timerIterator.next().key.time.millis) debug( - s"Exceeded $maxTimerFiresPerWatermark max timer fires per watermark. LastTimerFired: ${currentTimer.time.iso8601Millis} NextTimer: ${nextTimerTime.iso8601Millis}" + s"Exceeded $maxTimerFiresPerWatermark max timer fires per watermark. LastTimerFired: ${currentTimer.time.millis.iso8601Millis} NextTimer: ${nextTimerTime.iso8601Millis}" ) } else { assert(!timerIterator.hasNext) @@ -139,7 +141,7 @@ class PersistentTimerStore[TimerKey]( private def fireAndDeleteTimer(timer: Timer[TimerKey]): Unit = { trace(s"fireAndDeleteTimer $timer") - onTimer(Time(timer.time), timer.metadata, timer.key) + onTimer(timer.time, timer.metadata, timer.key) timersStore.deleteWithoutGettingPriorValue(timer) } diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/PersistentTimers.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/stores/PersistentTimers.scala similarity index 78% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/PersistentTimers.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/stores/PersistentTimers.scala index 3581314f8a..d0712c3f86 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/PersistentTimers.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/stores/PersistentTimers.scala @@ -1,10 +1,10 @@ -package com.twitter.finatra.streams.transformer +package com.twitter.finatra.kafkastreams.transformer.stores import com.google.common.annotations.Beta -import com.twitter.finatra.streams.stores.FinatraKeyValueStore -import com.twitter.finatra.streams.transformer.domain.{Time, TimerMetadata, Watermark} -import com.twitter.finatra.streams.transformer.internal.OnInit -import com.twitter.finatra.streams.transformer.internal.domain.Timer +import com.twitter.finatra.kafkastreams.transformer.domain.{Time, TimerMetadata} +import com.twitter.finatra.kafkastreams.transformer.lifecycle.{OnInit, OnWatermark} +import com.twitter.finatra.kafkastreams.transformer.stores.internal.Timer +import com.twitter.finatra.kafkastreams.transformer.watermarks.Watermark import java.util import org.apache.kafka.streams.processor.PunctuationType import scala.reflect.ClassTag @@ -13,8 +13,8 @@ import scala.reflect.ClassTag * Per-Key Persistent Timers inspired by Flink's ProcessFunction: * https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/process_function.html * - * Note: Timers are based on a sorted RocksDB KeyValueStore - * Note: Timers that fire at the same time MAY NOT fire in the order which they were added + * @note Timers are based on a sorted RocksDB KeyValueStore + * @note Timers that fire at the same time MAY NOT fire in the order which they were added */ @Beta trait PersistentTimers extends OnWatermark with OnInit { diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/stores/internal/CachingFinatraKeyValueStoreImpl.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/stores/internal/CachingFinatraKeyValueStoreImpl.scala similarity index 96% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/stores/internal/CachingFinatraKeyValueStoreImpl.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/stores/internal/CachingFinatraKeyValueStoreImpl.scala index c3e8efe6fc..a019ef0e9a 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/stores/internal/CachingFinatraKeyValueStoreImpl.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/stores/internal/CachingFinatraKeyValueStoreImpl.scala @@ -1,8 +1,7 @@ -package com.twitter.finatra.streams.stores.internal +package com.twitter.finatra.kafkastreams.transformer.stores.internal import com.twitter.finagle.stats.{Gauge, StatsReceiver} -import com.twitter.finatra.streams.stores.{CachingFinatraKeyValueStore, FinatraKeyValueStore} -import com.twitter.finatra.streams.transformer.domain.TimerResult +import com.twitter.finatra.kafkastreams.transformer.stores.{CachingFinatraKeyValueStore, FinatraKeyValueStore} import com.twitter.inject.Logging import it.unimi.dsi.fastutil.objects.Object2ObjectOpenHashMap import java.util @@ -204,9 +203,9 @@ class CachingFinatraKeyValueStoreImpl[K: ClassTag, V]( keyValueStore.deleteRangeExperimentalWithNoChangelogUpdates(beginKeyInclusive, endKeyExclusive) } - override def deleteRange(from: K, to: K, maxDeletes: Int): TimerResult[K] = { + override def deleteRange(from: K, to: K): Unit = { flushObjectCache() - keyValueStore.deleteRange(from, to, maxDeletes) + keyValueStore.deleteRange(from, to) } override def approximateNumEntries(): Long = { diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/stores/internal/FinatraKeyValueStoreImpl.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/stores/internal/FinatraKeyValueStoreImpl.scala similarity index 87% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/stores/internal/FinatraKeyValueStoreImpl.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/stores/internal/FinatraKeyValueStoreImpl.scala index 22e84cea9d..2e95e3d758 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/stores/internal/FinatraKeyValueStoreImpl.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/stores/internal/FinatraKeyValueStoreImpl.scala @@ -1,11 +1,11 @@ -package com.twitter.finatra.streams.stores.internal +package com.twitter.finatra.kafkastreams.transformer.stores.internal import com.twitter.finagle.stats.{Gauge, Stat, StatsReceiver} import com.twitter.finatra.kafkastreams.internal.utils.ReflectionUtils -import com.twitter.finatra.streams.stores.FinatraKeyValueStore -import com.twitter.finatra.streams.stores.internal.FinatraKeyValueStoreImpl._ -import com.twitter.finatra.streams.transformer.IteratorImplicits -import com.twitter.finatra.streams.transformer.domain.{DeleteTimer, RetainTimer, TimerResult} +import FinatraKeyValueStoreImpl._ +import com.twitter.finatra.kafkastreams.transformer.stores.FinatraKeyValueStore +import com.twitter.finatra.kafkastreams.transformer.utils.IteratorImplicits +import com.twitter.finatra.kafkastreams.utils.RocksKeyValueIterator import com.twitter.inject.Logging import java.util import java.util.Comparator @@ -14,14 +14,9 @@ import org.apache.kafka.common.serialization.{Deserializer, Serializer} import org.apache.kafka.common.utils.Bytes import org.apache.kafka.streams.KeyValue import org.apache.kafka.streams.processor.{ProcessorContext, StateStore, TaskId} -import org.apache.kafka.streams.state.internals.{ - MeteredKeyValueBytesStore, - RocksDBStore, - RocksKeyValueIterator -} +import org.apache.kafka.streams.state.internals.{MeteredKeyValueBytesStore, RocksDBStore} import org.apache.kafka.streams.state.{KeyValueIterator, KeyValueStore, StateSerdes} import org.rocksdb.{RocksDB, WriteOptions} -import scala.collection.JavaConverters._ import scala.reflect.ClassTag object FinatraKeyValueStoreImpl { @@ -175,19 +170,13 @@ case class FinatraKeyValueStoreImpl[K: ClassTag, V]( /* Finatra Additions */ - @deprecated("no longer supported", "1/7/2019") - override def deleteRange(from: K, to: K, maxDeletes: Int = 25000): TimerResult[K] = { + override def deleteRange(from: K, to: K): Unit = { meterLatency(deleteRangeLatencyStat) { val iterator = range(from, to) try { - val keysToDelete = iterator.asScala - .take(maxDeletes) - .map(keyValue => new KeyValue(keyValue.key, null.asInstanceOf[V])) - .toList - .asJava - - putAll(keysToDelete) - deleteOrRetainTimer(iterator) + while (iterator.hasNext) { + delete(iterator.next.key) + } } finally { iterator.close() } @@ -241,7 +230,7 @@ case class FinatraKeyValueStoreImpl[K: ClassTag, V]( override def hasNext: Boolean = { super.hasNext && - comparator.compare(iterator.key(), toBytesExclusive) < 0 // < 0 since to is exclusive + comparator.compare(iterator.key(), toBytesExclusive) < 0 // < 0 since to is exclusive } } } @@ -269,19 +258,6 @@ case class FinatraKeyValueStoreImpl[K: ClassTag, V]( } } - @deprecated - private def deleteOrRetainTimer( - iterator: KeyValueIterator[K, _], - onDeleteTimer: => Unit = () => () - ): TimerResult[K] = { - if (iterator.hasNext) { - RetainTimer(stateStoreCursor = iterator.peekNextKeyOpt, throttled = true) - } else { - onDeleteTimer - DeleteTimer() - } - } - private def keyValueStore: KeyValueStore[K, V] = { assert( _keyValueStore != null, diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/stores/internal/FinatraStoresGlobalManager.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/stores/internal/FinatraStoresGlobalManager.scala similarity index 86% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/stores/internal/FinatraStoresGlobalManager.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/stores/internal/FinatraStoresGlobalManager.scala index 68bff217ad..43ca5776f3 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/stores/internal/FinatraStoresGlobalManager.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/stores/internal/FinatraStoresGlobalManager.scala @@ -1,9 +1,10 @@ -package com.twitter.finatra.streams.stores.internal +package com.twitter.finatra.kafkastreams.transformer.stores.internal -import scala.collection.JavaConverters._ import com.google.common.collect.{ArrayListMultimap, Multimaps} -import com.twitter.finatra.streams.stores.FinatraKeyValueStore -import com.twitter.finatra.streams.transformer.domain.{CompositeKey, TimeWindowed} +import com.twitter.finatra.kafkastreams.transformer.aggregation.TimeWindowed +import com.twitter.finatra.kafkastreams.transformer.domain.CompositeKey +import com.twitter.finatra.kafkastreams.transformer.stores.FinatraKeyValueStore +import scala.collection.JavaConverters._ import scala.reflect.ClassTag /** diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/stores/internal/Timer.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/stores/internal/Timer.scala new file mode 100644 index 0000000000..e296f8a289 --- /dev/null +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/stores/internal/Timer.scala @@ -0,0 +1,14 @@ +package com.twitter.finatra.kafkastreams.transformer.stores.internal + +import com.twitter.finatra.kafkastreams.transformer.domain.{Time, TimerMetadata} +import com.twitter.finatra.kafkastreams.utils.time._ + +/** + * @param time Time to fire the timer + */ +case class Timer[K](time: Time, metadata: TimerMetadata, key: K) { + + override def toString: String = { + s"Timer(${metadata.getClass.getName} $key @${time.millis.iso8601Millis})" + } +} diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/internal/domain/TimerSerde.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/stores/internal/TimerSerde.scala similarity index 81% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/internal/domain/TimerSerde.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/stores/internal/TimerSerde.scala index 7145774126..78104ef5b1 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/internal/domain/TimerSerde.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/stores/internal/TimerSerde.scala @@ -2,7 +2,8 @@ package com.twitter.finatra.streams.transformer.internal.domain import com.google.common.primitives.Longs import com.twitter.finatra.kafka.serde.AbstractSerde -import com.twitter.finatra.streams.transformer.domain.TimerMetadata +import com.twitter.finatra.kafkastreams.transformer.domain.{Time, TimerMetadata} +import com.twitter.finatra.kafkastreams.transformer.stores.internal.Timer import java.nio.ByteBuffer import org.apache.kafka.common.serialization.Serde @@ -19,6 +20,11 @@ object TimerSerde { } } +/** + * Serde for the [[Timer]] class. + * + * @param inner Serde for [[Timer.key]]. + */ class TimerSerde[K](inner: Serde[K]) extends AbstractSerde[Timer[K]] { private val TimerTimeSizeBytes = Longs.BYTES @@ -29,7 +35,7 @@ class TimerSerde[K](inner: Serde[K]) extends AbstractSerde[Timer[K]] { final override def deserialize(bytes: Array[Byte]): Timer[K] = { val bb = ByteBuffer.wrap(bytes) - val time = bb.getLong() + val time = Time(bb.getLong()) val metadata = TimerMetadata(bb.get) val keyBytes = new Array[Byte](bb.remaining()) @@ -44,7 +50,7 @@ class TimerSerde[K](inner: Serde[K]) extends AbstractSerde[Timer[K]] { val timerBytes = new Array[Byte](TimerTimeSizeBytes + MetadataSizeBytes + keyBytes.length) val bb = ByteBuffer.wrap(timerBytes) - bb.putLong(timer.time) + bb.putLong(timer.time.millis) bb.put(timer.metadata.value) bb.put(keyBytes) timerBytes diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/IteratorImplicits.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/utils/IteratorImplicits.scala similarity index 98% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/IteratorImplicits.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/utils/IteratorImplicits.scala index 91022fbadf..4fe4703f54 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/IteratorImplicits.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/utils/IteratorImplicits.scala @@ -1,8 +1,9 @@ -package com.twitter.finatra.streams.transformer +package com.twitter.finatra.kafkastreams.transformer.utils import org.agrona.collections.{Hashing, Object2ObjectHashMap} import org.apache.kafka.streams.state.KeyValueIterator import scala.collection.JavaConverters._ + trait IteratorImplicits { implicit class RichIterator[T](iterator: Iterator[T]) { diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/MultiSpanIterator.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/utils/MultiSpanIterator.scala similarity index 96% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/MultiSpanIterator.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/utils/MultiSpanIterator.scala index c0f15271df..936bbe78de 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/MultiSpanIterator.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/utils/MultiSpanIterator.scala @@ -1,4 +1,4 @@ -package com.twitter.finatra.streams.transformer +package com.twitter.finatra.kafkastreams.transformer.utils /** * This Iterator will take an Iterator and split it into subiterators, where each subiterator diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/SamplingUtils.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/utils/SamplingUtils.scala similarity index 84% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/SamplingUtils.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/utils/SamplingUtils.scala index cf68035731..d0eda67774 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/SamplingUtils.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/utils/SamplingUtils.scala @@ -1,4 +1,4 @@ -package com.twitter.finatra.streams.transformer +package com.twitter.finatra.kafkastreams.transformer.utils object SamplingUtils { def getNumCountsStoreName(sampleName: String): String = { diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/watermarks/DefaultWatermarkAssignor.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/watermarks/DefaultWatermarkAssignor.scala similarity index 80% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/watermarks/DefaultWatermarkAssignor.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/watermarks/DefaultWatermarkAssignor.scala index 665c05e529..cb5d6dbf01 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/watermarks/DefaultWatermarkAssignor.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/watermarks/DefaultWatermarkAssignor.scala @@ -1,6 +1,6 @@ -package com.twitter.finatra.streams.transformer.watermarks +package com.twitter.finatra.kafkastreams.transformer.watermarks -import com.twitter.finatra.streams.transformer.domain.{Time, Watermark} +import com.twitter.finatra.kafkastreams.transformer.domain.Time import com.twitter.inject.Logging class DefaultWatermarkAssignor[K, V] extends WatermarkAssignor[K, V] with Logging { diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/PeriodicWatermarkManager.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/watermarks/PeriodicWatermarkManager.scala similarity index 86% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/PeriodicWatermarkManager.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/watermarks/PeriodicWatermarkManager.scala index 6b3f9a131d..318eda6e66 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/PeriodicWatermarkManager.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/watermarks/PeriodicWatermarkManager.scala @@ -1,4 +1,4 @@ -package com.twitter.finatra.streams.transformer +package com.twitter.finatra.kafkastreams.transformer.watermarks trait PeriodicWatermarkManager[K, V] { diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/watermarks/Watermark.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/watermarks/Watermark.scala new file mode 100644 index 0000000000..b1772cfd59 --- /dev/null +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/watermarks/Watermark.scala @@ -0,0 +1,14 @@ +package com.twitter.finatra.kafkastreams.transformer.watermarks + +import com.twitter.finatra.kafkastreams.utils.time._ + +object Watermark { + val unknown: Watermark = Watermark(0L) +} + +case class Watermark(timeMillis: Long) extends AnyVal { + + override def toString: String = { + s"Watermark(${timeMillis.iso8601Millis})" + } +} diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/watermarks/WatermarkAssignor.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/watermarks/WatermarkAssignor.scala similarity index 51% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/watermarks/WatermarkAssignor.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/watermarks/WatermarkAssignor.scala index a70ebc68c7..9753a9ea8c 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/watermarks/WatermarkAssignor.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/watermarks/WatermarkAssignor.scala @@ -1,6 +1,6 @@ -package com.twitter.finatra.streams.transformer.watermarks +package com.twitter.finatra.kafkastreams.transformer.watermarks -import com.twitter.finatra.streams.transformer.domain.{Time, Watermark} +import com.twitter.finatra.kafkastreams.transformer.domain.Time trait WatermarkAssignor[K, V] { def onMessage(topic: String, timestamp: Time, key: K, value: V): Unit diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/watermarks/WatermarkManager.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/watermarks/WatermarkManager.scala new file mode 100644 index 0000000000..7b144cc32c --- /dev/null +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/transformer/watermarks/WatermarkManager.scala @@ -0,0 +1,62 @@ +package com.twitter.finatra.kafkastreams.transformer.watermarks + +import com.twitter.finatra.kafkastreams.transformer.domain.Time +import com.twitter.finatra.kafkastreams.transformer.lifecycle.OnWatermark +import com.twitter.inject.Logging +import org.apache.kafka.streams.processor.TaskId + +/** + * WatermarkManager coordinates with a Transformers WatermarkAssignor to keep track of the latest assigned + * watermark and the last emitted watermark. + * + * @param taskId TaskId of the FinatraTransformer being managed used for internal logging + * @param transformerName Transformer name of the FinatraTransformer being managed used for internal logging + * @param onWatermark OnWatermark callback which is called when a new watermark is emitted + * @param watermarkAssignor The WatermarkAssignor used in the FinatraTransformer being managed. + * @param emitWatermarkPerMessage Whether to check if a new watermark needs to be emitted after each + * message is read in onMessage. If false, callOnWatermarkIfChanged must + * be called to check if a new watermark is to be emitted. + * @tparam K Message key for the FinatraTransformer being managed + * @tparam V Message value for the FinatraTransformer being managed + */ +class WatermarkManager[K, V]( + taskId: TaskId, + transformerName: String, + onWatermark: OnWatermark, + watermarkAssignor: WatermarkAssignor[K, V], + emitWatermarkPerMessage: Boolean) + extends Logging { + + @volatile private var lastEmittedWatermark = Watermark.unknown + + /* Public */ + + def close(): Unit = { + setLastEmittedWatermark(Watermark(0L)) + } + + def watermark: Watermark = { + lastEmittedWatermark + } + + def onMessage(messageTime: Time, topic: String, key: K, value: V): Unit = { + watermarkAssignor.onMessage(topic = topic, timestamp = messageTime, key = key, value = value) + + if (lastEmittedWatermark == Watermark.unknown || emitWatermarkPerMessage) { + callOnWatermarkIfChanged() + } + } + + def callOnWatermarkIfChanged(): Unit = { + val latestAssignedWatermark = watermarkAssignor.getWatermark + trace(s"callOnWatermarkIfChanged $transformerName $taskId $latestAssignedWatermark") + if (latestAssignedWatermark.timeMillis > lastEmittedWatermark.timeMillis) { + onWatermark.onWatermark(latestAssignedWatermark) + setLastEmittedWatermark(latestAssignedWatermark) + } + } + + protected[kafkastreams] def setLastEmittedWatermark(newWatermark: Watermark): Unit = { + lastEmittedWatermark = newWatermark + } +} diff --git a/kafka-streams/kafka-streams/src/main/scala/org/apache/kafka/streams/state/internals/RocksKeyValueIterator.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/utils/RocksKeyValueIterator.scala similarity index 96% rename from kafka-streams/kafka-streams/src/main/scala/org/apache/kafka/streams/state/internals/RocksKeyValueIterator.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/utils/RocksKeyValueIterator.scala index 64b8efda3a..5a8c913326 100644 --- a/kafka-streams/kafka-streams/src/main/scala/org/apache/kafka/streams/state/internals/RocksKeyValueIterator.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/utils/RocksKeyValueIterator.scala @@ -1,4 +1,4 @@ -package org.apache.kafka.streams.state.internals +package com.twitter.finatra.kafkastreams.utils import java.util.NoSuchElementException import org.apache.kafka.common.serialization.Deserializer diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/ScalaStreamsImplicits.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/utils/ScalaStreamsImplicits.scala similarity index 98% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/ScalaStreamsImplicits.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/utils/ScalaStreamsImplicits.scala index f0d53568dc..573d5de604 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/ScalaStreamsImplicits.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/utils/ScalaStreamsImplicits.scala @@ -1,4 +1,4 @@ -package com.twitter.finatra.kafkastreams.internal +package com.twitter.finatra.kafkastreams.utils import org.apache.kafka.streams.kstream.{Transformer, TransformerSupplier, KStream => KStreamJ} import org.apache.kafka.streams.processor.ProcessorContext diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/StatelessKafkaStreamsTwitterServer.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/utils/StatelessKafkaStreamsTwitterServer.scala similarity index 89% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/StatelessKafkaStreamsTwitterServer.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/utils/StatelessKafkaStreamsTwitterServer.scala index 8f59d6ecf9..512aa0db56 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/StatelessKafkaStreamsTwitterServer.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/utils/StatelessKafkaStreamsTwitterServer.scala @@ -1,5 +1,6 @@ -package com.twitter.finatra.kafkastreams +package com.twitter.finatra.kafkastreams.utils +import com.twitter.finatra.kafkastreams.KafkaStreamsTwitterServer import com.twitter.finatra.kafkastreams.internal.utils.TopologyReflectionUtils import org.apache.kafka.streams.Topology diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/utils/time.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/utils/time.scala new file mode 100644 index 0000000000..6d086b046d --- /dev/null +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/utils/time.scala @@ -0,0 +1,26 @@ +package com.twitter.finatra.kafkastreams.utils + +import com.twitter.finatra.kafkastreams.transformer.domain.Time +import org.joda.time.DateTime +import org.joda.time.format.ISODateTimeFormat + +/** + * Time conversion utilities. + */ +object time { + implicit class RichFinatraKafkaStreamsLong(val long: Long) extends AnyVal { + def iso8601Millis: String = { + ISODateTimeFormat.dateTime.print(long) + } + + def iso8601: String = { + ISODateTimeFormat.dateTimeNoMillis.print(long) + } + } + + implicit class RichFinatraKafkaStreamsDatetime(val datetime: DateTime) extends AnyVal { + def toTime: Time = { + Time.create(datetime) + } + } +} diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/processors/package.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/utils/utils.scala similarity index 72% rename from kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/processors/package.scala rename to kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/utils/utils.scala index 9e32f2ff97..b6e58e63b8 100644 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/processors/package.scala +++ b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/utils/utils.scala @@ -1,5 +1,5 @@ package com.twitter.finatra.kafkastreams -package object processors { +package object utils { type MessageTimestamp = Long } diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/converters/time.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/converters/time.scala deleted file mode 100644 index 3a54920473..0000000000 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/converters/time.scala +++ /dev/null @@ -1,15 +0,0 @@ -package com.twitter.finatra.streams.converters - -import org.joda.time.format.ISODateTimeFormat - -object time { - implicit class RichLong(long: Long) { - def iso8601Millis: String = { - ISODateTimeFormat.dateTime.print(long) - } - - def iso8601: String = { - ISODateTimeFormat.dateTimeNoMillis.print(long) - } - } -} diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/flags/RocksDbFlags.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/flags/RocksDbFlags.scala deleted file mode 100644 index 43e75e75f3..0000000000 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/flags/RocksDbFlags.scala +++ /dev/null @@ -1,39 +0,0 @@ -package com.twitter.finatra.streams.flags - -import com.twitter.conversions.StorageUnitOps._ -import com.twitter.finatra.kafkastreams.config.FinatraRocksDBConfig -import com.twitter.inject.server.TwitterServer - -trait RocksDbFlags extends TwitterServer { - - protected val rocksDbCountsStoreBlockCacheSize = - flag( - name = FinatraRocksDBConfig.RocksDbBlockCacheSizeConfig, - default = 200.megabytes, - help = - "Size of the rocksdb block cache per task. We recommend that this should be about 1/3 of your total memory budget. The remaining free memory can be left for the OS page cache" - ) - - protected val rocksDbEnableStatistics = - flag( - name = FinatraRocksDBConfig.RocksDbEnableStatistics, - default = false, - help = - "Enable RocksDB statistics. Note: RocksDB Statistics could add 5-10% degradation in performance (see https://github.com/facebook/rocksdb/wiki/Statistics)" - ) - - protected val rocksDbStatCollectionPeriodMs = - flag( - name = FinatraRocksDBConfig.RocksDbStatCollectionPeriodMs, - default = 60000, - help = "Set the period in milliseconds for stats collection." - ) - - protected val rocksDbEnableLZ4 = - flag( - name = FinatraRocksDBConfig.RocksDbLZ4Config, - default = false, - help = - "Enable RocksDB LZ4 compression. (See https://github.com/facebook/rocksdb/wiki/Compression)" - ) -} diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/thriftscala/WindowResultType.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/thriftscala/WindowResultType.scala deleted file mode 100644 index 563eec85aa..0000000000 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/thriftscala/WindowResultType.scala +++ /dev/null @@ -1,11 +0,0 @@ -package com.twitter.finatra.streams.thriftscala - -object WindowResultType { - @deprecated("Use com.twitter.finatra.streams.transformer.domain.WindowClosed") - object WindowClosed - extends com.twitter.finatra.streams.transformer.domain.WindowResultType( - com.twitter.finatra.streams.transformer.domain.WindowClosed.value) { - - override def toString: String = "WindowClosed" - } -} diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/CompositeSumAggregator.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/CompositeSumAggregator.scala deleted file mode 100644 index 95375ce2af..0000000000 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/CompositeSumAggregator.scala +++ /dev/null @@ -1,142 +0,0 @@ -package com.twitter.finatra.streams.transformer - -import com.twitter.finagle.stats.StatsReceiver -import com.twitter.finatra.streams.transformer.FinatraTransformer.WindowStartTime -import com.twitter.finatra.streams.transformer.domain._ -import com.twitter.util.Duration -import org.apache.kafka.streams.state.KeyValueIterator - -@deprecated("Use AggregatorTransformer", "1/7/2019") -class CompositeSumAggregator[K, A, CK <: CompositeKey[K, A]]( - commitInterval: Duration, - compositeKeyRangeStart: CK, - statsReceiver: StatsReceiver, - stateStoreName: String, - timerStoreName: String, - windowSize: Duration, - allowedLateness: Duration, - queryableAfterClose: Duration, - emitOnClose: Boolean = true, - maxActionsPerTimer: Int = 25000) - extends FinatraTransformer[ - CK, - Int, - TimeWindowed[CK], - WindowStartTime, - TimeWindowed[K], - WindowedValue[ - scala.collection.Map[A, Int] - ]](timerStoreName = timerStoreName, statsReceiver = statsReceiver, cacheTimers = true) { - - private val windowSizeMillis = windowSize.inMillis - private val allowedLatenessMillis = allowedLateness.inMillis - private val queryableAfterCloseMillis = queryableAfterClose.inMillis - - private val restatementsCounter = statsReceiver.counter("numRestatements") - private val deletesCounter = statsReceiver.counter("numDeletes") - - private val closedCounter = statsReceiver.counter("closedWindows") - private val expiredCounter = statsReceiver.counter("expiredWindows") - private val getLatencyStat = statsReceiver.stat("getLatency") - private val putLatencyStat = statsReceiver.stat("putLatency") - - private val stateStore = getKeyValueStore[TimeWindowed[CK], Int](stateStoreName) - - override def onMessage(time: Time, compositeKey: CK, count: Int): Unit = { - val windowedCompositeKey = TimeWindowed.forSize(time.hourMillis, windowSizeMillis, compositeKey) - if (windowedCompositeKey.isLate(allowedLatenessMillis, Watermark(watermark))) { - restatementsCounter.incr() - forward(windowedCompositeKey.map { _ => - compositeKey.primary - }, WindowedValue(Restatement, Map(compositeKey.secondary -> count))) - } else { - val newCount = stateStore.increment( - windowedCompositeKey, - count, - getStat = getLatencyStat, - putStat = putLatencyStat - ) - if (newCount == count) { - val closeTime = windowedCompositeKey.startMs + windowSizeMillis + allowedLatenessMillis - if (emitOnClose) { - addEventTimeTimer(Time(closeTime), Close, windowedCompositeKey.startMs) - } - addEventTimeTimer( - Time(closeTime + queryableAfterCloseMillis), - Expire, - windowedCompositeKey.startMs - ) - } - } - } - - /* - * TimeWindowedKey(2018-08-04T10:00:00.000Z-20-displayed) -> 50 - * TimeWindowedKey(2018-08-04T10:00:00.000Z-20-fav) -> 10 - * TimeWindowedKey(2018-08-04T10:00:00.000Z-30-displayed) -> 30 - * TimeWindowedKey(2018-08-04T10:00:00.000Z-40-retweet) -> 4 - */ - //Note: We use the cursor even for deletes to skip tombstones that may otherwise slow down the range scan - override def onEventTimer( - time: Time, - timerMetadata: TimerMetadata, - windowStartMs: WindowStartTime, - cursor: Option[TimeWindowed[CK]] - ): TimerResult[TimeWindowed[CK]] = { - debug(s"onEventTimer $time $timerMetadata") - val windowIterator = stateStore.range( - cursor getOrElse TimeWindowed - .forSize(windowStartMs, windowSizeMillis, compositeKeyRangeStart), - TimeWindowed.forSize(windowStartMs + 1, windowSizeMillis, compositeKeyRangeStart) - ) - - try { - if (timerMetadata == Close) { - onClosed(windowStartMs, windowIterator) - } else { - onExpired(windowIterator) - } - } finally { - windowIterator.close() - } - } - - private def onClosed( - windowStartMs: Long, - windowIterator: KeyValueIterator[TimeWindowed[CK], Int] - ): TimerResult[TimeWindowed[CK]] = { - windowIterator - .groupBy( - primaryKey = timeWindowed => timeWindowed.value.primary, - secondaryKey = timeWindowed => timeWindowed.value.secondary, - mapValue = count => count, - sharedMap = true - ) - .take(maxActionsPerTimer) - .foreach { - case (key, countsMap) => - forward( - key = TimeWindowed.forSize(windowStartMs, windowSizeMillis, key), - value = WindowedValue(resultState = WindowClosed, value = countsMap) - ) - } - - deleteOrRetainTimer(windowIterator, onDeleteTimer = closedCounter.incr()) - } - - //Note: We call "put" w/ a null value instead of calling "delete" since "delete" also gets the previous value :-/ - //TODO: Consider performing deletes in a transaction so that queryable state sees all or no keys per "primary key" - private def onExpired( - windowIterator: KeyValueIterator[TimeWindowed[CK], Int] - ): TimerResult[TimeWindowed[CK]] = { - windowIterator - .take(maxActionsPerTimer) - .foreach { - case (timeWindowedCompositeKey, count) => - deletesCounter.incr() - stateStore.put(timeWindowedCompositeKey, null.asInstanceOf[Int]) - } - - deleteOrRetainTimer(windowIterator, onDeleteTimer = expiredCounter.incr()) - } -} diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/FinatraTransformer.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/FinatraTransformer.scala deleted file mode 100644 index f40f4ac784..0000000000 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/FinatraTransformer.scala +++ /dev/null @@ -1,396 +0,0 @@ -package com.twitter.finatra.streams.transformer - -import com.google.common.annotations.Beta -import com.twitter.conversions.DurationOps._ -import com.twitter.finagle.stats.{LoadedStatsReceiver, StatsReceiver} -import com.twitter.finatra.kafkastreams.internal.utils.ProcessorContextLogging -import com.twitter.finatra.streams.config.DefaultTopicConfig -import com.twitter.finatra.streams.stores.internal.{ - FinatraKeyValueStoreImpl, - FinatraStoresGlobalManager -} -import com.twitter.finatra.streams.stores.FinatraKeyValueStore -import com.twitter.finatra.streams.stores.internal.FinatraStoresGlobalManager -import com.twitter.finatra.streams.transformer.FinatraTransformer.TimerTime -import com.twitter.finatra.streams.transformer.domain.{ - DeleteTimer, - RetainTimer, - Time, - TimerMetadata, - TimerResult -} -import com.twitter.finatra.streams.transformer.internal.domain.{Timer, TimerSerde} -import com.twitter.finatra.streams.transformer.internal.{ - OnClose, - OnInit, - ProcessorContextUtils, - StateStoreImplicits, - WatermarkTracker -} -import com.twitter.util.Duration -import org.agrona.collections.ObjectHashSet -import org.apache.kafka.common.serialization.{Serde, Serdes} -import org.apache.kafka.streams.kstream.Transformer -import org.apache.kafka.streams.processor.{ - Cancellable, - ProcessorContext, - PunctuationType, - Punctuator -} -import org.apache.kafka.streams.state.{KeyValueIterator, KeyValueStore, StoreBuilder, Stores} -import org.joda.time.DateTime -import scala.collection.JavaConverters._ -import scala.reflect.ClassTag - -object FinatraTransformer { - type TimerTime = Long - type WindowStartTime = Long - type DateTimeMillis = Long - - def timerStore[TimerKey]( - name: String, - timerKeySerde: Serde[TimerKey] - ): StoreBuilder[KeyValueStore[Timer[TimerKey], Array[Byte]]] = { - Stores - .keyValueStoreBuilder( - Stores.persistentKeyValueStore(name), - TimerSerde(timerKeySerde), - Serdes.ByteArray - ) - .withLoggingEnabled(DefaultTopicConfig.FinatraChangelogConfig) - } -} - -/** - * A KafkaStreams Transformer supporting Per-Key Persistent Timers - * Inspired by Flink's ProcessFunction: https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/process_function.html - * - * Note: Timers are based on a sorted RocksDB KeyValueStore - * Note: Timers that fire at the same time MAY NOT fire in the order which they were added - * - * Example Timer Key Structures (w/ corresponding CountsStore Key Structures) - * {{{ - * ImpressionsCounter (w/ TimerKey storing TweetId) - * TimeWindowedKey(2018-08-04T10:00:00.000Z-20) - * Timer( 2018-08-04T12:00:00.000Z-Expire-2018-08-04T10:00:00.000Z-20 - * TimeWindowedKey(2018-08-04T10:00:00.000Z-30) - * Timer( 2018-08-04T12:00:00.000Z-Expire-2018-08-04T10:00:00.000Z-30 - * - * ImpressionsCounter (w/ TimerKey storing windowStartMs) - * TimeWindowedKey(2018-08-04T10:00:00.000Z-20) - * TimeWindowedKey(2018-08-04T10:00:00.000Z-30) - * TimeWindowedKey(2018-08-04T10:00:00.000Z-40) - * TimeWindowedKey(2018-08-04T11:00:00.000Z-20) - * TimeWindowedKey(2018-08-04T11:00:00.000Z-30) - * Timer( 2018-08-04T12:00:00.000Z-Expire-2018-08-04T10:00:00.000Z - * Timer( 2018-08-04T13:00:00.000Z-Expire-2018-08-04T11:00:00.000Z - * - * EngagementCounter (w/ TimerKey storing windowStartMs) - * TimeWindowedKey(2018-08-04T10:00:00.000Z-20-displayed) -> 5 - * TimeWindowedKey(2018-08-04T10:00:00.000Z-20-fav) -> 10 - * Timer( 2018-08-04T12:00:00.000Z-Expire-2018-08-04T10:00:00.000Z - * - * @tparam InputKey Type of the input keys - * @tparam InputValue Type of the input values - * @tparam StoreKey Type of the key being stored in the state store (needed to support onEventTimer cursoring) - * @tparam TimerKey Type of the timer key - * @tparam OutputKey Type of the output keys - * @tparam OutputValue Type of the output values - * }}} - */ -//TODO: Create variant for when there are no timers (e.g. avoid the extra time params and need to specify a timer store -@Beta -abstract class FinatraTransformer[InputKey, InputValue, StoreKey, TimerKey, OutputKey, OutputValue]( - commitInterval: Duration = null, //TODO: This field is currently only used by one external customer (but unable to @deprecate a constructor param). Will remove from caller and here in followup Phab. - cacheTimers: Boolean = true, - throttlingResetDuration: Duration = 3.seconds, - disableTimers: Boolean = false, - timerStoreName: String, - statsReceiver: StatsReceiver = LoadedStatsReceiver) //TODO - extends Transformer[InputKey, InputValue, (OutputKey, OutputValue)] - with OnInit - with OnClose - with StateStoreImplicits - with IteratorImplicits - with ProcessorContextLogging { - - /* Private Mutable */ - - @volatile private var _context: ProcessorContext = _ - @volatile private var cancellableThrottlingResetTimer: Cancellable = _ - @volatile private var processingTimerCancellable: Cancellable = _ - @volatile private var nextTimer: Long = Long.MaxValue //Maintain to avoid iterating timerStore every time fireTimers is called - - //TODO: Persist cursor in stateStore to avoid duplicate cursored work after a restart - @volatile private var throttled: Boolean = false - @volatile private var lastThrottledCursor: Option[StoreKey] = None - - /* Private */ - - private val watermarkTracker = new WatermarkTracker - private val cachedTimers = new ObjectHashSet[Timer[TimerKey]](16) - private val finatraKeyValueStores = - scala.collection.mutable.Map[String, FinatraKeyValueStore[_, _]]() - - protected[finatra] final val timersStore = if (disableTimers) { - null - } else { - getKeyValueStore[Timer[TimerKey], Array[Byte]](timerStoreName) - } - - /* Abstract */ - - protected[finatra] def onMessage(messageTime: Time, key: InputKey, value: InputValue): Unit - - protected def onProcessingTimer(time: TimerTime): Unit = {} - - /** - * Callback for when an Event timer is ready for processing - * - * @return TimerResult indicating if this timer should be retained or deleted - */ - protected def onEventTimer( - time: Time, - metadata: TimerMetadata, - key: TimerKey, - cursor: Option[StoreKey] - ): TimerResult[StoreKey] = { - warn(s"Unhandled timer $time $metadata $key") - DeleteTimer() - } - - /* Protected */ - - final override def init(processorContext: ProcessorContext): Unit = { - _context = processorContext - - for ((name, store) <- finatraKeyValueStores) { - store.init(processorContext, null) - } - - if (!disableTimers) { - cancellableThrottlingResetTimer = _context - .schedule( - throttlingResetDuration.inMillis, - PunctuationType.WALL_CLOCK_TIME, - new Punctuator { - override def punctuate(timestamp: TimerTime): Unit = { - resetThrottled() - fireEventTimeTimers() - } - } - ) - - findAndSetNextTimer() - cacheTimersIfEnabled() - } - - onInit() - } - - override protected def processorContext: ProcessorContext = _context - - final override def transform(k: InputKey, v: InputValue): (OutputKey, OutputValue) = { - if (watermarkTracker.track(_context.topic(), _context.timestamp)) { - fireEventTimeTimers() - } - - debug(s"onMessage ${_context.timestamp.iso8601Millis} $k $v") - onMessage(Time(_context.timestamp()), k, v) - - null - } - - final override def close(): Unit = { - setNextTimerTime(0) - cachedTimers.clear() - watermarkTracker.reset() - - if (cancellableThrottlingResetTimer != null) { - cancellableThrottlingResetTimer.cancel() - cancellableThrottlingResetTimer = null - } - - if (processingTimerCancellable != null) { - processingTimerCancellable.cancel() - processingTimerCancellable = null - } - - for ((name, store) <- finatraKeyValueStores) { - store.close() - FinatraStoresGlobalManager.removeStore(store) - } - - onClose() - } - - final protected def getKeyValueStore[KK: ClassTag, VV]( - name: String - ): FinatraKeyValueStore[KK, VV] = { - val store = new FinatraKeyValueStoreImpl[KK, VV](name, statsReceiver) - val previousStore = finatraKeyValueStores.put(name, store) - FinatraStoresGlobalManager.addStore(store) - assert(previousStore.isEmpty, s"getKeyValueStore was called for store $name more than once") - - // Initialize stores that are still using the "lazy val store" pattern - if (processorContext != null) { - store.init(processorContext, null) - } - - store - } - - //TODO: Add a forwardOnCommit which just takes a key - final protected def forward(key: OutputKey, value: OutputValue): Unit = { - trace(f"${"Forward:"}%-20s $key $value") - _context.forward(key, value) - } - - final protected def forward(key: OutputKey, value: OutputValue, timestamp: Long): Unit = { - trace(f"${"Forward:"}%-20s $key $value @${new DateTime(timestamp)}") - ProcessorContextUtils.setTimestamp(_context, timestamp) - _context.forward(key, value) - } - - final protected def watermark: Long = { - watermarkTracker.watermark - } - - final protected def addEventTimeTimer( - time: Time, - metadata: TimerMetadata, - key: TimerKey - ): Unit = { - trace( - f"${"AddEventTimer:"}%-20s ${metadata.getClass.getSimpleName}%-12s Key $key Timer ${time.millis.iso8601Millis}" - ) - val timer = Timer(time = time.millis, metadata = metadata, key = key) - if (cacheTimers && cachedTimers.contains(timer)) { - trace(s"Deduped unkeyed timer: $timer") - } else { - timersStore.put(timer, Array.emptyByteArray) - if (time.millis < nextTimer) { - setNextTimerTime(time.millis) - } - if (cacheTimers) { - cachedTimers.add(timer) - } - } - } - - final protected def addProcessingTimeTimer(duration: Duration): Unit = { - assert( - processingTimerCancellable == null, - "NonPersistentProcessingTimer already set. We currently only support a single processing timer being set through addProcessingTimeTimer." - ) - processingTimerCancellable = - processorContext.schedule(duration.inMillis, PunctuationType.WALL_CLOCK_TIME, new Punctuator { - override def punctuate(time: Long): Unit = { - onProcessingTimer(time) - } - }) - } - - final protected def deleteOrRetainTimer( - iterator: KeyValueIterator[StoreKey, _], - onDeleteTimer: => Unit = () => () - ): TimerResult[StoreKey] = { - if (iterator.hasNext) { - RetainTimer(stateStoreCursor = iterator.peekNextKeyOpt, throttled = true) - } else { - onDeleteTimer - DeleteTimer() - } - } - - /* Private */ - - private def fireEventTimeTimers(): Unit = { - trace( - s"FireTimers watermark ${watermark.iso8601Millis} nextTimer ${nextTimer.iso8601Millis} throttled $throttled" - ) - if (!disableTimers && !isThrottled && watermark >= nextTimer) { - val timerIterator = timersStore.all() - try { - timerIterator.asScala - .takeWhile { timerAndEmptyValue => - !isThrottled && watermark >= timerAndEmptyValue.key.time - } - .foreach { timerAndEmptyValue => - fireEventTimeTimer(timerAndEmptyValue.key) - } - } finally { - timerIterator.close() - findAndSetNextTimer() //TODO: Optimize by avoiding the need to re-read from the timersStore iterator - } - } - } - - //Note: LastThrottledCursor is shared per Task. However, since the timers are sorted, we should only be cursoring the head timer at a time. - private def fireEventTimeTimer(timer: Timer[TimerKey]): Unit = { - trace( - s"fireEventTimeTimer ${timer.metadata.getClass.getName} key: ${timer.key} timerTime: ${timer.time.iso8601Millis}" - ) - - onEventTimer( - time = Time(timer.time), - metadata = timer.metadata, - key = timer.key, - lastThrottledCursor - ) match { - case DeleteTimer(throttledResult) => - lastThrottledCursor = None - throttled = throttledResult - - timersStore.deleteWithoutGettingPriorValue(timer) - if (cacheTimers) { - cachedTimers.remove(timer) - } - case RetainTimer(stateStoreCursor, throttledResult) => - lastThrottledCursor = stateStoreCursor - throttled = throttledResult - } - } - - private def findAndSetNextTimer(): Unit = { - val iterator = timersStore.all() - try { - if (iterator.hasNext) { - setNextTimerTime(iterator.peekNextKey.time) - } else { - setNextTimerTime(Long.MaxValue) - } - } finally { - iterator.close() - } - } - - private def setNextTimerTime(time: TimerTime): Unit = { - nextTimer = time - if (time != Long.MaxValue) { - trace(s"NextTimer: ${nextTimer.iso8601Millis}") - } - } - - private def cacheTimersIfEnabled(): Unit = { - if (cacheTimers) { - val iterator = timersStore.all() - try { - for (timerKeyValue <- iterator.asScala) { - val timer = timerKeyValue.key - cachedTimers.add(timer) - } - } finally { - iterator.close() - } - } - } - - private def resetThrottled(): Unit = { - throttled = false - } - - private def isThrottled: Boolean = { - throttled - } -} diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/OnWatermark.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/OnWatermark.scala deleted file mode 100644 index 249198150a..0000000000 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/OnWatermark.scala +++ /dev/null @@ -1,7 +0,0 @@ -package com.twitter.finatra.streams.transformer - -import com.twitter.finatra.streams.transformer.domain.Watermark - -trait OnWatermark { - def onWatermark(watermark: Watermark): Unit -} diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/SumAggregator.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/SumAggregator.scala deleted file mode 100644 index 3e7975232f..0000000000 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/SumAggregator.scala +++ /dev/null @@ -1,114 +0,0 @@ -package com.twitter.finatra.streams.transformer - -import com.twitter.finagle.stats.StatsReceiver -import com.twitter.finatra.streams.transformer.FinatraTransformer.WindowStartTime -import com.twitter.finatra.streams.transformer.domain._ -import com.twitter.util.Duration -import org.apache.kafka.streams.state.KeyValueIterator - -@deprecated("Use AggregatorTransformer") -class SumAggregator[K, V]( - commitInterval: Duration, - keyRangeStart: K, - statsReceiver: StatsReceiver, - stateStoreName: String, - timerStoreName: String, - windowSize: Duration, - allowedLateness: Duration, - queryableAfterClose: Duration, - windowStart: (Time, K, V) => Long, - countToAggregate: (K, V) => Int, - emitOnClose: Boolean = true, - maxActionsPerTimer: Int = 25000) - extends FinatraTransformer[ - K, - V, - TimeWindowed[K], - WindowStartTime, - TimeWindowed[K], - WindowedValue[ - Int - ]](timerStoreName = timerStoreName, statsReceiver = statsReceiver, cacheTimers = true) { - - private val windowSizeMillis = windowSize.inMillis - private val allowedLatenessMillis = allowedLateness.inMillis - private val queryableAfterCloseMillis = queryableAfterClose.inMillis - - private val restatementsCounter = statsReceiver.counter("numRestatements") - private val closedCounter = statsReceiver.counter("closedWindows") - private val expiredCounter = statsReceiver.counter("expiredWindows") - - private val stateStore = getKeyValueStore[TimeWindowed[K], Int](stateStoreName) - - override def onMessage(time: Time, key: K, value: V): Unit = { - val windowedKey = TimeWindowed.forSize( - startMs = windowStart(time, key, value), - sizeMs = windowSizeMillis, - value = key - ) - - val count = countToAggregate(key, value) - if (windowedKey.isLate(allowedLatenessMillis, Watermark(watermark))) { - restatementsCounter.incr() - forward(windowedKey, WindowedValue(Restatement, count)) - } else { - val newCount = stateStore.increment(windowedKey, count) - if (newCount == count) { - val closeTime = windowedKey.startMs + windowSizeMillis + allowedLatenessMillis - if (emitOnClose) { - addEventTimeTimer(Time(closeTime), Close, windowedKey.startMs) - } - addEventTimeTimer(Time(closeTime + queryableAfterCloseMillis), Expire, windowedKey.startMs) - } - } - } - - override def onEventTimer( - time: Time, - timerMetadata: TimerMetadata, - windowStartMs: WindowStartTime, - cursor: Option[TimeWindowed[K]] - ): TimerResult[TimeWindowed[K]] = { - val hourlyWindowIterator = stateStore.range( - cursor getOrElse TimeWindowed.forSize(windowStartMs, windowSizeMillis, keyRangeStart), - TimeWindowed.forSize(windowStartMs + 1, windowSizeMillis, keyRangeStart) - ) - - try { - if (timerMetadata == Close) { - onClosed(windowStartMs, hourlyWindowIterator) - } else { - onExpired(hourlyWindowIterator) - } - } finally { - hourlyWindowIterator.close() - } - } - - private def onClosed( - windowStartMs: Long, - windowIterator: KeyValueIterator[TimeWindowed[K], Int] - ): TimerResult[TimeWindowed[K]] = { - windowIterator - .take(maxActionsPerTimer) - .foreach { - case (key, value) => - forward(key = key, value = WindowedValue(resultState = WindowClosed, value = value)) - } - - deleteOrRetainTimer(windowIterator, closedCounter.incr()) - } - - private def onExpired( - windowIterator: KeyValueIterator[TimeWindowed[K], Int] - ): TimerResult[TimeWindowed[K]] = { - windowIterator - .take(maxActionsPerTimer) - .foreach { - case (key, value) => - stateStore.deleteWithoutGettingPriorValue(key) - } - - deleteOrRetainTimer(windowIterator, expiredCounter.incr()) - } -} diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/domain/Time.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/domain/Time.scala deleted file mode 100644 index b0edf56ba2..0000000000 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/domain/Time.scala +++ /dev/null @@ -1,36 +0,0 @@ -package com.twitter.finatra.streams.transformer.domain - -import com.twitter.util.Duration -import org.joda.time.DateTimeConstants -import com.twitter.finatra.streams.converters.time._ - -object Time { - def nextInterval(time: Long, duration: Duration): Long = { - val durationMillis = duration.inMillis - val currentNumIntervals = time / durationMillis - (currentNumIntervals + 1) * durationMillis - } -} - -//TODO: Refactor -case class Time(millis: Long) extends AnyVal { - - final def plus(duration: Duration): Time = { - new Time(millis + duration.inMillis) - } - - final def hourMillis: Long = { - val unitsSinceEpoch = millis / DateTimeConstants.MILLIS_PER_HOUR - unitsSinceEpoch * DateTimeConstants.MILLIS_PER_HOUR - } - - final def hourlyWindowed[K](key: K): TimeWindowed[K] = { - val start = hourMillis - val end = start + DateTimeConstants.MILLIS_PER_HOUR - TimeWindowed(start, end, key) - } - - override def toString: String = { - s"Time(${millis.iso8601Millis})" - } -} diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/domain/TimeWindowed.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/domain/TimeWindowed.scala deleted file mode 100644 index 195d367540..0000000000 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/domain/TimeWindowed.scala +++ /dev/null @@ -1,86 +0,0 @@ -package com.twitter.finatra.streams.transformer.domain - -import com.twitter.util.Duration -import org.joda.time.{DateTime, DateTimeConstants} - -object TimeWindowed { - - def forSize[V](startMs: Long, sizeMs: Long, value: V): TimeWindowed[V] = { - TimeWindowed(startMs, startMs + sizeMs, value) - } - - def forSizeFromMessageTime[V](messageTime: Time, sizeMs: Long, value: V): TimeWindowed[V] = { - val windowStartMs = windowStart(messageTime, sizeMs) - TimeWindowed(windowStartMs, windowStartMs + sizeMs, value) - } - - def hourly[V](startMs: Long, value: V): TimeWindowed[V] = { - TimeWindowed(startMs, startMs + DateTimeConstants.MILLIS_PER_HOUR, value) - } - - def windowStart(messageTime: Time, sizeMs: Long): Long = { - (messageTime.millis / sizeMs) * sizeMs - } -} - -/** - * A time windowed value specified by a start and end time - * @param startMs the start timestamp of the window (inclusive) - * @param endMs the end timestamp of the window (exclusive) - */ -case class TimeWindowed[V](startMs: Long, endMs: Long, value: V) { - - /** - * Determine if this windowed value is late given the allowedLateness configuration and the - * current watermark - * - * @param allowedLateness the configured amount of allowed lateness specified in milliseconds - * @param watermark a watermark used to determine if this windowed value is late - * @return If the windowed value is late - */ - def isLate(allowedLateness: Long, watermark: Watermark): Boolean = { - watermark.timeMillis > endMs + allowedLateness - } - - /** - * Determine the start of the next fixed window interval - */ - def nextInterval(time: Long, duration: Duration): Long = { - val intervalStart = math.max(startMs, time) - Time.nextInterval(intervalStart, duration) - } - - /** - * Map the time windowed value into another value occurring in the same window - */ - def map[KK](f: V => KK): TimeWindowed[KK] = { - copy(value = f(value)) - } - - /** - * The size of this windowed value in milliseconds - */ - def sizeMillis: Long = endMs - startMs - - final override val hashCode: Int = { - var result = value.hashCode() - result = 31 * result + (startMs ^ (startMs >>> 32)).toInt - result = 31 * result + (endMs ^ (endMs >>> 32)).toInt - result - } - - final override def equals(obj: scala.Any): Boolean = { - obj match { - case other: TimeWindowed[V] => - startMs == other.startMs && - endMs == other.endMs && - value == other.value - case _ => - false - } - } - - override def toString: String = { - s"TimeWindowed(${new DateTime(startMs)}-${new DateTime(endMs)}-$value)" - } -} diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/domain/Watermark.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/domain/Watermark.scala deleted file mode 100644 index 3357d136ea..0000000000 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/domain/Watermark.scala +++ /dev/null @@ -1,10 +0,0 @@ -package com.twitter.finatra.streams.transformer.domain - -import com.twitter.finatra.streams.converters.time._ - -case class Watermark(timeMillis: Long) extends AnyVal { - - override def toString: String = { - s"Watermark(${timeMillis.iso8601Millis})" - } -} diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/domain/WindowedValue.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/domain/WindowedValue.scala deleted file mode 100644 index 9a619e8817..0000000000 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/domain/WindowedValue.scala +++ /dev/null @@ -1,9 +0,0 @@ -package com.twitter.finatra.streams.transformer.domain - -//TODO: Rename resultState to WindowResultType -case class WindowedValue[V](resultState: WindowResultType, value: V) { - - def map[VV](f: V => VV): WindowedValue[VV] = { - copy(value = f(value)) - } -} diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/domain/timerResults.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/domain/timerResults.scala deleted file mode 100644 index 78aee918ab..0000000000 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/domain/timerResults.scala +++ /dev/null @@ -1,33 +0,0 @@ -package com.twitter.finatra.streams.transformer.domain - -/** - * Indicates the result of a Timer-based operation. - */ -sealed trait TimerResult[SK] { - def map[SKR](f: SK => SKR): TimerResult[SKR] = { - this match { - case result @ RetainTimer(Some(cursor), throttled) => - result.copy(stateStoreCursor = Some(f(cursor))) - case _ => - this.asInstanceOf[TimerResult[SKR]] - } - } -} - -/** - * A [[TimerResult]] that represents the completion of a deletion. - * - * @param throttled Indicates the number of operations has surpassed those allocated - * for a period of time. - */ -case class DeleteTimer[SK](throttled: Boolean = false) extends TimerResult[SK] - -/** - * A [[TimerResult]] that represents the retention of an incomplete deletion. - * - * @param stateStoreCursor A cursor representing the next key in an iterator. - * @param throttled Indicates the number of operations has surpassed those allocated - * for a period of time. - */ -case class RetainTimer[SK](stateStoreCursor: Option[SK] = None, throttled: Boolean = false) - extends TimerResult[SK] diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/internal/OnClose.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/internal/OnClose.scala deleted file mode 100644 index 960d7fac6b..0000000000 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/internal/OnClose.scala +++ /dev/null @@ -1,5 +0,0 @@ -package com.twitter.finatra.streams.transformer.internal - -trait OnClose { - protected def onClose(): Unit = {} -} diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/internal/OnInit.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/internal/OnInit.scala deleted file mode 100644 index 9070439f5b..0000000000 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/internal/OnInit.scala +++ /dev/null @@ -1,5 +0,0 @@ -package com.twitter.finatra.streams.transformer.internal - -trait OnInit { - protected def onInit(): Unit = {} -} diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/internal/StateStoreImplicits.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/internal/StateStoreImplicits.scala deleted file mode 100644 index 08088cdaeb..0000000000 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/internal/StateStoreImplicits.scala +++ /dev/null @@ -1,45 +0,0 @@ -package com.twitter.finatra.streams.transformer.internal - -import com.twitter.finagle.stats.Stat -import com.twitter.finatra.kafkastreams.internal.utils.ProcessorContextLogging -import com.twitter.util.Stopwatch -import org.apache.kafka.streams.state.KeyValueStore - -trait StateStoreImplicits extends ProcessorContextLogging { - - /* ------------------------------------------ */ - implicit class RichKeyIntValueStore[SK](keyValueStore: KeyValueStore[SK, Int]) { - - /** - * @return the new value associated with the specified key - */ - final def increment(key: SK, amount: Int): Int = { - val existingCount = keyValueStore.get(key) - val newCount = existingCount + amount - trace(s"keyValueStore.put($key, $newCount)") - keyValueStore.put(key, newCount) - newCount - } - - /** - * @return the new value associated with the specified key - */ - final def increment(key: SK, amount: Int, getStat: Stat, putStat: Stat): Int = { - val getElapsed = Stopwatch.start() - val existingCount = keyValueStore.get(key) - val getElapsedMillis = getElapsed.apply().inMillis - getStat.add(getElapsedMillis) - if (getElapsedMillis > 10) { - warn(s"SlowGet $getElapsedMillis ms for key $key") - } - - val newCount = existingCount + amount - - val putElapsed = Stopwatch.start() - keyValueStore.put(key, newCount) - putStat.add(putElapsed.apply().inMillis) - - newCount - } - } -} diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/internal/WatermarkTracker.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/internal/WatermarkTracker.scala deleted file mode 100644 index 010dc56696..0000000000 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/internal/WatermarkTracker.scala +++ /dev/null @@ -1,32 +0,0 @@ -package com.twitter.finatra.streams.transformer.internal - -//TODO: Need method called by processing timer so that watermarks can be emitted without input records -class WatermarkTracker { - private var _watermark: Long = 0L - reset() - - def watermark: Long = _watermark - - def reset(): Unit = { - _watermark = 0L - } - - /** - * @param timestamp - * - * @return True if watermark changed - */ - //TODO: Verify topic is correct when merging inputs - //TODO: Also take in deserialized key and value since we can extract source info (e.g. source of interactions) - //TODO: Also take in maxOutOfOrder param - //TODO: Use rolling histogram - def track(topic: String, timestamp: Long): Boolean = { - val potentialWatermark = timestamp - 1 - if (potentialWatermark > _watermark) { - _watermark = potentialWatermark - true - } else { - false - } - } -} diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/internal/domain/Timer.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/internal/domain/Timer.scala deleted file mode 100644 index 0c48bbef40..0000000000 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/internal/domain/Timer.scala +++ /dev/null @@ -1,14 +0,0 @@ -package com.twitter.finatra.streams.transformer.internal.domain - -import com.twitter.finatra.streams.converters.time._ -import com.twitter.finatra.streams.transformer.domain.TimerMetadata - -/** - * @param time Time to fire the timer - */ -case class Timer[K](time: Long, metadata: TimerMetadata, key: K) { - - override def toString: String = { - s"Timer(${metadata.getClass.getName} $key @${time.iso8601Millis})" - } -} diff --git a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/watermarks/internal/WatermarkManager.scala b/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/watermarks/internal/WatermarkManager.scala deleted file mode 100644 index 669080f36c..0000000000 --- a/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/watermarks/internal/WatermarkManager.scala +++ /dev/null @@ -1,45 +0,0 @@ -package com.twitter.finatra.streams.transformer.watermarks.internal - -import com.twitter.finatra.streams.transformer.OnWatermark -import com.twitter.finatra.streams.transformer.domain.{Time, Watermark} -import com.twitter.finatra.streams.transformer.watermarks.WatermarkAssignor -import com.twitter.inject.Logging - -class WatermarkManager[K, V]( - onWatermark: OnWatermark, - watermarkAssignor: WatermarkAssignor[K, V], - emitWatermarkPerMessage: Boolean) - extends Logging { - - @volatile private var lastEmittedWatermark = Watermark(0L) - - /* Public */ - - def close(): Unit = { - setLastEmittedWatermark(Watermark(0L)) - } - - def watermark: Watermark = { - lastEmittedWatermark - } - - def onMessage(messageTime: Time, topic: String, key: K, value: V): Unit = { - watermarkAssignor.onMessage(topic = topic, timestamp = messageTime, key = key, value = value) - - if (emitWatermarkPerMessage) { - callOnWatermarkIfChanged() - } - } - - def callOnWatermarkIfChanged(): Unit = { - val currentWatermark = watermarkAssignor.getWatermark - if (currentWatermark.timeMillis > lastEmittedWatermark.timeMillis) { - onWatermark.onWatermark(currentWatermark) - setLastEmittedWatermark(currentWatermark) - } - } - - protected[streams] def setLastEmittedWatermark(newWatermark: Watermark): Unit = { - lastEmittedWatermark = newWatermark - } -} diff --git a/kafka-streams/kafka-streams/src/main/scala/org/apache/kafka/streams/state/internals/FinatraAbstractStoreBuilder.scala b/kafka-streams/kafka-streams/src/main/scala/org/apache/kafka/streams/state/internals/FinatraAbstractStoreBuilder.scala deleted file mode 100644 index cd7578a65a..0000000000 --- a/kafka-streams/kafka-streams/src/main/scala/org/apache/kafka/streams/state/internals/FinatraAbstractStoreBuilder.scala +++ /dev/null @@ -1,13 +0,0 @@ -package org.apache.kafka.streams.state.internals - -import org.apache.kafka.common.serialization.Serde -import org.apache.kafka.common.utils.Time -import org.apache.kafka.streams.processor.StateStore - -/* Note: To avoid code duplication for now, this class is created for access to package protected AbstractStoreBuilder */ -abstract class FinatraAbstractStoreBuilder[K, V, T <: StateStore]( - name: String, - keySerde: Serde[K], - valueSerde: Serde[V], - time: Time) - extends AbstractStoreBuilder[K, V, T](name, keySerde, valueSerde, time) diff --git a/kafka-streams/kafka-streams/src/main/scala/org/apache/kafka/streams/state/internals/InMemoryKeyValueFlushingStoreBuilder.scala b/kafka-streams/kafka-streams/src/main/scala/org/apache/kafka/streams/state/internals/InMemoryKeyValueFlushingStoreBuilder.scala deleted file mode 100644 index 35240489ea..0000000000 --- a/kafka-streams/kafka-streams/src/main/scala/org/apache/kafka/streams/state/internals/InMemoryKeyValueFlushingStoreBuilder.scala +++ /dev/null @@ -1,20 +0,0 @@ -package org.apache.kafka.streams.state.internals - -import org.apache.kafka.common.serialization.Serde -import org.apache.kafka.common.utils.Time -import org.apache.kafka.streams.state.KeyValueStore - -class InMemoryKeyValueFlushingStoreBuilder[K, V]( - name: String, - keySerde: Serde[K], - valueSerde: Serde[V], - time: Time = Time.SYSTEM) - extends FinatraAbstractStoreBuilder[K, V, KeyValueStore[K, V]](name, keySerde, valueSerde, time) { - - override def build(): KeyValueStore[K, V] = { - val inMemoryKeyValueStore = new InMemoryKeyValueStore[K, V](name, keySerde, valueSerde) - val inMemoryFlushingKeyValueStore = - new InMemoryKeyValueFlushingLoggedStore[K, V](inMemoryKeyValueStore, keySerde, valueSerde) - new MeteredKeyValueStore[K, V](inMemoryFlushingKeyValueStore, "in-memory-state", time) - } -} diff --git a/kafka-streams/kafka-streams/src/test/scala/BUILD b/kafka-streams/kafka-streams/src/test/scala/BUILD index a0a7124dd8..edf5c0e1df 100644 --- a/kafka-streams/kafka-streams/src/test/scala/BUILD +++ b/kafka-streams/kafka-streams/src/test/scala/BUILD @@ -1,6 +1,82 @@ -target( +scala_library( name = "test-deps", + sources = globs( + "com/twitter/finatra/kafkastreams/test/*.scala", + "com/twitter/inject/*.scala", + ), + compiler_option_sets = {"fatal_warnings"}, + provides = scala_artifact( + org = "com.twitter", + name = "finatra-streams-tests", + repo = artifactory, + ), + strict_deps = False, dependencies = [ - "finatra/kafka-streams/kafka-streams/src/test/scala/com/twitter:test-deps", + "3rdparty/jvm/com/google/inject:guice", + "3rdparty/jvm/junit", + "3rdparty/jvm/org/apache/kafka", + "3rdparty/jvm/org/apache/kafka:kafka-clients", + "3rdparty/jvm/org/apache/kafka:kafka-clients-test", + "3rdparty/jvm/org/apache/kafka:kafka-streams-test", + "3rdparty/jvm/org/apache/kafka:kafka-streams-test-utils", + "3rdparty/jvm/org/apache/kafka:kafka-test", + "3rdparty/jvm/org/scalatest", + "finatra/inject/inject-core/src/main/scala", + "finatra/inject/inject-core/src/test/scala:test-deps", + "finatra/inject/inject-server/src/test/scala:test-deps", + "finatra/inject/inject-slf4j/src/main/scala", + "finatra/jackson/src/main/scala", + "finatra/kafka-streams/kafka-streams/src/main/scala", + "finatra/kafka/src/test/scala:test-deps", + "util/util-slf4j-api/src/main/scala", + ], + exports = [ + "3rdparty/jvm/com/google/inject:guice", + "3rdparty/jvm/junit", + "3rdparty/jvm/org/apache/kafka", + "3rdparty/jvm/org/apache/kafka:kafka-clients", + "3rdparty/jvm/org/apache/kafka:kafka-clients-test", + "3rdparty/jvm/org/apache/kafka:kafka-streams-test", + "3rdparty/jvm/org/apache/kafka:kafka-streams-test-utils", + "3rdparty/jvm/org/apache/kafka:kafka-test", + "3rdparty/jvm/org/scalatest", + "finatra/inject/inject-core/src/main/scala", + "finatra/inject/inject-core/src/test/scala:test-deps", + "finatra/inject/inject-server/src/test/scala:test-deps", + "finatra/inject/inject-slf4j/src/main/scala", + "finatra/jackson/src/main/scala", + "finatra/kafka-streams/kafka-streams/src/main/scala", + "finatra/kafka/src/test/scala:test-deps", + "util/util-slf4j-api/src/main/scala", + ], +) + +junit_tests( + sources = rglobs( + "com/twitter/finatra/kafkastreams/integration/*.scala", + "com/twitter/finatra/kafkastreams/transformer/*.scala", + ), + compiler_option_sets = {"fatal_warnings"}, + strict_deps = False, + dependencies = [ + "3rdparty/jvm/ch/qos/logback:logback-classic", + "3rdparty/jvm/org/apache/kafka:kafka-clients-test", + "3rdparty/jvm/org/apache/kafka:kafka-streams", + "3rdparty/jvm/org/apache/kafka:kafka-streams-test", + "3rdparty/jvm/org/apache/kafka:kafka-streams-test-utils", + "3rdparty/jvm/org/apache/kafka:kafka-test", + "3rdparty/jvm/org/apache/zookeeper:zookeeper-client", + "3rdparty/jvm/org/apache/zookeeper:zookeeper-server", + "finatra/inject/inject-app/src/main/scala", + "finatra/inject/inject-core/src/main/scala", + "finatra/inject/inject-core/src/test/scala:test-deps", + "finatra/inject/inject-server/src/main/scala", + "finatra/inject/inject-server/src/test/scala:test-deps", + "finatra/inject/inject-slf4j/src/main/scala", + "finatra/kafka-streams/kafka-streams/src/main/scala", + "finatra/kafka-streams/kafka-streams/src/test/resources", + "finatra/kafka-streams/kafka-streams/src/test/scala:test-deps", + "finatra/kafka/src/test/scala:test-deps", + "finatra/thrift/src/test/scala:test-deps", ], ) diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/BUILD b/kafka-streams/kafka-streams/src/test/scala/com/twitter/BUILD deleted file mode 100644 index ddad30417c..0000000000 --- a/kafka-streams/kafka-streams/src/test/scala/com/twitter/BUILD +++ /dev/null @@ -1,66 +0,0 @@ -scala_library( - name = "test-deps", - sources = globs( - "finatra/kafkastreams/test/*.scala", - "finatra/streams/tests/*.scala", - "inject/*.scala", - ), - compiler_option_sets = {"fatal_warnings"}, - provides = scala_artifact( - org = "com.twitter", - name = "finatra-streams-tests", - repo = artifactory, - ), - strict_deps = False, - dependencies = [ - "3rdparty/jvm/com/google/inject:guice", - "3rdparty/jvm/junit", - "3rdparty/jvm/org/apache/kafka", - "3rdparty/jvm/org/apache/kafka:kafka-clients", - "3rdparty/jvm/org/apache/kafka:kafka-clients-test", - "3rdparty/jvm/org/apache/kafka:kafka-streams-test", - "3rdparty/jvm/org/apache/kafka:kafka-streams-test-utils", - "3rdparty/jvm/org/apache/kafka:kafka-test", - "3rdparty/jvm/org/scalatest", - "finatra/inject/inject-core/src/main/scala", - "finatra/inject/inject-core/src/test/scala:test-deps", - "finatra/inject/inject-server/src/test/scala:test-deps", - "finatra/inject/inject-slf4j/src/main/scala", - "finatra/jackson/src/main/scala", - "finatra/kafka-streams/kafka-streams/src/main/scala", - "finatra/kafka/src/test/scala:test-deps", - "util/util-slf4j-api/src/main/scala", - ], - excludes = [ - exclude( - org = "com.twitter", - name = "twitter-server-internal-naming_2.11", - ), - exclude( - org = "com.twitter", - name = "loglens-log4j-logging_2.11", - ), - exclude( - org = "log4j", - name = "log4j", - ), - ], - exports = [ - "3rdparty/jvm/com/google/inject:guice", - "3rdparty/jvm/junit", - "3rdparty/jvm/org/apache/kafka", - "3rdparty/jvm/org/apache/kafka:kafka-clients", - "3rdparty/jvm/org/apache/kafka:kafka-clients-test", - "3rdparty/jvm/org/apache/kafka:kafka-streams-test", - "3rdparty/jvm/org/apache/kafka:kafka-test", - "3rdparty/jvm/org/scalatest", - "finatra/inject/inject-core/src/main/scala", - "finatra/inject/inject-core/src/test/scala:test-deps", - "finatra/inject/inject-server/src/test/scala:test-deps", - "finatra/inject/inject-slf4j/src/main/scala", - "finatra/jackson/src/main/scala", - "finatra/kafka-streams/kafka-streams/src/main/scala", - "finatra/kafka/src/test/scala:test-deps", - "util/util-slf4j-api/src/main/scala", - ], -) diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/admin/KafkaStreamsAdminServerFeatureTest.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/admin/KafkaStreamsAdminServerFeatureTest.scala new file mode 100644 index 0000000000..62c2734f77 --- /dev/null +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/admin/KafkaStreamsAdminServerFeatureTest.scala @@ -0,0 +1,60 @@ +package com.twitter.finatra.kafkastreams.integration.admin + +import com.twitter.finatra.kafka.serde.UnKeyedSerde +import com.twitter.finatra.kafkastreams.KafkaStreamsTwitterServer +import com.twitter.finatra.kafkastreams.test.KafkaStreamsFeatureTest +import com.twitter.inject.server.EmbeddedTwitterServer +import com.twitter.util.Await +import java.nio.charset.{Charset, StandardCharsets} +import org.apache.kafka.common.serialization.Serdes +import org.apache.kafka.streams.StreamsBuilder +import org.apache.kafka.streams.kstream.{Consumed, Produced} + +class KafkaStreamsAdminServerFeatureTest extends KafkaStreamsFeatureTest { + + override val server = new EmbeddedTwitterServer( + new KafkaStreamsTwitterServer { + override val name = "no-op" + override protected def configureKafkaStreams(builder: StreamsBuilder): Unit = { + builder.asScala + .stream("TextLinesTopic")(Consumed.`with`(UnKeyedSerde, Serdes.String)) + .to("sink")(Produced.`with`(UnKeyedSerde, Serdes.String)) + } + }, + flags = kafkaStreamsFlags ++ Map("kafka.application.id" -> "no-op") + ) + + override def beforeEach(): Unit = { + server.start() + } + + test("admin kafka streams properties") { + val bufBytes = getAdminResponseBytes("/admin/kafka/streams/properties") + val result = new String(bufBytes, StandardCharsets.UTF_8) + result.contains("application.id=no-op") should equal(true) + } + + test("admin kafka streams topology") { + val bufBytes = getAdminResponseBytes("/admin/kafka/streams/topology") + val result = new String(bufBytes, Charset.forName("UTF-8")) + result.trim() should equal( + """
+        |Topologies:
+        |   Sub-topology: 0
+        |    Source: KSTREAM-SOURCE-0000000000 (topics: [TextLinesTopic])
+        |      --> KSTREAM-SINK-0000000001
+        |    Sink: KSTREAM-SINK-0000000001 (topic: sink)
+        |      <-- KSTREAM-SOURCE-0000000000
+        |
""".stripMargin) + } + + private def getAdminResponseBytes(path: String): Array[Byte] = { + val response = server.httpGetAdmin(path) + val read = Await.result(response.reader.read()) + read.isDefined should equal(true) + val buf = read.get + val bufBytes: Array[Byte] = new Array[Byte](buf.length) + buf.write(bufBytes, off = 0) + bufBytes + } +} diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/async_transformer/WordLookupAsyncServer.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/async_transformer/WordLookupAsyncServer.scala similarity index 85% rename from kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/async_transformer/WordLookupAsyncServer.scala rename to kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/async_transformer/WordLookupAsyncServer.scala index 2907671fca..876396c629 100644 --- a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/async_transformer/WordLookupAsyncServer.scala +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/async_transformer/WordLookupAsyncServer.scala @@ -1,8 +1,8 @@ -package com.twitter.unittests.integration.async_transformer +package com.twitter.finatra.kafkastreams.integration.async_transformer import com.twitter.finatra.kafka.serde.{ScalaSerdes, UnKeyedSerde} import com.twitter.finatra.kafkastreams.KafkaStreamsTwitterServer -import com.twitter.finatra.kafkastreams.processors.FlushingAwareServer +import com.twitter.finatra.kafkastreams.flushing.FlushingAwareServer import org.apache.kafka.common.serialization.Serdes import org.apache.kafka.streams.StreamsBuilder import org.apache.kafka.streams.kstream.{Consumed, Produced} diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/async_transformer/WordLookupAsyncServerFeatureTest.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/async_transformer/WordLookupAsyncServerFeatureTest.scala similarity index 94% rename from kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/async_transformer/WordLookupAsyncServerFeatureTest.scala rename to kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/async_transformer/WordLookupAsyncServerFeatureTest.scala index da79ba819a..2f29ed7a68 100644 --- a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/async_transformer/WordLookupAsyncServerFeatureTest.scala +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/async_transformer/WordLookupAsyncServerFeatureTest.scala @@ -1,4 +1,4 @@ -package com.twitter.unittests.integration.async_transformer +package com.twitter.finatra.kafkastreams.integration.async_transformer import com.twitter.finatra.kafka.serde.ScalaSerdes import com.twitter.finatra.kafkastreams.test.KafkaStreamsFeatureTest diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/async_transformer/WordLookupAsyncServerTopologyFeatureTest.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/async_transformer/WordLookupAsyncServerTopologyFeatureTest.scala similarity index 87% rename from kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/async_transformer/WordLookupAsyncServerTopologyFeatureTest.scala rename to kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/async_transformer/WordLookupAsyncServerTopologyFeatureTest.scala index 568533f4f7..c8fdc3c9ec 100644 --- a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/async_transformer/WordLookupAsyncServerTopologyFeatureTest.scala +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/async_transformer/WordLookupAsyncServerTopologyFeatureTest.scala @@ -1,8 +1,8 @@ -package com.twitter.unittests.integration.async_transformer +package com.twitter.finatra.kafkastreams.integration.async_transformer import com.twitter.conversions.DurationOps._ import com.twitter.finatra.kafka.serde.ScalaSerdes -import com.twitter.finatra.streams.tests.{FinatraTopologyTester, TopologyFeatureTest} +import com.twitter.finatra.kafkastreams.test.{FinatraTopologyTester, TopologyFeatureTest} import org.apache.kafka.common.serialization.Serdes import org.joda.time.DateTime diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/async_transformer/WordLookupAsyncTransformer.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/async_transformer/WordLookupAsyncTransformer.scala similarity index 82% rename from kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/async_transformer/WordLookupAsyncTransformer.scala rename to kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/async_transformer/WordLookupAsyncTransformer.scala index c968d154bd..3c598fa920 100644 --- a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/async_transformer/WordLookupAsyncTransformer.scala +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/async_transformer/WordLookupAsyncTransformer.scala @@ -1,9 +1,10 @@ -package com.twitter.unittests.integration.async_transformer +package com.twitter.finatra.kafkastreams.integration.async_transformer import com.twitter.conversions.DurationOps._ import com.twitter.finagle.stats.StatsReceiver import com.twitter.finatra.kafka.serde.UnKeyed -import com.twitter.finatra.kafkastreams.processors.{AsyncTransformer, MessageTimestamp} +import com.twitter.finatra.kafkastreams.flushing.AsyncTransformer +import com.twitter.finatra.kafkastreams.utils.MessageTimestamp import com.twitter.util.{Duration, Future} class WordLookupAsyncTransformer(statsReceiver: StatsReceiver, commitInterval: Duration) diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/compositesum/UserClicks.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/compositesum/UserClicks.scala new file mode 100644 index 0000000000..e559c5b92a --- /dev/null +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/compositesum/UserClicks.scala @@ -0,0 +1,5 @@ +package com.twitter.finatra.kafkastreams.integration.compositesum + +import UserClicksTypes.UserId + +case class UserClicks(userId: UserId, clickType: Int) diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/compositesum/UserClicksSerde.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/compositesum/UserClicksSerde.scala similarity index 89% rename from kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/compositesum/UserClicksSerde.scala rename to kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/compositesum/UserClicksSerde.scala index 6a29fe7672..5eb61174ab 100644 --- a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/compositesum/UserClicksSerde.scala +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/compositesum/UserClicksSerde.scala @@ -1,4 +1,4 @@ -package com.twitter.unittests.integration.compositesum +package com.twitter.finatra.kafkastreams.integration.compositesum import com.google.common.primitives.Ints import com.twitter.finatra.kafka.serde.AbstractSerde diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/compositesum/UserClicksServer.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/compositesum/UserClicksServer.scala similarity index 79% rename from kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/compositesum/UserClicksServer.scala rename to kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/compositesum/UserClicksServer.scala index 7fcdf0be10..1befd3b533 100644 --- a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/compositesum/UserClicksServer.scala +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/compositesum/UserClicksServer.scala @@ -1,11 +1,11 @@ -package com.twitter.unittests.integration.compositesum +package com.twitter.finatra.kafkastreams.integration.compositesum import com.twitter.conversions.DurationOps._ import com.twitter.finatra.kafka.serde.ScalaSerdes import com.twitter.finatra.kafkastreams.KafkaStreamsTwitterServer import com.twitter.finatra.kafkastreams.dsl.FinatraDslWindowedAggregations -import com.twitter.finatra.streams.transformer.domain.{FixedTimeWindowedSerde, WindowedValueSerde} -import com.twitter.unittests.integration.compositesum.UserClicksTypes.{NumClicksSerde, UserIdSerde} +import com.twitter.finatra.kafkastreams.integration.compositesum.UserClicksTypes.{NumClicksSerde, UserIdSerde} +import com.twitter.finatra.kafkastreams.transformer.aggregation.{FixedTimeWindowedSerde, WindowedValueSerde} import org.apache.kafka.streams.StreamsBuilder import org.apache.kafka.streams.kstream.{Consumed, Produced} diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/compositesum/UserClicksTopologyFeatureTest.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/compositesum/UserClicksTopologyFeatureTest.scala similarity index 59% rename from kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/compositesum/UserClicksTopologyFeatureTest.scala rename to kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/compositesum/UserClicksTopologyFeatureTest.scala index a6b3589031..0e14a93714 100644 --- a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/compositesum/UserClicksTopologyFeatureTest.scala +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/compositesum/UserClicksTopologyFeatureTest.scala @@ -1,20 +1,10 @@ -package com.twitter.unittests.integration.compositesum +package com.twitter.finatra.kafkastreams.integration.compositesum import com.twitter.conversions.DurationOps._ -import com.twitter.finatra.streams.tests.{FinatraTopologyTester, TopologyFeatureTest} -import com.twitter.finatra.streams.transformer.domain.{ - FixedTimeWindowedSerde, - TimeWindowed, - WindowClosed, - WindowOpen, - WindowedValue, - WindowedValueSerde -} -import com.twitter.unittests.integration.compositesum.UserClicksTypes.{ - ClickTypeSerde, - NumClicksSerde, - UserIdSerde -} +import com.twitter.finatra.kafkastreams.integration.compositesum.UserClicksTypes._ +import com.twitter.finatra.kafkastreams.test.{FinatraTopologyTester, TopologyFeatureTest} +import com.twitter.finatra.kafkastreams.transformer.aggregation.{FixedTimeWindowedSerde, TimeWindowed, WindowClosed, WindowOpen, WindowedValue, WindowedValueSerde} +import com.twitter.finatra.kafkastreams.transformer.domain.Time import org.joda.time.DateTime class UserClicksTopologyFeatureTest extends TopologyFeatureTest { @@ -35,8 +25,8 @@ class UserClicksTopologyFeatureTest extends TopologyFeatureTest { test("windowed clicks") { val userId1 = 1 - val firstHourStartMillis = new DateTime("2018-01-01T00:00:00Z").getMillis - val fifthHourStartMillis = new DateTime("2018-01-01T05:00:00Z").getMillis + val firstHourStartTime = Time.create(new DateTime("2018-01-01T00:00:00Z")) + val fifthHourStartTime = Time.create(new DateTime("2018-01-01T05:00:00Z")) userIdToClicksTopic.pipeInput(userId1, 100) userIdToClicksTopic.pipeInput(userId1, 200) @@ -47,15 +37,15 @@ class UserClicksTopologyFeatureTest extends TopologyFeatureTest { topologyTester.advanceWallClockTime(30.seconds) hourlyWordAndCountTopic.assertOutput( - TimeWindowed.hourly(firstHourStartMillis, UserClicks(userId1, clickType = 100)), + TimeWindowed.hourly(firstHourStartTime, UserClicks(userId1, clickType = 100)), WindowedValue(WindowOpen, 1)) hourlyWordAndCountTopic.assertOutput( - TimeWindowed.hourly(firstHourStartMillis, UserClicks(userId1, clickType = 300)), + TimeWindowed.hourly(firstHourStartTime, UserClicks(userId1, clickType = 300)), WindowedValue(WindowOpen, 3)) hourlyWordAndCountTopic.assertOutput( - TimeWindowed.hourly(firstHourStartMillis, UserClicks(userId1, clickType = 200)), + TimeWindowed.hourly(firstHourStartTime, UserClicks(userId1, clickType = 200)), WindowedValue(WindowOpen, 2)) userIdToClicksTopic.pipeInput(userId1, 100) @@ -64,34 +54,34 @@ class UserClicksTopologyFeatureTest extends TopologyFeatureTest { topologyTester.advanceWallClockTime(5.hours) hourlyWordAndCountTopic.assertOutput( - TimeWindowed.hourly(firstHourStartMillis, UserClicks(userId1, clickType = 100)), + TimeWindowed.hourly(firstHourStartTime, UserClicks(userId1, clickType = 100)), WindowedValue(WindowOpen, 2)) hourlyWordAndCountTopic.assertOutput( - TimeWindowed.hourly(firstHourStartMillis, UserClicks(userId1, clickType = 300)), + TimeWindowed.hourly(firstHourStartTime, UserClicks(userId1, clickType = 300)), WindowedValue(WindowOpen, 4)) hourlyWordAndCountTopic.assertOutput( - TimeWindowed.hourly(firstHourStartMillis, UserClicks(userId1, clickType = 200)), + TimeWindowed.hourly(firstHourStartTime, UserClicks(userId1, clickType = 200)), WindowedValue(WindowOpen, 3)) userIdToClicksTopic.pipeInput(userId1, 1) topologyTester.advanceWallClockTime(30.seconds) hourlyWordAndCountTopic.assertOutput( - TimeWindowed.hourly(fifthHourStartMillis, UserClicks(userId1, clickType = 1)), + TimeWindowed.hourly(fifthHourStartTime, UserClicks(userId1, clickType = 1)), WindowedValue(WindowOpen, 1)) hourlyWordAndCountTopic.assertOutput( - TimeWindowed.hourly(firstHourStartMillis, UserClicks(userId1, clickType = 100)), + TimeWindowed.hourly(firstHourStartTime, UserClicks(userId1, clickType = 100)), WindowedValue(WindowClosed, 2)) hourlyWordAndCountTopic.assertOutput( - TimeWindowed.hourly(firstHourStartMillis, UserClicks(userId1, clickType = 200)), + TimeWindowed.hourly(firstHourStartTime, UserClicks(userId1, clickType = 200)), WindowedValue(WindowClosed, 3)) hourlyWordAndCountTopic.assertOutput( - TimeWindowed.hourly(firstHourStartMillis, UserClicks(userId1, clickType = 300)), + TimeWindowed.hourly(firstHourStartTime, UserClicks(userId1, clickType = 300)), WindowedValue(WindowClosed, 4)) } } diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/compositesum/UserClicksTypes.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/compositesum/UserClicksTypes.scala similarity index 84% rename from kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/compositesum/UserClicksTypes.scala rename to kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/compositesum/UserClicksTypes.scala index a8ba9dd98d..7162471524 100644 --- a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/compositesum/UserClicksTypes.scala +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/compositesum/UserClicksTypes.scala @@ -1,4 +1,4 @@ -package com.twitter.unittests.integration.compositesum +package com.twitter.finatra.kafkastreams.integration.compositesum import com.twitter.finatra.kafka.serde.ScalaSerdes import org.apache.kafka.common.serialization.Serde diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/config/FinatraRocksDBConfigFeatureTest.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/config/FinatraRocksDBConfigFeatureTest.scala new file mode 100644 index 0000000000..db37939980 --- /dev/null +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/config/FinatraRocksDBConfigFeatureTest.scala @@ -0,0 +1,174 @@ +package com.twitter.finatra.kafkastreams.integration.config + +import com.twitter.finatra.kafka.serde.{UnKeyed, UnKeyedSerde} +import com.twitter.finatra.kafkastreams.KafkaStreamsTwitterServer +import com.twitter.finatra.kafkastreams.config.FinatraRocksDBConfig._ +import com.twitter.finatra.kafkastreams.config.{DefaultTopicConfig, FinatraRocksDBConfig, KafkaStreamsConfig, RocksDbFlags} +import com.twitter.finatra.kafkastreams.test.{FinatraTopologyTester, TopologyFeatureTest} +import com.twitter.finatra.kafkastreams.transformer.FinatraTransformer +import com.twitter.finatra.kafkastreams.transformer.domain.Time +import org.apache.kafka.common.serialization.Serdes +import org.apache.kafka.streams.StreamsBuilder +import org.apache.kafka.streams.kstream.{Consumed, Produced} +import org.apache.kafka.streams.state.Stores +import org.joda.time.DateTime + +class FinatraRocksDBConfigFeatureTest extends TopologyFeatureTest { + private val appId = "no-op" + private val stateStoreName = "test-state-store" + + private val kafkaStreamsTwitterServer: KafkaStreamsTwitterServer = new KafkaStreamsTwitterServer with RocksDbFlags { + override val name: String = appId + override protected def configureKafkaStreams(builder: StreamsBuilder): Unit = { + builder.addStateStore( + Stores + .keyValueStoreBuilder( + Stores.persistentKeyValueStore(stateStoreName), + UnKeyedSerde, + Serdes.String + ).withLoggingEnabled(DefaultTopicConfig.FinatraChangelogConfig) + ) + + val finatraTransformerSupplier = () => + new FinatraTransformer[UnKeyed, String, UnKeyed, String](statsReceiver = statsReceiver) { + override def onInit(): Unit = { } + override def onClose(): Unit = { } + override def onMessage(messageTime: Time, unKeyed: UnKeyed, string: String): Unit = { } + } + + builder.asScala + .stream("source")(Consumed.`with`(UnKeyedSerde, Serdes.String)) + .transform(finatraTransformerSupplier, stateStoreName) + .to("sink")(Produced.`with`(UnKeyedSerde, Serdes.String)) + } + + override def streamsProperties(config: KafkaStreamsConfig): KafkaStreamsConfig = { + super + .streamsProperties(config) + .rocksDbConfigSetter[FinatraRocksDBConfig] + .withConfig(RocksDbBlockCacheSizeConfig, rocksDbCountsStoreBlockCacheSize()) + .withConfig(RocksDbBlockCacheShardBitsConfig, rocksDbBlockCacheShardBitsConfig()) + .withConfig(RocksDbLZ4Config, rocksDbEnableLZ4().toString) + .withConfig(RocksDbEnableStatistics, rocksDbEnableStatistics().toString) + .withConfig(RocksDbStatCollectionPeriodMs, rocksDbStatCollectionPeriodMs()) + .withConfig(RocksDbInfoLogLevel, rocksDbInfoLogLevel()) + .withConfig(RocksDbMaxLogFileSize, rocksDbMaxLogFileSize()) + .withConfig(RocksDbKeepLogFileNum, rocksDbKeepLogFileNum()) + .withConfig(RocksDbCacheIndexAndFilterBlocks, rocksDbCacheIndexAndFilterBlocks()) + .withConfig(RocksDbCachePinL0IndexAndFilterBlocks, rocksDbCachePinL0IndexAndFilterBlocks()) + .withConfig(RocksDbTableConfigBlockSize, rocksDbTableConfigBlockSize()) + .withConfig(RocksDbTableConfigBoomFilterKeyBits, rocksDbTableConfigBoomFilterKeyBits()) + .withConfig(RocksDbTableConfigBoomFilterMode, rocksDbTableConfigBoomFilterMode()) + .withConfig(RocksDbDatabaseWriteBufferSize, rocksDbDatabaseWriteBufferSize()) + .withConfig(RocksDbWriteBufferSize, rocksDbWriteBufferSize()) + .withConfig(RocksDbMinWriteBufferNumberToMerge, rocksDbMinWriteBufferNumberToMerge()) + .withConfig(RocksDbMaxWriteBufferNumber, rocksDbMaxWriteBufferNumber()) + .withConfig(RocksDbBytesPerSync, rocksDbBytesPerSync()) + .withConfig(RocksDbMaxBackgroundCompactions, rocksDbMaxBackgroundCompactions()) + .withConfig(RocksDbMaxBackgroundFlushes, rocksDbMaxBackgroundFlushes()) + .withConfig(RocksDbIncreaseParallelism, rocksDbIncreaseParallelism()) + .withConfig(RocksDbInplaceUpdateSupport, rocksDbInplaceUpdateSupport()) + .withConfig(RocksDbAllowConcurrentMemtableWrite, rocksDbAllowConcurrentMemtableWrite()) + .withConfig(RocksDbEnableWriteThreadAdaptiveYield, rocksDbEnableWriteThreadAdaptiveYield()) + .withConfig(RocksDbCompactionStyle, rocksDbCompactionStyle()) + .withConfig(RocksDbCompactionStyleOptimize, rocksDbCompactionStyleOptimize()) + .withConfig(RocksDbMaxBytesForLevelBase, rocksDbMaxBytesForLevelBase()) + .withConfig(RocksDbLevelCompactionDynamicLevelBytes, rocksDbLevelCompactionDynamicLevelBytes()) + .withConfig(RocksDbCompactionStyleMemtableBudget, rocksDbCompactionStyleMemtableBudget()) + } + } + + private val _topologyTester = FinatraTopologyTester( + kafkaApplicationId = appId, + server = kafkaStreamsTwitterServer, + startingWallClockTime = DateTime.now, + flags = Map( + "rocksdb.block.cache.size" -> "1.byte", + "rocksdb.block.cache.shard.bits" -> "2", + "rocksdb.lz4" -> "true", + "rocksdb.statistics" -> "true", + "rocksdb.statistics.collection.period.ms" -> "60001", + "rocksdb.log.info.level" -> "INFO_LEVEL", + "rocksdb.log.max.file.size" -> "2.bytes", + "rocksdb.log.keep.file.num" -> "3", + "rocksdb.cache.index.and.filter.blocks" -> "false", + "rocksdb.cache.pin.l0.index.and.filter.blocks" -> "false", + "rocksdb.tableconfig.block.size" -> "4.bytes", + "rocksdb.tableconfig.bloomfilter.key.bits" -> "5", + "rocksdb.tableconfig.bloomfilter.mode" -> "false", + "rocksdb.db.write.buffer.size" -> "6.bytes", + "rocksdb.write.buffer.size" -> "7.bytes", + "rocksdb.min.write.buffer.num.merge" -> "8", + "rocksdb.max.write.buffer.num" -> "9", + "rocksdb.bytes.per.sync" -> "10.bytes", + "rocksdb.max.background.compactions" -> "11", + "rocksdb.max.background.flushes" -> "12", + "rocksdb.parallelism" -> "2", + "rocksdb.inplace.update.support" -> "false", + "rocksdb.allow.concurrent.memtable.write" -> "true", + "rocksdb.enable.write.thread.adaptive.yield" -> "true", + "rocksdb.compaction.style" -> "UNIVERSAL", + "rocksdb.compaction.style.optimize" -> "false", + "rocksdb.max.bytes.for.level.base" -> "13.bytes", + "rocksdb.level.compaction.dynamic.level.bytes" -> "false", + "rocksdb.compaction.style.memtable.budget" -> "14.bytes" + ) + ) + + override protected def topologyTester: FinatraTopologyTester = { + _topologyTester + } + + override def beforeEach(): Unit = { + super.beforeEach() + topologyTester.reset() + topologyTester.topic( + "source", + UnKeyedSerde, + Serdes.String + ) + topologyTester.topic( + "sink", + UnKeyedSerde, + Serdes.String + ) + } + + test("rocksdb properties") { + val properties = topologyTester.properties + properties.getProperty("rocksdb.config.setter") should be( + "com.twitter.finatra.kafkastreams.config.FinatraRocksDBConfig") + properties.getProperty("rocksdb.block.cache.size") should be("1") + properties.getProperty("rocksdb.block.cache.shard.bits") should be("2") + properties.getProperty("rocksdb.lz4") should be("true") + properties.getProperty("rocksdb.statistics") should be("true") + properties.getProperty("rocksdb.statistics.collection.period.ms") should be("60001") + properties.getProperty("rocksdb.log.info.level") should be("INFO_LEVEL") + properties.getProperty("rocksdb.log.max.file.size") should be("2") + properties.getProperty("rocksdb.log.keep.file.num") should be("3") + properties.getProperty("rocksdb.cache.index.and.filter.blocks") should be("false") + properties.getProperty("rocksdb.cache.pin.l0.index.and.filter.blocks") should be("false") + properties.getProperty("rocksdb.tableconfig.block.size") should be("4") + properties.getProperty("rocksdb.tableconfig.bloomfilter.key.bits") should be("5") + properties.getProperty("rocksdb.tableconfig.bloomfilter.mode") should be("false") + properties.getProperty("rocksdb.db.write.buffer.size") should be("6") + properties.getProperty("rocksdb.write.buffer.size") should be("7") + properties.getProperty("rocksdb.min.write.buffer.num.merge") should be("8") + properties.getProperty("rocksdb.max.write.buffer.num") should be("9") + properties.getProperty("rocksdb.bytes.per.sync") should be("10") + properties.getProperty("rocksdb.max.background.compactions") should be("11") + properties.getProperty("rocksdb.max.background.flushes") should be("12") + properties.getProperty("rocksdb.parallelism") should be("2") + properties.getProperty("rocksdb.inplace.update.support") should be("false") + properties.getProperty("rocksdb.allow.concurrent.memtable.write") should be("true") + properties.getProperty("rocksdb.enable.write.thread.adaptive.yield") should be("true") + properties.getProperty("rocksdb.compaction.style") should be("UNIVERSAL") + properties.getProperty("rocksdb.compaction.style.optimize") should be("false") + properties.getProperty("rocksdb.max.bytes.for.level.base") should be("13") + properties.getProperty("rocksdb.level.compaction.dynamic.level.bytes") should be("false") + properties.getProperty("rocksdb.compaction.style.memtable.budget") should be("14") + + topologyTester.driver + .getKeyValueStore[UnKeyed, String](stateStoreName) shouldNot be(null) + } +} diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/default_serde/DefaultSerdeWordCountDbServer.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/default_serde/DefaultSerdeWordCountDbServer.scala similarity index 93% rename from kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/default_serde/DefaultSerdeWordCountDbServer.scala rename to kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/default_serde/DefaultSerdeWordCountDbServer.scala index 7de876fecd..6d560ed64b 100644 --- a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/default_serde/DefaultSerdeWordCountDbServer.scala +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/default_serde/DefaultSerdeWordCountDbServer.scala @@ -1,4 +1,4 @@ -package com.twitter.unittests.integration.default_serde +package com.twitter.finatra.kafkastreams.integration.default_serde import com.twitter.finatra.kafka.serde.ScalaSerdes import com.twitter.finatra.kafkastreams.KafkaStreamsTwitterServer diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/default_serde/DefaultSerdeWordCountServerFeatureTest.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/default_serde/DefaultSerdeWordCountServerFeatureTest.scala similarity index 94% rename from kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/default_serde/DefaultSerdeWordCountServerFeatureTest.scala rename to kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/default_serde/DefaultSerdeWordCountServerFeatureTest.scala index e324378432..9680b24870 100644 --- a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/default_serde/DefaultSerdeWordCountServerFeatureTest.scala +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/default_serde/DefaultSerdeWordCountServerFeatureTest.scala @@ -1,4 +1,4 @@ -package com.twitter.unittests.integration.default_serde +package com.twitter.finatra.kafkastreams.integration.default_serde import com.twitter.finatra.kafka.serde.ScalaSerdes import com.twitter.finatra.kafkastreams.KafkaStreamsTwitterServer diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/finatratransformer/WordLengthFinatraTransformerV2.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/finatratransformer/WordLengthFinatraTransformer.scala similarity index 52% rename from kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/finatratransformer/WordLengthFinatraTransformerV2.scala rename to kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/finatratransformer/WordLengthFinatraTransformer.scala index 6fa26b74aa..d7eb6fc4dd 100644 --- a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/finatratransformer/WordLengthFinatraTransformerV2.scala +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/finatratransformer/WordLengthFinatraTransformer.scala @@ -1,19 +1,20 @@ -package com.twitter.unittests.integration.finatratransformer +package com.twitter.finatra.kafkastreams.integration.finatratransformer import com.twitter.conversions.DurationOps._ import com.twitter.finagle.stats.StatsReceiver -import com.twitter.finatra.streams.transformer.domain.{Expire, Time, TimerMetadata} -import com.twitter.finatra.streams.transformer.{FinatraTransformerV2, PersistentTimers} -import com.twitter.unittests.integration.finatratransformer.WordLengthFinatraTransformerV2._ +import com.twitter.finatra.kafkastreams.integration.finatratransformer.WordLengthFinatraTransformer._ +import com.twitter.finatra.kafkastreams.transformer.FinatraTransformer +import com.twitter.finatra.kafkastreams.transformer.domain.{Expire, Time, TimerMetadata} +import com.twitter.finatra.kafkastreams.transformer.stores.PersistentTimers import com.twitter.util.Duration import org.apache.kafka.streams.processor.PunctuationType -object WordLengthFinatraTransformerV2 { +object WordLengthFinatraTransformer { val delayedMessageTime: Duration = 5.seconds } -class WordLengthFinatraTransformerV2(statsReceiver: StatsReceiver, timerStoreName: String) - extends FinatraTransformerV2[String, String, String, String](statsReceiver) +class WordLengthFinatraTransformer(statsReceiver: StatsReceiver, timerStoreName: String) + extends FinatraTransformer[String, String, String, String](statsReceiver) with PersistentTimers { private val timerStore = @@ -22,7 +23,7 @@ class WordLengthFinatraTransformerV2(statsReceiver: StatsReceiver, timerStoreNam override def onMessage(messageTime: Time, key: String, value: String): Unit = { forward(key, "onMessage " + key + " " + key.length) - val time = messageTime.plus(delayedMessageTime) + val time = messageTime + delayedMessageTime timerStore.addTimer(time, Expire, key) } diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/finatratransformer/WordLengthServer.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/finatratransformer/WordLengthServer.scala similarity index 75% rename from kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/finatratransformer/WordLengthServer.scala rename to kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/finatratransformer/WordLengthServer.scala index 98c8c6d26d..099b6c781c 100644 --- a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/finatratransformer/WordLengthServer.scala +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/finatratransformer/WordLengthServer.scala @@ -1,8 +1,8 @@ -package com.twitter.unittests.integration.finatratransformer +package com.twitter.finatra.kafkastreams.integration.finatratransformer import com.twitter.finatra.kafkastreams.KafkaStreamsTwitterServer -import com.twitter.finatra.streams.transformer.FinatraTransformer -import com.twitter.unittests.integration.finatratransformer.WordLengthServer._ +import com.twitter.finatra.kafkastreams.integration.finatratransformer.WordLengthServer._ +import com.twitter.finatra.kafkastreams.transformer.FinatraTransformer import org.apache.kafka.common.serialization.Serdes import org.apache.kafka.streams.StreamsBuilder import org.apache.kafka.streams.kstream.{Consumed, Produced} @@ -21,7 +21,7 @@ class WordLengthServer extends KafkaStreamsTwitterServer { FinatraTransformer.timerStore(timerStoreName, Serdes.String())) val transformerSupplier = () => - new WordLengthFinatraTransformerV2(statsReceiver, timerStoreName) + new WordLengthFinatraTransformer(statsReceiver, timerStoreName) streamsBuilder.asScala .stream(stringsAndInputsTopic)( diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/finatratransformer/WordLengthServerTopologyFeatureTest.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/finatratransformer/WordLengthServerTopologyFeatureTest.scala similarity index 89% rename from kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/finatratransformer/WordLengthServerTopologyFeatureTest.scala rename to kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/finatratransformer/WordLengthServerTopologyFeatureTest.scala index 7fc06b441c..2c4306b56a 100644 --- a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/finatratransformer/WordLengthServerTopologyFeatureTest.scala +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/finatratransformer/WordLengthServerTopologyFeatureTest.scala @@ -1,7 +1,7 @@ -package com.twitter.unittests.integration.finatratransformer +package com.twitter.finatra.kafkastreams.integration.finatratransformer import com.twitter.conversions.DurationOps._ -import com.twitter.finatra.streams.tests.{FinatraTopologyTester, TopologyFeatureTest} +import com.twitter.finatra.kafkastreams.test.{FinatraTopologyTester, TopologyFeatureTest} import org.apache.kafka.common.serialization.Serdes import org.joda.time.DateTime diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/globaltable/GlobalTableServer.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/globaltable/GlobalTableServer.scala similarity index 92% rename from kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/globaltable/GlobalTableServer.scala rename to kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/globaltable/GlobalTableServer.scala index d50819b5f3..f6c67464ca 100644 --- a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/globaltable/GlobalTableServer.scala +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/globaltable/GlobalTableServer.scala @@ -1,4 +1,4 @@ -package com.twitter.unittests.integration.globaltable +package com.twitter.finatra.kafkastreams.integration.globaltable import com.twitter.finatra.kafka.serde.ScalaSerdes import com.twitter.finatra.kafkastreams.KafkaStreamsTwitterServer diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/globaltable/GlobalTableServerFeatureTest.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/globaltable/GlobalTableServerFeatureTest.scala similarity index 96% rename from kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/globaltable/GlobalTableServerFeatureTest.scala rename to kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/globaltable/GlobalTableServerFeatureTest.scala index 3fa8438156..3dd8e0d460 100644 --- a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/globaltable/GlobalTableServerFeatureTest.scala +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/globaltable/GlobalTableServerFeatureTest.scala @@ -1,4 +1,4 @@ -package com.twitter.unittests.integration.globaltable +package com.twitter.finatra.kafkastreams.integration.globaltable import com.twitter.finatra.kafka.serde.ScalaSerdes import com.twitter.finatra.kafkastreams.config.KafkaStreamsConfig diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/punctuator/HeartBeatPunctuator.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/punctuator/HeartBeatPunctuator.scala new file mode 100644 index 0000000000..a31310ad68 --- /dev/null +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/punctuator/HeartBeatPunctuator.scala @@ -0,0 +1,13 @@ +package com.twitter.finatra.kafkastreams.integration.punctuator + +import com.twitter.finagle.stats.StatsReceiver +import org.apache.kafka.streams.processor.{ProcessorContext, Punctuator} + +class HeartBeatPunctuator(processorContext: ProcessorContext, statsReceiver: StatsReceiver) extends Punctuator { + def punctuate(timestampMillis: Long): Unit = { + punctuateCounter.incr() + processorContext.forward(timestampMillis, timestampMillis) + } + + private val punctuateCounter = statsReceiver.counter("punctuate") +} diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/punctuator/HeartBeatServer.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/punctuator/HeartBeatServer.scala new file mode 100644 index 0000000000..21d69050aa --- /dev/null +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/punctuator/HeartBeatServer.scala @@ -0,0 +1,44 @@ +package com.twitter.finatra.kafkastreams.integration.punctuator + +import com.twitter.conversions.DurationOps._ +import com.twitter.finatra.kafka.serde.ScalaSerdes +import com.twitter.finatra.kafkastreams.KafkaStreamsTwitterServer +import org.apache.kafka.streams.StreamsBuilder +import org.apache.kafka.streams.kstream.{Consumed, Produced, Transformer} +import org.apache.kafka.streams.processor.{Cancellable, ProcessorContext, PunctuationType} + +class HeartBeatServer extends KafkaStreamsTwitterServer { + override val name = "heartbeat" + + private val transformerSupplier: () => Transformer[Long, Long, (Long, Long)] = () => + new Transformer[Long, Long, (Long, Long)] { + private val transformCounter = streamsStatsReceiver.counter("transform") + + private var heartBeatPunctuatorCancellable: Cancellable = _ + + override def close(): Unit = { + if (heartBeatPunctuatorCancellable != null) { + heartBeatPunctuatorCancellable.cancel() + } + } + + override def init(processorContext: ProcessorContext): Unit = { + heartBeatPunctuatorCancellable = processorContext.schedule( + 1.second.inMillis, + PunctuationType.WALL_CLOCK_TIME, + new HeartBeatPunctuator(processorContext, streamsStatsReceiver)) + } + + override def transform(k: Long, v: Long): (Long, Long) = { + transformCounter.incr() + (k, v) + } + } + + override protected def configureKafkaStreams(builder: StreamsBuilder): Unit = { + builder.asScala + .stream[Long, Long]("input-topic")(Consumed.`with`(ScalaSerdes.Long, ScalaSerdes.Long)) + .transform[Long, Long](transformerSupplier) + .to("output-topic")(Produced.`with`(ScalaSerdes.Long, ScalaSerdes.Long)) + } +} diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/punctuator/HeartBeatServerTopologyFeatureTest.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/punctuator/HeartBeatServerTopologyFeatureTest.scala new file mode 100644 index 0000000000..d9da9aed72 --- /dev/null +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/punctuator/HeartBeatServerTopologyFeatureTest.scala @@ -0,0 +1,131 @@ +package com.twitter.finatra.kafkastreams.integration.punctuator + +import com.twitter.conversions.DurationOps._ +import com.twitter.finatra.kafka.serde.ScalaSerdes +import com.twitter.finatra.kafkastreams.test.{FinatraTopologyTester, TopologyFeatureTest} +import org.joda.time.DateTime + +class HeartBeatServerTopologyFeatureTest extends TopologyFeatureTest { + private val startingWallClockTime = new DateTime("1970-01-01T00:00:00Z") + + override val topologyTester = FinatraTopologyTester( + kafkaApplicationId = "wordcount-prod-bob", + server = new HeartBeatServer, + startingWallClockTime = startingWallClockTime + ) + + private val inputTopic = + topologyTester.topic("input-topic", ScalaSerdes.Long, ScalaSerdes.Long) + + private val outputTopic = + topologyTester.topic("output-topic", ScalaSerdes.Long, ScalaSerdes.Long) + + private val transformStatName = "kafka/stream/transform" + private val punctuateStatName = "kafka/stream/punctuate" + + test("Publish single value from input to output") { + topologyTester.stats.assertCounter(transformStatName, 0) + topologyTester.stats.assertCounter(punctuateStatName, 0) + inputTopic.pipeInput(1,1) + outputTopic.assertOutput(1,1) + outputTopic.readAllOutput().isEmpty should be(true) + topologyTester.stats.assertCounter(transformStatName, 1) + topologyTester.stats.assertCounter(punctuateStatName, 0) + } + + test("Publish single value and single heartbeat from input to output") { + topologyTester.stats.assertCounter(transformStatName, 0) + topologyTester.stats.assertCounter(punctuateStatName, 0) + inputTopic.pipeInput(1,1) + outputTopic.assertOutput(1,1) + topologyTester.advanceWallClockTime(1.seconds) + val punctuatedTime = startingWallClockTime.getMillis + 1.second.inMillis + outputTopic.assertOutput(punctuatedTime, punctuatedTime) + outputTopic.readAllOutput().isEmpty should be(true) + topologyTester.stats.assertCounter(transformStatName, 1) + topologyTester.stats.assertCounter(punctuateStatName, 1) + } + + test("Publish heartbeat from advanced wall clock time to output") { + topologyTester.stats.assertCounter(transformStatName, 0) + topologyTester.stats.assertCounter(punctuateStatName, 0) + topologyTester.advanceWallClockTime(1.seconds) + val punctuatedTime = startingWallClockTime.getMillis + 1.second.inMillis + outputTopic.assertOutput(punctuatedTime, punctuatedTime) + outputTopic.readAllOutput().isEmpty should be(true) + topologyTester.stats.assertCounter(transformStatName, 0) + topologyTester.stats.assertCounter(punctuateStatName, 1) + } + + test("Publish multiple values from input to output") { + topologyTester.stats.assertCounter(transformStatName, 0) + topologyTester.stats.assertCounter(punctuateStatName, 0) + + val messages = 1 to 10 + for { + message <- messages + } { + inputTopic.pipeInput(message, message) + } + + for { + message <- messages + } { + outputTopic.assertOutput(message, message) + } + + outputTopic.readAllOutput().isEmpty should be(true) + topologyTester.stats.assertCounter(transformStatName, 10) + topologyTester.stats.assertCounter(punctuateStatName, 0) + } + + test("Publish multiple heartbeat from advanced wall clock time to output") { + topologyTester.stats.assertCounter(transformStatName, 0) + topologyTester.stats.assertCounter(punctuateStatName, 0) + + val secondsRange = 1 to 10 + for { + _ <- secondsRange + } { + topologyTester.advanceWallClockTime(1.seconds) + } + + for { + s <- secondsRange + } { + val punctuatedTime = startingWallClockTime.getMillis + s.seconds.inMillis + outputTopic.assertOutput(punctuatedTime, punctuatedTime) + } + + outputTopic.readAllOutput().isEmpty should be(true) + topologyTester.stats.assertCounter(transformStatName, 0) + topologyTester.stats.assertCounter(punctuateStatName, 10) + } + + test("Publish multiple values and multiple heartbeats from input to output") { + topologyTester.stats.assertCounter(transformStatName, 0) + topologyTester.stats.assertCounter(punctuateStatName, 0) + inputTopic.pipeInput(1,1) + topologyTester.advanceWallClockTime(1.seconds) + inputTopic.pipeInput(2,2) + topologyTester.advanceWallClockTime(1.seconds) + inputTopic.pipeInput(3,3) + topologyTester.advanceWallClockTime(1.seconds) + + outputTopic.assertOutput(1,1) + val punctuatedTime1 = startingWallClockTime.getMillis + 1.second.inMillis + outputTopic.assertOutput(punctuatedTime1, punctuatedTime1) + + outputTopic.assertOutput(2,2) + val punctuatedTime2 = punctuatedTime1 + 1.second.inMillis + outputTopic.assertOutput(punctuatedTime2, punctuatedTime2) + + outputTopic.assertOutput(3,3) + val punctuatedTime3 = punctuatedTime2 + 1.second.inMillis + outputTopic.assertOutput(punctuatedTime3, punctuatedTime3) + + outputTopic.readAllOutput().isEmpty should be(true) + topologyTester.stats.assertCounter(transformStatName, 3) + topologyTester.stats.assertCounter(punctuateStatName, 3) + } +} diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/sampling/SamplingServer.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/sampling/SamplingServer.scala similarity index 85% rename from kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/sampling/SamplingServer.scala rename to kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/sampling/SamplingServer.scala index c5c32a43dd..afae4821a8 100644 --- a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/sampling/SamplingServer.scala +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/sampling/SamplingServer.scala @@ -1,22 +1,14 @@ -package com.twitter.unittests.integration.sampling +package com.twitter.finatra.kafkastreams.integration.sampling import com.twitter.conversions.DurationOps._ import com.twitter.conversions.StorageUnitOps._ import com.twitter.finatra.kafka.serde.ScalaSerdes import com.twitter.finatra.kafkastreams.KafkaStreamsTwitterServer -import com.twitter.finatra.kafkastreams.config.FinatraRocksDBConfig.{ - RocksDbBlockCacheSizeConfig, - RocksDbEnableStatistics, - RocksDbLZ4Config -} -import com.twitter.finatra.kafkastreams.config.{FinatraRocksDBConfig, KafkaStreamsConfig} +import com.twitter.finatra.kafkastreams.config.FinatraRocksDBConfig.{RocksDbBlockCacheSizeConfig, RocksDbEnableStatistics, RocksDbLZ4Config} +import com.twitter.finatra.kafkastreams.config.FinatraTransformerFlags.{AutoWatermarkInterval, EmitWatermarkPerMessage} +import com.twitter.finatra.kafkastreams.config.{FinatraRocksDBConfig, KafkaStreamsConfig, RocksDbFlags} import com.twitter.finatra.kafkastreams.dsl.FinatraDslSampling -import com.twitter.finatra.streams.flags.FinatraTransformerFlags.{ - AutoWatermarkInterval, - EmitWatermarkPerMessage -} -import com.twitter.finatra.streams.flags.RocksDbFlags -import com.twitter.unittests.integration.sampling.SamplingServer._ +import com.twitter.finatra.kafkastreams.integration.sampling.SamplingServer._ import org.apache.kafka.common.record.CompressionType import org.apache.kafka.streams.StreamsBuilder import org.apache.kafka.streams.kstream._ diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/sampling/SamplingServerTopologyFeatureTest.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/sampling/SamplingServerTopologyFeatureTest.scala similarity index 76% rename from kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/sampling/SamplingServerTopologyFeatureTest.scala rename to kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/sampling/SamplingServerTopologyFeatureTest.scala index d95341056f..fd849d2427 100644 --- a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/sampling/SamplingServerTopologyFeatureTest.scala +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/sampling/SamplingServerTopologyFeatureTest.scala @@ -1,14 +1,14 @@ -package com.twitter.unittests.integration.sampling +package com.twitter.finatra.kafkastreams.integration.sampling import com.twitter.conversions.DurationOps._ import com.twitter.finatra.kafka.serde.ScalaSerdes -import com.twitter.finatra.streams.tests.{FinatraTopologyTester, TopologyFeatureTest} -import com.twitter.finatra.streams.transformer.domain.IndexedSampleKey -import com.twitter.finatra.streams.transformer.{IteratorImplicits, SamplingUtils} +import com.twitter.finatra.kafkastreams.internal.utils.sampling.IndexedSampleKey +import com.twitter.finatra.kafkastreams.test.{FinatraTopologyTester, TopologyFeatureTest} +import com.twitter.finatra.kafkastreams.transformer.utils.{IteratorImplicits, SamplingUtils} import org.apache.kafka.streams.state.KeyValueStore import org.joda.time.DateTime -class SamplingServerTopologyFeatureTest extends TopologyFeatureTest with IteratorImplicits{ +class SamplingServerTopologyFeatureTest extends TopologyFeatureTest with IteratorImplicits { override val topologyTester = FinatraTopologyTester( kafkaApplicationId = "sampling-server-prod-alice", @@ -17,18 +17,22 @@ class SamplingServerTopologyFeatureTest extends TopologyFeatureTest with Iterato ) private val tweetIdToImpressingUserId = - topologyTester.topic(SamplingServer.tweetToImpressingUserTopic, ScalaSerdes.Long, ScalaSerdes.Long) + topologyTester.topic( + SamplingServer.tweetToImpressingUserTopic, + ScalaSerdes.Long, + ScalaSerdes.Long) private var countStore: KeyValueStore[Long, Long] = _ private var sampleStore: KeyValueStore[IndexedSampleKey[Long], Long] = _ - override def beforeEach(): Unit = { super.beforeEach() - countStore = topologyTester.driver.getKeyValueStore[Long, Long](SamplingUtils.getNumCountsStoreName(SamplingServer.sampleName)) - sampleStore = topologyTester.driver.getKeyValueStore[IndexedSampleKey[Long], Long](SamplingUtils.getSampleStoreName(SamplingServer.sampleName)) + countStore = topologyTester.driver.getKeyValueStore[Long, Long]( + SamplingUtils.getNumCountsStoreName(SamplingServer.sampleName)) + sampleStore = topologyTester.driver.getKeyValueStore[IndexedSampleKey[Long], Long]( + SamplingUtils.getSampleStoreName(SamplingServer.sampleName)) } test("test that a sample does what you want") { @@ -74,12 +78,14 @@ class SamplingServerTopologyFeatureTest extends TopologyFeatureTest with Iterato } private def assertSampleSize(tweetId: Int, expectedSize: Int): Unit = { - val range = sampleStore.range(IndexedSampleKey(tweetId, 0), IndexedSampleKey(tweetId, Int.MaxValue)) + val range = + sampleStore.range(IndexedSampleKey(tweetId, 0), IndexedSampleKey(tweetId, Int.MaxValue)) range.values.toSet.size should be(expectedSize) } private def assertSampleEquals(tweetId: Int, expectedSample: Set[Int]): Unit = { - val range = sampleStore.range(IndexedSampleKey(tweetId, 0), IndexedSampleKey(tweetId, Int.MaxValue)) + val range = + sampleStore.range(IndexedSampleKey(tweetId, 0), IndexedSampleKey(tweetId, Int.MaxValue)) range.values.toSet should be(expectedSample) } diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/stateless/VerifyFailureServer.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/stateless/VerifyFailureServer.scala similarity index 84% rename from kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/stateless/VerifyFailureServer.scala rename to kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/stateless/VerifyFailureServer.scala index 06148bf77d..ebcebf1f4c 100644 --- a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/stateless/VerifyFailureServer.scala +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/stateless/VerifyFailureServer.scala @@ -1,7 +1,7 @@ -package com.twitter.unittests.integration.stateless +package com.twitter.finatra.kafkastreams.integration.stateless import com.twitter.finatra.kafka.serde.ScalaSerdes -import com.twitter.finatra.kafkastreams.StatelessKafkaStreamsTwitterServer +import com.twitter.finatra.kafkastreams.utils.StatelessKafkaStreamsTwitterServer import org.apache.kafka.common.serialization.Serdes import org.apache.kafka.streams.StreamsBuilder import org.apache.kafka.streams.kstream.{Consumed, Materialized, Produced, Serialized} diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/stateless/VerifyFailureServerFeatureTest.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/stateless/VerifyFailureServerFeatureTest.scala similarity index 92% rename from kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/stateless/VerifyFailureServerFeatureTest.scala rename to kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/stateless/VerifyFailureServerFeatureTest.scala index 4dd8b95599..0029723a07 100644 --- a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/stateless/VerifyFailureServerFeatureTest.scala +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/stateless/VerifyFailureServerFeatureTest.scala @@ -1,4 +1,4 @@ -package com.twitter.unittests.integration.stateless +package com.twitter.finatra.kafkastreams.integration.stateless import com.twitter.finatra.kafka.serde.ScalaSerdes import com.twitter.finatra.kafkastreams.test.KafkaStreamsMultiServerFeatureTest diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/window/WindowedTweetWordCountServer.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/window/WindowedTweetWordCountServer.scala similarity index 87% rename from kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/window/WindowedTweetWordCountServer.scala rename to kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/window/WindowedTweetWordCountServer.scala index decbfb29ed..eefd2de35a 100644 --- a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/window/WindowedTweetWordCountServer.scala +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/window/WindowedTweetWordCountServer.scala @@ -1,10 +1,10 @@ -package com.twitter.unittests.integration.window +package com.twitter.finatra.kafkastreams.integration.window import com.twitter.conversions.DurationOps._ import com.twitter.finatra.kafka.serde.ScalaSerdes import com.twitter.finatra.kafkastreams.KafkaStreamsTwitterServer import com.twitter.finatra.kafkastreams.dsl.FinatraDslWindowedAggregations -import com.twitter.finatra.streams.transformer.domain._ +import com.twitter.finatra.kafkastreams.transformer.aggregation.{FixedTimeWindowedSerde, WindowedValueSerde} import org.apache.kafka.common.serialization.Serdes import org.apache.kafka.streams.StreamsBuilder import org.apache.kafka.streams.kstream.{Consumed, Produced} diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/window/WindowedTweetWordCountServerTopologyFeatureTest.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/window/WindowedTweetWordCountServerTopologyFeatureTest.scala similarity index 84% rename from kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/window/WindowedTweetWordCountServerTopologyFeatureTest.scala rename to kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/window/WindowedTweetWordCountServerTopologyFeatureTest.scala index 06650991db..6ff926b67c 100644 --- a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/window/WindowedTweetWordCountServerTopologyFeatureTest.scala +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/window/WindowedTweetWordCountServerTopologyFeatureTest.scala @@ -1,10 +1,10 @@ -package com.twitter.unittests.integration.window +package com.twitter.finatra.kafkastreams.integration.window import com.twitter.conversions.DurationOps._ import com.twitter.finatra.kafka.serde.ScalaSerdes -import com.twitter.finatra.streams.query.QueryableFinatraWindowStore -import com.twitter.finatra.streams.tests.{FinatraTopologyTester, TopologyFeatureTest} -import com.twitter.finatra.streams.transformer.domain.WindowedValueSerde +import com.twitter.finatra.kafkastreams.query.QueryableFinatraWindowStore +import com.twitter.finatra.kafkastreams.test.{FinatraTopologyTester, TopologyFeatureTest} +import com.twitter.finatra.kafkastreams.transformer.aggregation.WindowedValueSerde import org.apache.kafka.common.serialization.Serdes import org.joda.time.DateTime diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/wordcount/WordCountRocksDbServer.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/wordcount/WordCountRocksDbServer.scala similarity index 93% rename from kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/wordcount/WordCountRocksDbServer.scala rename to kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/wordcount/WordCountRocksDbServer.scala index 79728e54e7..95d5bb61e7 100644 --- a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/wordcount/WordCountRocksDbServer.scala +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/wordcount/WordCountRocksDbServer.scala @@ -1,4 +1,4 @@ -package com.twitter.unittests.integration.wordcount +package com.twitter.finatra.kafkastreams.integration.wordcount import com.twitter.finatra.kafka.serde.ScalaSerdes import com.twitter.finatra.kafkastreams.KafkaStreamsTwitterServer diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/wordcount/WordCountServerFeatureTest.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/wordcount/WordCountServerFeatureTest.scala similarity index 95% rename from kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/wordcount/WordCountServerFeatureTest.scala rename to kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/wordcount/WordCountServerFeatureTest.scala index e4bca39e69..5b93eca8bd 100644 --- a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/wordcount/WordCountServerFeatureTest.scala +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/wordcount/WordCountServerFeatureTest.scala @@ -1,4 +1,4 @@ -package com.twitter.unittests.integration.wordcount +package com.twitter.finatra.kafkastreams.integration.wordcount import com.twitter.finatra.kafka.serde.ScalaSerdes import com.twitter.finatra.kafka.test.utils.InMemoryStatsUtil @@ -96,6 +96,11 @@ class WordCountServerFeatureTest extends KafkaStreamsMultiServerFeatureTest { val serverAfterRestart = createServer() serverAfterRestart.start() + val serverAfterRestartStats = InMemoryStatsUtil(serverAfterRestart.injector) + serverAfterRestartStats.waitForGaugeUntil( + "kafka/stream/finatra_state_restore_listener/restore_time_elapsed_ms", + _ >= 0 + ) textLinesTopic.publish(1L -> "world world") wordsWithCountsTopic.consumeAsManyMessagesUntilMap(Map("world" -> 5L)) diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/wordcount/WordCountServerTopologyFeatureTest.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/wordcount/WordCountServerTopologyFeatureTest.scala similarity index 90% rename from kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/wordcount/WordCountServerTopologyFeatureTest.scala rename to kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/wordcount/WordCountServerTopologyFeatureTest.scala index c189d8b8bd..edc5f0b6bc 100644 --- a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/wordcount/WordCountServerTopologyFeatureTest.scala +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/wordcount/WordCountServerTopologyFeatureTest.scala @@ -1,7 +1,7 @@ -package com.twitter.unittests.integration.wordcount +package com.twitter.finatra.kafkastreams.integration.wordcount import com.twitter.finatra.kafka.serde.ScalaSerdes -import com.twitter.finatra.streams.tests.{FinatraTopologyTester, TopologyFeatureTest} +import com.twitter.finatra.kafkastreams.test.{FinatraTopologyTester, TopologyFeatureTest} import org.apache.kafka.common.serialization.Serdes import org.joda.time.DateTime diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/wordcount_in_memory/WordCountInMemoryServer.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/wordcount_in_memory/WordCountInMemoryServer.scala similarity index 94% rename from kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/wordcount_in_memory/WordCountInMemoryServer.scala rename to kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/wordcount_in_memory/WordCountInMemoryServer.scala index 0a4dee1dab..d0c4b6c255 100644 --- a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/wordcount_in_memory/WordCountInMemoryServer.scala +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/wordcount_in_memory/WordCountInMemoryServer.scala @@ -1,4 +1,4 @@ -package com.twitter.unittests.integration.wordcount_in_memory +package com.twitter.finatra.kafkastreams.integration.wordcount_in_memory import com.twitter.finatra.kafka.serde.ScalaSerdes import com.twitter.finatra.kafkastreams.KafkaStreamsTwitterServer diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/wordcount_in_memory/WordCountInMemoryServerFeatureTest.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/wordcount_in_memory/WordCountInMemoryServerFeatureTest.scala similarity index 96% rename from kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/wordcount_in_memory/WordCountInMemoryServerFeatureTest.scala rename to kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/wordcount_in_memory/WordCountInMemoryServerFeatureTest.scala index e9a1baac0a..ae3d5ec345 100644 --- a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/wordcount_in_memory/WordCountInMemoryServerFeatureTest.scala +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/integration/wordcount_in_memory/WordCountInMemoryServerFeatureTest.scala @@ -1,4 +1,4 @@ -package com.twitter.unittests.integration.wordcount_in_memory +package com.twitter.finatra.kafkastreams.integration.wordcount_in_memory import com.twitter.finatra.kafka.serde.ScalaSerdes import com.twitter.finatra.kafkastreams.test.KafkaStreamsFeatureTest diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/streams/tests/FinatraTopologyTester.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/test/FinatraTopologyTester.scala similarity index 93% rename from kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/streams/tests/FinatraTopologyTester.scala rename to kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/test/FinatraTopologyTester.scala index c19fd2f967..5a10266ce5 100644 --- a/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/streams/tests/FinatraTopologyTester.scala +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/test/FinatraTopologyTester.scala @@ -1,4 +1,4 @@ -package com.twitter.finatra.streams.tests +package com.twitter.finatra.kafkastreams.test import com.github.nscala_time.time.DurationBuilder import com.google.inject.Module @@ -7,15 +7,11 @@ import com.twitter.finagle.stats.{InMemoryStatsReceiver, StatsReceiver} import com.twitter.finatra.kafka.modules.KafkaBootstrapModule import com.twitter.finatra.kafka.test.utils.InMemoryStatsUtil import com.twitter.finatra.kafkastreams.KafkaStreamsTwitterServer -import com.twitter.finatra.kafkastreams.test.TestDirectoryUtils -import com.twitter.finatra.streams.converters.time._ -import com.twitter.finatra.streams.flags.FinatraTransformerFlags -import com.twitter.finatra.streams.query.{ - QueryableFinatraKeyValueStore, - QueryableFinatraWindowStore -} -import com.twitter.finatra.streams.transformer.domain.TimeWindowed -import com.twitter.finatra.streams.transformer.internal.domain.Timer +import com.twitter.finatra.kafkastreams.config.FinatraTransformerFlags +import com.twitter.finatra.kafkastreams.query.{QueryableFinatraKeyValueStore, QueryableFinatraWindowStore} +import com.twitter.finatra.kafkastreams.transformer.aggregation.TimeWindowed +import com.twitter.finatra.kafkastreams.transformer.stores.internal.Timer +import com.twitter.finatra.kafkastreams.utils.time._ import com.twitter.inject.{AppAccessor, Injector, Logging, TwitterModule} import com.twitter.util.Duration import java.util.Properties @@ -212,9 +208,9 @@ case class FinatraTopologyTester private ( } def reset(): Unit = { + inMemoryStatsReceiver.clear() close() createTopologyTester() - DateTimeUtils.setCurrentMillisFixed(startingWallClockTime.getMillis) } def close(): Unit = { @@ -306,6 +302,8 @@ case class FinatraTopologyTester private ( } private def createTopologyTester(): Unit = { - _driver = new TopologyTestDriver(topology, properties) + DateTimeUtils.setCurrentMillisFixed(startingWallClockTime.getMillis) + debug(s"Creating TopologyTestDriver with wall clock ${DateTimeUtils.currentTimeMillis().iso8601Millis}") + _driver = new TopologyTestDriver(topology, properties, startingWallClockTime.getMillis) } } diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/streams/tests/TopologyFeatureTest.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/test/TopologyFeatureTest.scala similarity index 97% rename from kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/streams/tests/TopologyFeatureTest.scala rename to kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/test/TopologyFeatureTest.scala index c1b331f0c6..e3db22ce24 100644 --- a/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/streams/tests/TopologyFeatureTest.scala +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/test/TopologyFeatureTest.scala @@ -1,4 +1,4 @@ -package com.twitter.finatra.streams.tests +package com.twitter.finatra.kafkastreams.test import com.twitter.inject.Test diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/streams/tests/TopologyTesterTopic.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/test/TopologyTesterTopic.scala similarity index 95% rename from kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/streams/tests/TopologyTesterTopic.scala rename to kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/test/TopologyTesterTopic.scala index 60e4d31d0a..6ddbe5ac41 100644 --- a/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/streams/tests/TopologyTesterTopic.scala +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/test/TopologyTesterTopic.scala @@ -1,6 +1,6 @@ -package com.twitter.finatra.streams.tests +package com.twitter.finatra.kafkastreams.test -import com.twitter.finatra.streams.converters.time._ +import com.twitter.finatra.kafkastreams.utils.time._ import org.apache.kafka.clients.producer.ProducerRecord import org.apache.kafka.common.serialization.Serde import org.apache.kafka.streams.TopologyTestDriver diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/transformer/FinatraTransformerTest.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/transformer/FinatraTransformerTest.scala new file mode 100644 index 0000000000..8d05e470f6 --- /dev/null +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/transformer/FinatraTransformerTest.scala @@ -0,0 +1,158 @@ +package com.twitter.finatra.kafkastreams.transformer + +import com.twitter.conversions.DurationOps._ +import com.twitter.finagle.stats.{NullStatsReceiver, StatsReceiver} +import com.twitter.finatra.kafkastreams.config.KafkaStreamsConfig +import com.twitter.finatra.kafkastreams.transformer.domain.Time +import com.twitter.finatra.kafkastreams.transformer.stores.CachingKeyValueStores +import com.twitter.finatra.kafkastreams.transformer.watermarks.Watermark +import com.twitter.inject.Test +import com.twitter.util.Duration +import org.apache.kafka.common.serialization.Serdes +import org.apache.kafka.streams.StreamsConfig +import org.apache.kafka.streams.processor._ +import org.apache.kafka.streams.processor.internals.{RecordCollector, ToInternal} +import org.apache.kafka.streams.state.Stores +import org.apache.kafka.test.{InternalMockProcessorContext, NoOpRecordCollector, TestUtils} +import org.hamcrest.{BaseMatcher, Description} +import org.mockito.{Matchers, Mockito} + +class FinatraTransformerTest extends Test with com.twitter.inject.Mockito { + val firstMessageTimestamp = 100000 + val firstKey = "key1" + val firstValue = "value1" + + val secondMessageTimestamp = 200000 + val secondKey = "key2" + val secondValue = "value2" + + test("watermark processing when forwarding from onMessage") { + val transformer = + new FinatraTransformer[String, String, String, String](NullStatsReceiver) { + override def onMessage(messageTime: Time, key: String, value: String): Unit = { + forward(key, value, watermark.timeMillis) + } + } + + val context = smartMock[ProcessorContext] + context.taskId() returns new TaskId(0, 0) + context.timestamp returns firstMessageTimestamp + + transformer.init(context) + transformer.transform(firstKey, firstValue) + transformer.watermark should be(Watermark(firstMessageTimestamp - 1)) + assertForwardedMessage(context, firstKey, firstValue, firstMessageTimestamp) + + context.timestamp returns secondMessageTimestamp + transformer.transform(secondKey, secondValue) + transformer.watermark should be(Watermark(firstMessageTimestamp - 1)) + assertForwardedMessage(context, secondKey, secondValue, firstMessageTimestamp) + + transformer.onFlush() + transformer.watermark should be(Watermark(secondMessageTimestamp - 1)) + } + + test("watermark processing when forwarding from caching flush listener") { + val transformer = + new FinatraTransformer[String, String, String, String](NullStatsReceiver) + with CachingKeyValueStores[String, String, String, String] { + private val cache = getCachingKeyValueStore[String, String]("mystore") + + override def statsReceiver: StatsReceiver = NullStatsReceiver + override def commitInterval: Duration = 1.second + + override def onInit(): Unit = { + super.onInit() + cache.registerFlushListener(onFlushed) + } + + override def onMessage(messageTime: Time, key: String, value: String): Unit = { + cache.put(key, value) + } + + private def onFlushed(key: String, value: String): Unit = { + forward(key = key, value = value, timestamp = watermark.timeMillis) + } + } + + val context = Mockito.spy(new FinatraMockProcessorContext) + transformer.init(context) + + context.setTime(firstMessageTimestamp) + transformer.transform(firstKey, firstValue) + + context.setTime(secondMessageTimestamp) + transformer.transform(secondKey, secondValue) + + transformer.onFlush() + assertForwardedMessage(context, firstKey, firstValue, secondMessageTimestamp) + assertForwardedMessage(context, secondKey, secondValue, secondMessageTimestamp) + } + + private def assertForwardedMessage( + context: ProcessorContext, + firstKey: String, + firstValue: String, + firstMessageTimestamp: Int + ): Unit = { + org.mockito.Mockito + .verify(context) + .forward(meq(firstKey), meq(firstValue), matchTo(firstMessageTimestamp - 1)) + } + + private def matchTo(expectedTimestamp: Int): To = { + Matchers.argThat(new BaseMatcher[To] { + override def matches(to: scala.Any): Boolean = { + val toInternal = new ToInternal + toInternal.update(to.asInstanceOf[To]) + toInternal.timestamp() == expectedTimestamp + } + + override def describeTo(description: Description): Unit = { + description.appendText(s"To(timestamp = $expectedTimestamp)") + } + }) + } + + val config = new KafkaStreamsConfig() + .commitInterval(Duration.Top) + .applicationId("test-app") + .bootstrapServers("127.0.0.1:1000") + + class FinatraMockProcessorContext + extends InternalMockProcessorContext( + TestUtils.tempDirectory, + new StreamsConfig(config.properties)) { + + override def schedule( + interval: Long, + `type`: PunctuationType, + callback: Punctuator + ): Cancellable = { + new Cancellable { + override def cancel(): Unit = { + //no-op + } + } + } + override def getStateStore(name: String): StateStore = { + val storeBuilder = Stores + .keyValueStoreBuilder( + Stores.persistentKeyValueStore(name), + Serdes.String(), + Serdes.String() + ) + + val store = storeBuilder.build + store.init(this, store) + store + } + + override def recordCollector(): RecordCollector = { + new NoOpRecordCollector + } + + override def forward[K, V](key: K, value: V, to: To): Unit = {} + } + +} diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/transformer/domain/TimeTest.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/transformer/domain/TimeTest.scala new file mode 100644 index 0000000000..db392346c4 --- /dev/null +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/transformer/domain/TimeTest.scala @@ -0,0 +1,60 @@ +package com.twitter.finatra.kafkastreams.transformer.domain + +import com.twitter.inject.Test +import com.twitter.util.Duration +import java.util.concurrent.TimeUnit +import org.joda.time.{DateTime, DateTimeConstants} + +class TimeTest extends Test { + + private val timestamp = DateTimeConstants.MILLIS_PER_WEEK + private val durationMs = DateTimeConstants.MILLIS_PER_MINUTE + + test("create Time from DateTime") { + val datetime = new DateTime(timestamp) + val time = Time.create(datetime) + assert(time.millis == datetime.getMillis) + } + + test("test nextInterval") { + val baseTime = Time(0L) + val duration = Duration(durationMs, TimeUnit.MILLISECONDS) + val offsetTime = baseTime + duration + assert(Time.nextInterval(baseTime, duration).millis == durationMs) + assert(Time.nextInterval(offsetTime, duration).millis == 2 * durationMs) + } + + test("test Time equality") { + val timeA = Time(timestamp) + val timeB = Time(timestamp) + assert(timeA == timeB) + } + + test("add Time with Time") { + val time = Time(timestamp) + Time(timestamp) + assert(time == Time(2 * timestamp)) + } + + test("add Time with Duration") { + val adjustedTime = Time(timestamp) + Duration(durationMs, TimeUnit.MILLISECONDS) + assert(adjustedTime == Time(timestamp + durationMs)) + } + + test("test nearest hour") { + val oneHourTime = Time(DateTimeConstants.MILLIS_PER_HOUR) + val twoHourTime = Time(2 * DateTimeConstants.MILLIS_PER_HOUR) + val hourTimePositiveOffset = oneHourTime + Duration(1, TimeUnit.MILLISECONDS) + val twoHourTimeNegativeOffset = twoHourTime + Duration(-1, TimeUnit.MILLISECONDS) + assert(oneHourTime.hour == oneHourTime) + assert(twoHourTime.hour == twoHourTime) + assert(hourTimePositiveOffset.hour == oneHourTime) + assert(twoHourTimeNegativeOffset.hour == oneHourTime) + } + + test("test roundDown") { + val baseTimestamp = 100L + val roundingMillis = 20L + val offsetTimestamp = baseTimestamp + (roundingMillis / 2) + assert(Time(offsetTimestamp).roundDown(roundingMillis) == Time(baseTimestamp)) + } +} diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/PersistentTimerStoreTest.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/transformer/stores/PersistentTimerStoreTest.scala similarity index 93% rename from kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/PersistentTimerStoreTest.scala rename to kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/transformer/stores/PersistentTimerStoreTest.scala index 1149a40d95..5e34035ebd 100644 --- a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/PersistentTimerStoreTest.scala +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/transformer/stores/PersistentTimerStoreTest.scala @@ -1,11 +1,11 @@ -package com.twitter.unittests +package com.twitter.finatra.kafkastreams.transformer.stores import com.twitter.finagle.stats.NullStatsReceiver import com.twitter.finatra.json.JsonDiff -import com.twitter.finatra.streams.stores.internal.FinatraKeyValueStoreImpl -import com.twitter.finatra.streams.transformer.PersistentTimerStore -import com.twitter.finatra.streams.transformer.domain.{Expire, Time, TimerMetadata, Watermark} -import com.twitter.finatra.streams.transformer.internal.domain.{Timer, TimerSerde} +import com.twitter.finatra.kafkastreams.transformer.domain.{Expire, Time, TimerMetadata} +import com.twitter.finatra.kafkastreams.transformer.stores.internal.{FinatraKeyValueStoreImpl, Timer} +import com.twitter.finatra.kafkastreams.transformer.watermarks.Watermark +import com.twitter.finatra.streams.transformer.internal.domain.TimerSerde import com.twitter.inject.Test import org.apache.kafka.common.metrics.Metrics import org.apache.kafka.common.serialization.Serdes diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/FinatraKeyValueStoreLatencyTest.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/transformer/stores/internal/FinatraKeyValueStoreLatencyTest.scala similarity index 98% rename from kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/FinatraKeyValueStoreLatencyTest.scala rename to kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/transformer/stores/internal/FinatraKeyValueStoreLatencyTest.scala index b33eac4d72..09a5890536 100644 --- a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/FinatraKeyValueStoreLatencyTest.scala +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/transformer/stores/internal/FinatraKeyValueStoreLatencyTest.scala @@ -1,8 +1,7 @@ -package com.twitter.unittests +package com.twitter.finatra.kafkastreams.transformer.stores.internal import com.twitter.finagle.stats.InMemoryStatsReceiver import com.twitter.finatra.kafka.test.utils.InMemoryStatsUtil -import com.twitter.finatra.streams.stores.internal.FinatraKeyValueStoreImpl import com.twitter.inject.Test import org.apache.kafka.common.metrics.Metrics import org.apache.kafka.common.serialization.Serdes diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/MultiSpanIteratorTest.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/transformer/utils/MultiSpanIteratorTest.scala similarity index 93% rename from kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/MultiSpanIteratorTest.scala rename to kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/transformer/utils/MultiSpanIteratorTest.scala index c62d7d5b80..f3089c5842 100644 --- a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/MultiSpanIteratorTest.scala +++ b/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/kafkastreams/transformer/utils/MultiSpanIteratorTest.scala @@ -1,6 +1,5 @@ -package com.twitter.unittests +package com.twitter.finatra.kafkastreams.transformer.utils -import com.twitter.finatra.streams.transformer.MultiSpanIterator import com.twitter.inject.Test class MultiSpanIteratorTest extends Test { diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/BUILD b/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/BUILD deleted file mode 100644 index 150a15777c..0000000000 --- a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/BUILD +++ /dev/null @@ -1,27 +0,0 @@ -junit_tests( - sources = rglobs("*.scala"), - compiler_option_sets = {"fatal_warnings"}, - strict_deps = False, - dependencies = [ - "3rdparty/jvm/ch/qos/logback:logback-classic", - "3rdparty/jvm/org/apache/kafka:kafka-clients-test", - "3rdparty/jvm/org/apache/kafka:kafka-streams", - "3rdparty/jvm/org/apache/kafka:kafka-streams-test", - "3rdparty/jvm/org/apache/kafka:kafka-streams-test-utils", - "3rdparty/jvm/org/apache/kafka:kafka-test", - "3rdparty/jvm/org/apache/zookeeper:zookeeper-client", - "3rdparty/jvm/org/apache/zookeeper:zookeeper-server", - "finatra-internal/streams/examples/tweet-word-count/src/main/scala", - "finatra/inject/inject-app/src/main/scala", - "finatra/inject/inject-core/src/main/scala", - "finatra/inject/inject-core/src/test/scala:test-deps", - "finatra/inject/inject-server/src/main/scala", - "finatra/inject/inject-server/src/test/scala:test-deps", - "finatra/inject/inject-slf4j/src/main/scala", - "finatra/kafka-streams/kafka-streams/src/main/scala", - "finatra/kafka-streams/kafka-streams/src/test/resources", - "finatra/kafka-streams/kafka-streams/src/test/scala/com/twitter:test-deps", - "finatra/kafka/src/test/scala:test-deps", - "finatra/thrift/src/test/scala:test-deps", - ], -) diff --git a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/compositesum/UserClicks.scala b/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/compositesum/UserClicks.scala deleted file mode 100644 index 2887e0c560..0000000000 --- a/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration/compositesum/UserClicks.scala +++ /dev/null @@ -1,5 +0,0 @@ -package com.twitter.unittests.integration.compositesum - -import com.twitter.unittests.integration.compositesum.UserClicksTypes.UserId - -case class UserClicks(userId: UserId, clickType: Int) diff --git a/kafka/PROJECT b/kafka/PROJECT index c57fb17484..878b3f9f1e 100644 --- a/kafka/PROJECT +++ b/kafka/PROJECT @@ -1,7 +1,9 @@ owners: + - csl-team:ldap - messaging-group:ldap - scosenza - dbress - adams watchers: + - csl-team@twitter.com - ds-messaging@twitter.com diff --git a/kafka/src/main/scala/com/twitter/finatra/kafka/config/KafkaConfig.scala b/kafka/src/main/scala/com/twitter/finatra/kafka/config/KafkaConfig.scala index 24ec5d7a20..59fef3a5da 100644 --- a/kafka/src/main/scala/com/twitter/finatra/kafka/config/KafkaConfig.scala +++ b/kafka/src/main/scala/com/twitter/finatra/kafka/config/KafkaConfig.scala @@ -44,6 +44,12 @@ trait KafkaConfigMethods[Self] extends KafkaConfig { def withConfig(key: String, value: String): This = fromConfigMap(configMap + (key -> value)) + def withConfig(key: String, value: Int): This = + fromConfigMap(configMap + (key -> value.toString)) + + def withConfig(key: String, value: Boolean): This = + fromConfigMap(configMap + (key -> value.toString)) + def withConfig(key: String, value: Duration): This = { fromConfigMap(configMap + (key -> value.inMilliseconds.toString)) } diff --git a/kafka/src/main/scala/com/twitter/finatra/kafka/consumers/FinagleKafkaConsumer.scala b/kafka/src/main/scala/com/twitter/finatra/kafka/consumers/FinagleKafkaConsumer.scala index a8e69a7b64..696e87e2f5 100644 --- a/kafka/src/main/scala/com/twitter/finatra/kafka/consumers/FinagleKafkaConsumer.scala +++ b/kafka/src/main/scala/com/twitter/finatra/kafka/consumers/FinagleKafkaConsumer.scala @@ -136,6 +136,19 @@ class FinagleKafkaConsumer[K, V](config: FinagleKafkaConsumerConfig[K, V]) singleThreadFuturePool(consumer.offsetsForTimes(timestampsToSearch)) } + /** + * Get the end offsets for the given partitions. In the default {@code read_uncommitted} isolation level, the end + * offset is the high watermark (that is, the offset of the last successfully replicated message plus one). For + * {@code read_committed} consumers, the end offset is the last stable offset (LSO), which is the minimum of + * the high watermark and the smallest offset of any open transaction. Finally, if the partition has never been + * written to, the end offset is 0. + */ + def endOffsets( + partitions: Seq[TopicPartition] + ): Future[util.Map[TopicPartition, java.lang.Long]] = { + singleThreadFuturePool(consumer.endOffsets(partitions.asJavaCollection)) + } + /** * @param timeout The time, in milliseconds, spent waiting in poll if data is not available in the buffer. * If 0, returns immediately with any records that are available currently in the buffer, else returns empty. diff --git a/kafka/src/main/scala/com/twitter/finatra/kafka/consumers/Flaggables.scala b/kafka/src/main/scala/com/twitter/finatra/kafka/consumers/Flaggables.scala new file mode 100644 index 0000000000..328c90edf5 --- /dev/null +++ b/kafka/src/main/scala/com/twitter/finatra/kafka/consumers/Flaggables.scala @@ -0,0 +1,59 @@ +package com.twitter.finatra.kafka.consumers + +import com.twitter.app.Flaggable +import com.twitter.finatra.kafka.domain.SeekStrategy +import org.apache.kafka.clients.consumer.OffsetResetStrategy + +/** + * Contains implicit Flaggable implementations for various kafka configuration types. + */ +object Flaggables { + + /** + * Allows you to create a flag which will convert the flag's input String into a + * [[com.twitter.finatra.kafka.domain.SeekStrategy]] + * + * {{{ + * import com.twitter.fanatra.kafka.consumers.Flaggables.seekStrategyFlaggable + * + * private val seekStrategyFlag = flag[SeekStrategy]( + * "seek.strategy.flag", + * SeekStrategy.RESUME, + * "This is the seek strategy flag" + * ) + * }}} + */ + implicit val seekStrategyFlaggable: Flaggable[SeekStrategy] = new Flaggable[SeekStrategy] { + override def parse(s: String): SeekStrategy = s match { + case "beginning" => SeekStrategy.BEGINNING + case "end" => SeekStrategy.END + case "resume" => SeekStrategy.RESUME + case "rewind" => SeekStrategy.REWIND + case _ => throw new IllegalArgumentException(s"$s is not a valid seek strategy.") + } + } + + /** + * Allows you to create a flag which will convert the flag's input String into a + * [[org.apache.kafka.clients.consumer.OffsetResetStrategy]] + * + * {{{ + * import org.apache.kafka.clients.consumer.OffsetResetStrategy + * + * private val offsetResetStrategyFlag = flag[OffsetResetStrategy]( + * "offset.reset.strategy.flag", + * OffsetResetStrategy.LATEST, + * "This is the offset reset strategy flag" + * ) + * }}} + */ + implicit val offsetResetStrategyFlaggable: Flaggable[OffsetResetStrategy] = + new Flaggable[OffsetResetStrategy] { + override def parse(s: String): OffsetResetStrategy = s match { + case "latest" => OffsetResetStrategy.LATEST + case "earliest" => OffsetResetStrategy.EARLIEST + case "none" => OffsetResetStrategy.NONE + case _ => throw new IllegalArgumentException(s"$s is not a valid offset reset strategy") + } + } +} diff --git a/kafka/src/main/scala/com/twitter/finatra/kafka/consumers/KafkaConsumerConfig.scala b/kafka/src/main/scala/com/twitter/finatra/kafka/consumers/KafkaConsumerConfig.scala index 8fc558ccee..ed8526060e 100644 --- a/kafka/src/main/scala/com/twitter/finatra/kafka/consumers/KafkaConsumerConfig.scala +++ b/kafka/src/main/scala/com/twitter/finatra/kafka/consumers/KafkaConsumerConfig.scala @@ -21,7 +21,25 @@ object KafkaConsumerConfig { } trait KafkaConsumerConfigMethods[Self] extends KafkaConfigMethods[Self] with Logging { - def dest(dest: String): This = bootstrapServers(BootstrapServerUtils.lookupBootstrapServers(dest)) + /** + * Configure the Kafka server the consumer will connect to. + * + * @param dest the Kafka server address + * @return the [[KafkaConsumerConfigMethods]] instance. + */ + def dest(dest: String): This = + bootstrapServers(BootstrapServerUtils.lookupBootstrapServers(dest)) + + + /** + * Configure the Kafka server the consumer will connect to. + * + * @param dest the Kafka server address + * @param timeout the timeout duration when trying to resolve the [[dest]] server. + * @return the [[KafkaConsumerConfigMethods]] instance. + */ + def dest(dest: String, timeout: Duration): This = + bootstrapServers(BootstrapServerUtils.lookupBootstrapServers(dest, timeout)) def autoCommitInterval(duration: Duration): This = withConfig(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, duration) diff --git a/kafka/src/main/scala/com/twitter/finatra/kafka/utils/BootstrapServerUtils.scala b/kafka/src/main/scala/com/twitter/finatra/kafka/utils/BootstrapServerUtils.scala index a6de38a8a4..b641bc3463 100644 --- a/kafka/src/main/scala/com/twitter/finatra/kafka/utils/BootstrapServerUtils.scala +++ b/kafka/src/main/scala/com/twitter/finatra/kafka/utils/BootstrapServerUtils.scala @@ -4,20 +4,36 @@ import com.twitter.finagle.Addr.{Bound, Failed, Neg, Pending} import com.twitter.finagle.Address.Inet import com.twitter.finagle.{Addr, Address, Namer} import com.twitter.inject.Logging -import com.twitter.util.{Await, Promise, Witness} +import com.twitter.util.{Await, Duration, Promise, Witness} import java.net.InetSocketAddress + object BootstrapServerUtils extends Logging { - def lookupBootstrapServers(dest: String): String = { + /** + * Translates the dest path into a list of servers that can be used to initialize a Kafka + * producer or consumer. It uses [[com.twitter.util.Duration.Top]] as the timeout, effectively + * waiting infinitely for the name resolution. + * @param dest The path to translate. + * @return A comma separated list of server addresses. + */ + def lookupBootstrapServers(dest: String): String = lookupBootstrapServers(dest, Duration.Top) + + /** + * Translates the dest path into a list of servers that can be used to initialize a Kafka + * producer or consumer using the specified timeout. + * @param dest The path to translate. + * @param timeout The maximum timeout for the name resolution. + * @return A comma separated list of server addresses. + */ + def lookupBootstrapServers(dest: String, timeout: Duration): String = { if (!dest.startsWith("/")) { info(s"Resolved Kafka Dest = $dest") dest } else { info(s"Resolving Kafka Bootstrap Servers: $dest") val promise = new Promise[Seq[InetSocketAddress]]() - val resolveResult = Namer - .resolve(dest).changes + val resolveResult = Namer.resolve(dest).changes .register(new Witness[Addr] { override def notify(note: Addr): Unit = note match { case Pending => @@ -32,7 +48,7 @@ object BootstrapServerUtils extends Logging { } }) - val socketAddress = Await.result(promise) + val socketAddress = Await.result(promise, timeout) resolveResult.close() val servers = socketAddress.take(5).map(a => s"${a.getAddress.getHostAddress}:${a.getPort}").mkString(",") diff --git a/kafka/src/test/scala/BUILD b/kafka/src/test/scala/BUILD index 084f3a53ce..e62ddc2012 100644 --- a/kafka/src/test/scala/BUILD +++ b/kafka/src/test/scala/BUILD @@ -31,20 +31,6 @@ scala_library( "finatra/kafka/src/test/thrift:thrift-scala", "util/util-slf4j-api/src/main/scala", ], - excludes = [ - exclude( - org = "com.twitter", - name = "twitter-server-internal-naming_2.11", - ), - exclude( - org = "com.twitter", - name = "loglens-log4j-logging_2.11", - ), - exclude( - org = "log4j", - name = "log4j", - ), - ], exports = [ "3rdparty/jvm/com/google/inject:guice", "3rdparty/jvm/junit", diff --git a/kafka/src/test/scala/com/twitter/finatra/kafka/consumers/FlaggablesTest.scala b/kafka/src/test/scala/com/twitter/finatra/kafka/consumers/FlaggablesTest.scala new file mode 100644 index 0000000000..867c41fab1 --- /dev/null +++ b/kafka/src/test/scala/com/twitter/finatra/kafka/consumers/FlaggablesTest.scala @@ -0,0 +1,24 @@ +package com.twitter.finatra.kafka.consumers + +import com.twitter.finatra.kafka.consumers.Flaggables.{offsetResetStrategyFlaggable, seekStrategyFlaggable} +import com.twitter.finatra.kafka.domain.SeekStrategy +import com.twitter.inject.Test +import org.apache.kafka.clients.consumer.OffsetResetStrategy + +class FlaggablesTest extends Test { + + test("Flaggables#seekStrategyFlaggable") { + seekStrategyFlaggable.parse("beginning") should equal(SeekStrategy.BEGINNING) + seekStrategyFlaggable.parse("resume") should equal(SeekStrategy.RESUME) + seekStrategyFlaggable.parse("rewind") should equal(SeekStrategy.REWIND) + seekStrategyFlaggable.parse("end") should equal(SeekStrategy.END) + an [IllegalArgumentException] should be thrownBy seekStrategyFlaggable.parse("unknown") + } + + test("Flaggables#offsetResetStrategyFlaggable ") { + offsetResetStrategyFlaggable.parse("latest") should equal(OffsetResetStrategy.LATEST) + offsetResetStrategyFlaggable.parse("earliest") should equal(OffsetResetStrategy.EARLIEST) + offsetResetStrategyFlaggable.parse("none") should equal(OffsetResetStrategy.NONE) + an [IllegalArgumentException] should be thrownBy offsetResetStrategyFlaggable.parse("unknown") + } +} diff --git a/kafka/src/test/scala/com/twitter/finatra/kafka/test/BootstrapServerUtilsTest.scala b/kafka/src/test/scala/com/twitter/finatra/kafka/test/BootstrapServerUtilsTest.scala new file mode 100644 index 0000000000..d488732136 --- /dev/null +++ b/kafka/src/test/scala/com/twitter/finatra/kafka/test/BootstrapServerUtilsTest.scala @@ -0,0 +1,41 @@ +package com.twitter.finatra.kafka.test + +import com.twitter.conversions.DurationOps._ +import com.twitter.finagle.{Addr, Dtab, Name, NameTree, Path} +import com.twitter.finatra.kafka.{utils => KafkaUtils} +import com.twitter.inject.Test +import com.twitter.util.{Activity, Duration, TimeoutException, Var} +import com.twitter.finagle.naming.{DefaultInterpreter, NameInterpreter} + + +class BootstrapServerUtilsTest extends Test { + + override protected def afterEach(): Unit = { + NameInterpreter.global = DefaultInterpreter + } + + test("lookup success") { + KafkaUtils.BootstrapServerUtils.lookupBootstrapServers( + "/$/inet/localhost/88", Duration.Top) should equal("127.0.0.1:88") + } + + test("lookup with timeout") { + val testingPath: String = "/s/kafka/cluster" + + // Bind the testing path to a pending address, so the name resolution will time out + NameInterpreter.global = new NameInterpreter { + override def bind(dtab: Dtab, path: Path): Activity[NameTree[Name.Bound]] = { + if (path.equals(Path.read(testingPath))) { + Activity.value(NameTree.Leaf(Name.Bound(Var.value(Addr.Pending), new Object()))) + } else { + DefaultInterpreter.bind(dtab, path) + } + } + } + + val ex = the[TimeoutException] thrownBy { + KafkaUtils.BootstrapServerUtils.lookupBootstrapServers(testingPath, 10.milliseconds) + } + ex.getMessage should equal("10.milliseconds") + } +} diff --git a/kafka/src/test/scala/com/twitter/finatra/kafka/test/integration/FinagleKafkaConsumerIntegrationTest.scala b/kafka/src/test/scala/com/twitter/finatra/kafka/test/integration/FinagleKafkaConsumerIntegrationTest.scala new file mode 100644 index 0000000000..dae558f0f8 --- /dev/null +++ b/kafka/src/test/scala/com/twitter/finatra/kafka/test/integration/FinagleKafkaConsumerIntegrationTest.scala @@ -0,0 +1,72 @@ +package com.twitter.finatra.kafka.test.integration + +import com.twitter.finatra.kafka.consumers.FinagleKafkaConsumerBuilder +import com.twitter.finatra.kafka.domain.{AckMode, KafkaGroupId} +import com.twitter.finatra.kafka.producers.FinagleKafkaProducerBuilder +import com.twitter.finatra.kafka.test.EmbeddedKafka +import com.twitter.util.Duration +import org.apache.kafka.common.TopicPartition +import org.apache.kafka.common.serialization.Serdes + +class FinagleKafkaConsumerIntegrationTest extends EmbeddedKafka { + private val testTopic = kafkaTopic(Serdes.String, Serdes.String, "test-topic") + private val emptyTestTopic = kafkaTopic(Serdes.String, Serdes.String, "empty-test-topic") + + protected lazy val producer = FinagleKafkaProducerBuilder() + .dest(brokers.map(_.brokerList()).mkString(",")) + .clientId("test-producer") + .ackMode(AckMode.ALL) + .keySerializer(Serdes.String.serializer) + .valueSerializer(Serdes.String.serializer) + .build() + + protected lazy val consumer = FinagleKafkaConsumerBuilder() + .dest(brokers.map(_.brokerList()).mkString(",")) + .clientId("test-consumer") + .groupId(KafkaGroupId("test-group")) + .keyDeserializer(Serdes.String.deserializer) + .valueDeserializer(Serdes.String.deserializer) + .requestTimeout(Duration.fromSeconds(1)) + .build() + + test("endOffset returns 0 for empty topic with no events") { + val emptyTopicPartition = new TopicPartition(emptyTestTopic.topic, 0) + val endOffsets = await(consumer.endOffsets(Seq(emptyTopicPartition))) + assert(endOffsets.get(emptyTopicPartition) == 0) + } + + test("endOffset increases by 1 after publish") { + val topicPartition = new TopicPartition(testTopic.topic, 0) + val initEndOffsets = await(consumer.endOffsets(Seq(topicPartition))) + val initEndOffset = initEndOffsets.get(topicPartition) + + await(producer.send(testTopic.topic, "Foo", "Bar", System.currentTimeMillis)) + + val writtenEndOffsets = await(consumer.endOffsets(Seq(topicPartition))) + assert(writtenEndOffsets.get(topicPartition) == initEndOffset + 1) + } + + test("endOffset increases by 3 after 3 publishes") { + val topicPartition = new TopicPartition(testTopic.topic, 0) + val initEndOffsets = await(consumer.endOffsets(Seq(topicPartition))) + val initEndOffset = initEndOffsets.get(topicPartition) + + await(producer.send(testTopic.topic, "Fee", "Bee", System.currentTimeMillis)) + await(producer.send(testTopic.topic, "Fi", "Bye", System.currentTimeMillis)) + await(producer.send(testTopic.topic, "Foo", "Boo", System.currentTimeMillis)) + + val writtenEndOffsets = await(consumer.endOffsets(Seq(topicPartition))) + assert(writtenEndOffsets.get(topicPartition) == initEndOffset + 3) + } + + test("endOffset returns empty map for empty sequence of partitions") { + val emptyEndOffsets = await(consumer.endOffsets(Seq.empty[TopicPartition])) + assert(emptyEndOffsets.size == 0) + } + + test("endOffset times out for non-existent topic") { + val notExistTopicPartition = new TopicPartition("topic-does-not-exist", 0) + assertThrows[org.apache.kafka.common.errors.TimeoutException]( + await(consumer.endOffsets(Seq(notExistTopicPartition)))) + } +} diff --git a/project/plugins.sbt b/project/plugins.sbt index 5c8c1abfae..ff035bf2db 100644 --- a/project/plugins.sbt +++ b/project/plugins.sbt @@ -3,7 +3,7 @@ resolvers ++= Seq( Resolver.sonatypeRepo("snapshots") ) -val releaseVersion = "19.1.0" +val releaseVersion = "19.2.0" addSbtPlugin("com.twitter" % "scrooge-sbt-plugin" % releaseVersion) diff --git a/thrift/src/main/scala/com/twitter/finatra/thrift/Controller.scala b/thrift/src/main/scala/com/twitter/finatra/thrift/Controller.scala index a13a07b59d..d88d319e18 100644 --- a/thrift/src/main/scala/com/twitter/finatra/thrift/Controller.scala +++ b/thrift/src/main/scala/com/twitter/finatra/thrift/Controller.scala @@ -52,9 +52,9 @@ abstract class Controller private (val config: Controller.Config) extends Loggin */ class MethodDSL[M <: ThriftMethod] (val m: M, chain: Filter.TypeAgnostic) { - private[this] def nonLegacy[T](f: ControllerConfig => T): T = config match { - case cc: ControllerConfig => f(cc) - case _: LegacyConfig => throw new IllegalStateException("Legacy controllers cannot use method DSLs") + private[this] def nonLegacy[T](f: ControllerConfig => T): T = { + assert(config.isInstanceOf[ControllerConfig], "Legacy controllers cannot use method DSLs") + f(config.asInstanceOf[ControllerConfig]) } /** @@ -67,6 +67,8 @@ abstract class Controller private (val config: Controller.Config) extends Loggin /** * Provide an implementation for this method in the form of a [[com.twitter.finagle.Service]] * + * @note The service will be called for each request. + * * @param svc the service to use as an implementation */ def withService(svc: Service[Request[M#Args], Response[M#SuccessType]]): Unit = nonLegacy { cc => @@ -75,7 +77,9 @@ abstract class Controller private (val config: Controller.Config) extends Loggin /** * Provide an implementation for this method in the form of a function of - * Request => Future[Response] + * Request => Future[Response]. + * + * @note The given function will be invoked for each request. * * @param fn the function to use */ @@ -88,6 +92,8 @@ abstract class Controller private (val config: Controller.Config) extends Loggin * This exists for legacy compatibility reasons. Users should instead use Request/Response * based functionality. * + * @note The implementation given will be invoked for each request. + * * @param f the implementation * @return a ThriftMethodService, which is used in legacy controller configurations */ @@ -119,7 +125,10 @@ abstract class Controller private (val config: Controller.Config) extends Loggin * implementation. All thrift methods that a ThriftSerivce handles must be registered using * this method to properly construct a Controller. * + * @note The provided implementation will be invoked for each request. + * * @param m The thrift method to handle. + * */ protected def handle[M <: ThriftMethod](m: M) = new MethodDSL[M](m, Filter.TypeAgnostic.Identity) } diff --git a/thrift/src/main/scala/com/twitter/finatra/thrift/modules/darktrafficmodules.scala b/thrift/src/main/scala/com/twitter/finatra/thrift/modules/darktrafficmodules.scala index d1a5ecbf90..58bf694006 100644 --- a/thrift/src/main/scala/com/twitter/finatra/thrift/modules/darktrafficmodules.scala +++ b/thrift/src/main/scala/com/twitter/finatra/thrift/modules/darktrafficmodules.scala @@ -86,7 +86,13 @@ private[modules] abstract class AbstractDarkTrafficFilterModule } /** - * A [[TwitterModule]] which configures and binds a [[DarkTrafficFilter]] to the object graph. + * A [[TwitterModule]] which configures and binds a [[DarkTrafficFilter]] to the object graph, for + * use with [[Controllers]] constructed using the legacy method. + * + * @note This [[DarkTrafficFilter]] module is to be used with [[Controllers]] which are constructed using + * the deprecated method of extending the `BaseServiceIface` of the generated Thrift service. + * For services that construct their Controllers by extending + * `Controller(GeneratedThriftService)`, use the [[ReqRepDarkTrafficFilter]] instead * * @note This is only applicable in Scala as it uses generated Scala classes and expects to configure * the [[DarkTrafficFilter]] over a [[com.twitter.finagle.Service]] that is generated from @@ -122,6 +128,19 @@ abstract class DarkTrafficFilterModule[ServiceIface <: Filterable[ServiceIface]: } } +/** + * A [[TwitterModule]] which configures and binds a [[DarkTrafficFilter]] to the object graph. + * + * @note This [[DarkTrafficFilter]] module is to be used with [[Controllers]] which are constructed by + * extending `Controller(GeneratedThriftService)`. For Controllers that are constructed using + * the deprecated method of extending `Controller with GeneratedThriftService.BaseServiceIface`, + * Use the [[DarkTrafficFilterModule]] above. + * + * @note This is only applicable in Scala as it uses generated Scala classes and expects to configure + * the [[DarkTrafficFilter]] over a [[com.twitter.finagle.Service]] that is generated from + * Finagle via generated Scala code. Users of generated Java code should use the + * [[JavaDarkTrafficFilterModule]]. + */ abstract class ReqRepDarkTrafficFilterModule[MethodIface <: Filterable[MethodIface]: ClassTag]( implicit serviceBuilder: ReqRepServicePerEndpointBuilder[MethodIface] ) extends AbstractDarkTrafficFilterModule { @@ -140,7 +159,7 @@ abstract class ReqRepDarkTrafficFilterModule[MethodIface <: Filterable[MethodIfa client: ThriftMux.Client, injector: Injector, stats: StatsReceiver - ): DarkTrafficFilter[MethodIface] = { + ): Filter.TypeAgnostic = { new DarkTrafficFilter[MethodIface]( client.servicePerEndpoint[MethodIface](dest, label), enableSampling(injector), diff --git a/thrift/src/main/scala/com/twitter/finatra/thrift/routing/routers.scala b/thrift/src/main/scala/com/twitter/finatra/thrift/routing/routers.scala index d56d7c9780..ca23d06b63 100644 --- a/thrift/src/main/scala/com/twitter/finatra/thrift/routing/routers.scala +++ b/thrift/src/main/scala/com/twitter/finatra/thrift/routing/routers.scala @@ -239,15 +239,13 @@ class ThriftRouter @Inject()(injector: Injector, exceptionManager: ExceptionMana controller: Controller, conf: Controller.ControllerConfig ): ThriftService = { - if (!conf.isValid) { - val expectStr = conf.methods.map(_.method.name).mkString("{,", ", ", "}") - val message = - s"${controller.getClass.getSimpleName} for service " + - s"${conf.gen.getClass.getSimpleName} is misconfigured. " + - s"Expected exactly one implementation for each of $expectStr but found:\n" + - conf.methods.map(m => s" - ${m.method.name}").mkString("\n") - error(message) - } + assert(conf.isValid, { + val expectStr = conf.gen.methods.map(_.name).mkString("{", ", ", "}") + val actualStr = conf.methods.map(_.method.name).mkString("{", ", ", "}") + s"${controller.getClass.getSimpleName} for service " + + s"${conf.gen.getClass.getSimpleName} is misconfigured. " + + s"Expected exactly one implementation for each of $expectStr but found $actualStr" + }) routes = conf.methods.map { cm => val method: ThriftMethod = cm.method diff --git a/thrift/src/test/scala/com/twitter/finatra/thrift/EmbeddedThriftServer.scala b/thrift/src/test/scala/com/twitter/finatra/thrift/EmbeddedThriftServer.scala index ea5f700d47..d788b97d56 100644 --- a/thrift/src/test/scala/com/twitter/finatra/thrift/EmbeddedThriftServer.scala +++ b/thrift/src/test/scala/com/twitter/finatra/thrift/EmbeddedThriftServer.scala @@ -18,7 +18,7 @@ import scala.collection.JavaConverters._ * we default to Stage.DEVELOPMENT. This makes it possible to only mock objects that are used in a given test, * at the expense of not checking that the entire object graph is valid. As such, you should always have at * least one Stage.PRODUCTION test for your service (which eagerly creates all classes at startup). - * @param useSocksProxy Use a tunneled socks proxy for external service discovery/calls (useful for manually run external + * @param useSocksProxy Use a tunneled socks proxy for external service discovery/calls (useful for manually running external * integration tests that connect to external services). * @param thriftPortFlag Name of the flag that defines the external thrift port for the server. * @param verbose Enable verbose logging during test runs. diff --git a/thrift/src/test/scala/com/twitter/finatra/thrift/tests/ControllerTest.scala b/thrift/src/test/scala/com/twitter/finatra/thrift/tests/ControllerTest.scala index 7dee184429..54124aabae 100644 --- a/thrift/src/test/scala/com/twitter/finatra/thrift/tests/ControllerTest.scala +++ b/thrift/src/test/scala/com/twitter/finatra/thrift/tests/ControllerTest.scala @@ -89,15 +89,15 @@ class ControllerTest extends Test { Future.value(Response(req.args.msg)) } - intercept[IllegalStateException] { + intercept[AssertionError] { dsl.filtered(Filter.TypeAgnostic.Identity) { args: Echo.Args => Future.value(args.msg) } } - intercept[IllegalStateException] { + intercept[AssertionError] { dsl.withFn(fn) } - intercept[IllegalStateException] { + intercept[AssertionError] { dsl.withService(Service.mk(fn)) } true diff --git a/thrift/src/test/scala/com/twitter/finatra/thrift/tests/DoEverythingThriftServerFeatureTest.scala b/thrift/src/test/scala/com/twitter/finatra/thrift/tests/DoEverythingThriftServerFeatureTest.scala index 56981343f8..6aea92e0ce 100644 --- a/thrift/src/test/scala/com/twitter/finatra/thrift/tests/DoEverythingThriftServerFeatureTest.scala +++ b/thrift/src/test/scala/com/twitter/finatra/thrift/tests/DoEverythingThriftServerFeatureTest.scala @@ -117,11 +117,11 @@ class DoEverythingThriftServerFeatureTest extends FeatureTest { await(client123.magicNum()) should equal("57") } - test("blacklist") { - val notWhitelistClient = - server.thriftClient[DoEverything[Future]](clientId = "not_on_whitelist") + test("denylist") { + val notAcceptlistClient = + server.thriftClient[DoEverything[Future]](clientId = "not_on_acceptlist") assertFailedFuture[UnknownClientIdError] { - notWhitelistClient.echo("Hi") + notAcceptlistClient.echo("Hi") } } diff --git a/thrift/src/test/scala/com/twitter/finatra/thrift/tests/EmbeddedThriftServerControllerFeatureTest.scala b/thrift/src/test/scala/com/twitter/finatra/thrift/tests/EmbeddedThriftServerControllerFeatureTest.scala index 57dc587eed..77e9483c6b 100644 --- a/thrift/src/test/scala/com/twitter/finatra/thrift/tests/EmbeddedThriftServerControllerFeatureTest.scala +++ b/thrift/src/test/scala/com/twitter/finatra/thrift/tests/EmbeddedThriftServerControllerFeatureTest.scala @@ -133,10 +133,10 @@ class EmbeddedThriftServerControllerFeatureTest extends FeatureTest { e.getMessage should include("oops") } - test("blacklist") { - val notWhitelistClient = server.thriftClient[Converter[Future]](clientId = "not_on_whitelist") + test("denylist") { + val notAcceptlistClient = server.thriftClient[Converter[Future]](clientId = "not_on_acceptlist") assertFailedFuture[UnknownClientIdError] { - notWhitelistClient.uppercase("Hi") + notAcceptlistClient.uppercase("Hi") } } diff --git a/thrift/src/test/scala/com/twitter/finatra/thrift/tests/LegacyDoEverythingThriftServerFeatureTest.scala b/thrift/src/test/scala/com/twitter/finatra/thrift/tests/LegacyDoEverythingThriftServerFeatureTest.scala index 02b2b3cc15..720128a1f1 100644 --- a/thrift/src/test/scala/com/twitter/finatra/thrift/tests/LegacyDoEverythingThriftServerFeatureTest.scala +++ b/thrift/src/test/scala/com/twitter/finatra/thrift/tests/LegacyDoEverythingThriftServerFeatureTest.scala @@ -1,6 +1,6 @@ package com.twitter.finatra.thrift.tests -import com.twitter.conversions.time._ +import com.twitter.conversions.DurationOps._ import com.twitter.doeverything.thriftscala.{Answer, DoEverything, Question} import com.twitter.finagle.http.Status import com.twitter.finagle.tracing.Trace @@ -118,11 +118,11 @@ class LegacyDoEverythingThriftServerFeatureTest extends FeatureTest { await(client123.magicNum()) should equal("57") } - test("blacklist") { - val notWhitelistClient = - server.thriftClient[DoEverything[Future]](clientId = "not_on_whitelist") + test("denylist") { + val notAcceptlistClient = + server.thriftClient[DoEverything[Future]](clientId = "not_on_acceptlist") assertFailedFuture[UnknownClientIdError] { - notWhitelistClient.echo("Hi") + notAcceptlistClient.echo("Hi") } } diff --git a/thrift/src/test/scala/com/twitter/finatra/thrift/tests/doeverything/controllers/LegacyDoEverythingThriftController.scala b/thrift/src/test/scala/com/twitter/finatra/thrift/tests/doeverything/controllers/LegacyDoEverythingThriftController.scala index a1d82a179c..0a6fc95664 100644 --- a/thrift/src/test/scala/com/twitter/finatra/thrift/tests/doeverything/controllers/LegacyDoEverythingThriftController.scala +++ b/thrift/src/test/scala/com/twitter/finatra/thrift/tests/doeverything/controllers/LegacyDoEverythingThriftController.scala @@ -1,6 +1,6 @@ package com.twitter.finatra.thrift.tests.doeverything.controllers -import com.twitter.conversions.time._ +import com.twitter.conversions.DurationOps._ import com.twitter.doeverything.thriftscala.{Answer, DoEverything, DoEverythingException} import com.twitter.doeverything.thriftscala.DoEverything.{Ask, Echo, Echo2, MagicNum, MoreThanTwentyTwoArgs, Uppercase} import com.twitter.finagle.{ChannelException, RequestException, RequestTimeoutException} diff --git a/utils/src/main/java/BUILD b/utils/src/main/java/BUILD index 4888fd8297..03bae23416 100644 --- a/utils/src/main/java/BUILD +++ b/utils/src/main/java/BUILD @@ -13,3 +13,10 @@ java_library( "3rdparty/jvm/com/google/inject:guice", ], ) + +# TODO: Remove this and references to it, +# when a fix for https://github.com/pantsbuild/pants/issues/7200 has landed. +files( + name = "pants-workaround", + sources = rglobs("*.java"), +)