Skip to content

Commit

Permalink
Merge commit 'e5ae4fe81f84ed269e95eacdf9d0f9e11d7e1532'
Browse files Browse the repository at this point in the history
  • Loading branch information
jenkins authored and jenkins committed May 8, 2018
2 parents bafb946 + e5ae4fe commit b56d939
Show file tree
Hide file tree
Showing 86 changed files with 1,484 additions and 279 deletions.
59 changes: 59 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,65 @@ All notable changes to this project will be documented in this file. Note that `

### Closed

## [finatra-18.5.0](https://github.com/twitter/finatra/tree/finatra-18.5.0) (2018-05-07)

### Added

* examples: Add external TwitterServer example. ``PHAB_ID=D161204``

### Changed

* inject-utils: Remove deprecated `c.t.inject.RootMonitor`. ``PHAB_ID=D161036``

* finatra-http: Updated `c.t.finatra.http.AdminHttpServer` to isolate routes added to the
admin. ``PHAB_ID=D157818``

### Fixed

* inject-slf4j, finatra-http: Fix `c.t.inject.logging.FinagleMDCAdapter` to initialize
properly. We were lazily initializing the backing `java.util.Map` of the `FinagleMDCAdapter`
which could cause values to disappear when the map was not created eagerly enough. Typical
usage would add one of the MDC logging filters to the top of the request filter chain which would
put a value into the MDC thus creating the backing `java.util.Map` early in the request chain.
However, if a filter which puts to the MDC was not included and the first put happened in a
Future closure the map state would be lost upon exiting the closure.

This change updates how the MDC mapping is stored to move from a `Local` to a `LocalContext`
and introduces new ergonomics for using/initializing the framework MDC integration.

Initialization of the MDC integration should now go through the `c.t.inject.logging.MDCInitializer`
(that is users are not expected to need to interact directly with the `FinagleMDCAdapter`). E.g.,
to initialize the MDC:

```
com.twitter.inject.logging.MDCInitializer.init()
```

This will initialize the `org.slf4j.MDC` and swap out the default `org.slf4j.spi.MDCAdapter` with
an instance of the `c.t.inject.logging.FinagleMDCAdapter` allowing for reading/writing MDC values
across Future boundaries.

Then to start the scoping of an MDC context, use `c.t.inject.logging.MDCInitializer#let`:

```
com.twitter.inject.logging.MDCInitializer.let {
// operations which set and read MDC values
???
}
```

Typically, this is done in a Filter wrapping the execution of the service in the Filter's apply,
For example, the framework provides this initialization and scoping in both the
`c.t.finatra.http.filters.LoggingMDCFilter` and the `c.t.finatra.thrift.filters.LoggingMDCFilter`.

Simply including these at the top of the request filter chain for a service will allow MDC
integration to function properly. ``PHAB_ID=D159536``

* inject-app: Ensure that installed modules are de-duped before creating injector.
``PHAB_ID=D160955``

### Closed

## [finatra-18.4.0](https://github.com/twitter/finatra/tree/finatra-18.4.0) (2018-04-10)

### Added
Expand Down
57 changes: 35 additions & 22 deletions build.sbt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import scoverage.ScoverageKeys
concurrentRestrictions in Global += Tags.limit(Tags.Test, 1)

// All Twitter library releases are date versioned as YY.MM.patch
val releaseVersion = "18.4.0"
val releaseVersion = "18.5.0"

lazy val buildSettings = Seq(
version := releaseVersion,
Expand Down Expand Up @@ -91,12 +91,12 @@ lazy val scalaCompilerOptions = scalacOptions ++= Seq(

lazy val baseSettings = Seq(
libraryDependencies ++= Seq(
"org.mockito" % "mockito-core" % versions.mockito % "test",
"org.scalacheck" %% "scalacheck" % versions.scalaCheck % "test",
"org.scalatest" %% "scalatest" % versions.scalaTest % "test",
"org.specs2" %% "specs2-core" % versions.specs2 % "test",
"org.specs2" %% "specs2-junit" % versions.specs2 % "test",
"org.specs2" %% "specs2-mock" % versions.specs2 % "test"
"org.mockito" % "mockito-core" % versions.mockito % Test,
"org.scalacheck" %% "scalacheck" % versions.scalaCheck % Test,
"org.scalatest" %% "scalatest" % versions.scalaTest % Test,
"org.specs2" %% "specs2-core" % versions.specs2 % Test,
"org.specs2" %% "specs2-junit" % versions.specs2 % Test,
"org.specs2" %% "specs2-mock" % versions.specs2 % Test
),
resolvers ++= Seq(
Resolver.sonatypeRepo("releases"),
Expand Down Expand Up @@ -170,7 +170,7 @@ lazy val publishSettings = Seq(

lazy val slf4jSimpleTestDependency = Seq(
libraryDependencies ++= Seq(
"org.slf4j" % "slf4j-simple" % versions.slf4j % "test"
"org.slf4j" % "slf4j-simple" % versions.slf4j % Test
)
)

Expand All @@ -191,9 +191,12 @@ lazy val baseServerSettings = baseSettings ++ buildSettings ++ publishSettings +
)

lazy val exampleServerSettings = baseServerSettings ++ Seq(
fork in run := true,
javaOptions in Test ++= Seq("-Dlog.service.output=/dev/stdout", "-Dlog.access.output=/dev/stdout", "-Dlog_level=INFO"),
libraryDependencies ++= Seq(
"org.slf4j" % "slf4j-simple" % versions.slf4j % "test",
"ch.qos.logback" % "logback-classic" % versions.logback)
"com.twitter" %% "twitter-server-logback-classic" % versions.twLibVersion,
"ch.qos.logback" % "logback-classic" % versions.logback
)
)

lazy val finatraModules = Seq[sbt.ProjectReference](
Expand All @@ -220,6 +223,7 @@ lazy val finatraExamples =
benchmarkServer,
exampleHttpJavaServer,
exampleInjectJavaServer,
exampleTwitterServer,
exampleWebDashboard,
helloWorld,
streamingExample,
Expand Down Expand Up @@ -247,7 +251,7 @@ lazy val root = (project in file("."))
-- inProjects(benchmarks)
// START EXAMPLES
-- inProjects(benchmarkServer, exampleHttpJavaServer, exampleInjectJavaServer,
exampleWebDashboard, helloWorld,
exampleTwitterServer, exampleWebDashboard, helloWorld,
streamingExample, thriftExampleIdl, thriftExampleServer,
thriftJavaExampleIdl, thriftJavaExampleServer, twitterClone)
// END EXAMPLES
Expand Down Expand Up @@ -282,8 +286,8 @@ lazy val injectCore = (project in file("inject/inject-core"))
"net.codingwell" %% "scala-guice" % versions.scalaGuice,
"org.joda" % "joda-convert" % versions.jodaConvert,
"org.scala-lang" % "scalap" % scalaVersion.value,
"com.google.inject" % "guice" % versions.guice % "test",
"com.google.inject.extensions" % "guice-testlib" % versions.guice % "test"
"com.google.inject" % "guice" % versions.guice % Test,
"com.google.inject.extensions" % "guice-testlib" % versions.guice % Test
),
publishArtifact in Test := true,
mappings in (Test, packageBin) := {
Expand Down Expand Up @@ -362,7 +366,7 @@ lazy val injectApp = (project in file("inject/inject-app"))
}
).dependsOn(
injectCore % "test->test;compile->compile",
injectModules % "test")
injectModules % Test)

lazy val injectServerTestJarSources =
Seq(
Expand All @@ -380,7 +384,7 @@ lazy val injectServer = (project in file("inject/inject-server"))
moduleName := "inject-server",
ScoverageKeys.coverageExcludedPackages := "<empty>;.*Ports.*;.*FinagleBuildRevision.*",
libraryDependencies ++= Seq(
"com.google.guava" % "guava" % versions.guava % "test",
"com.google.guava" % "guava" % versions.guava % Test,
"com.twitter" %% "finagle-stats" % versions.twLibVersion,
"com.twitter" %% "twitter-server" % versions.twLibVersion
),
Expand Down Expand Up @@ -408,10 +412,10 @@ lazy val injectSlf4j = (project in file("inject/inject-slf4j"))
.settings(
name := "inject-slf4j",
moduleName := "inject-slf4j",
ScoverageKeys.coverageExcludedPackages := "<empty>;.*LoggerModule.*;.*Slf4jBridgeUtility.*",
ScoverageKeys.coverageExcludedPackages := "<empty>;",
libraryDependencies ++= Seq(
"com.fasterxml.jackson.core" % "jackson-annotations" % versions.jackson,
"com.twitter" %% "util-core" % versions.twLibVersion,
"com.twitter" %% "finagle-core" % versions.twLibVersion,
"com.twitter" %% "util-slf4j-api" % versions.twLibVersion,
"org.slf4j" % "jcl-over-slf4j" % versions.slf4j,
"org.slf4j" % "jul-to-slf4j" % versions.slf4j,
Expand Down Expand Up @@ -458,7 +462,7 @@ lazy val injectThriftClient = (project in file("inject/inject-thrift-client"))
"com.twitter" %% "finagle-thrift" % versions.twLibVersion,
"com.twitter" %% "finagle-thriftmux" % versions.twLibVersion,
"com.github.nscala-time" %% "nscala-time" % versions.nscalaTime,
"com.twitter" %% "finagle-http" % versions.twLibVersion % "test")
"com.twitter" %% "finagle-http" % versions.twLibVersion % Test)
).dependsOn(
injectCore % "test->test;compile->compile",
injectUtils,
Expand All @@ -474,7 +478,6 @@ lazy val injectUtils = (project in file("inject/inject-utils"))
moduleName := "inject-utils",
libraryDependencies ++= Seq(
"com.twitter" %% "finagle-core" % versions.twLibVersion,
"com.twitter" %% "finagle-mux" % versions.twLibVersion,
"com.twitter" %% "util-core" % versions.twLibVersion,
"commons-lang" % "commons-lang" % versions.commonsLang
)
Expand All @@ -498,7 +501,7 @@ lazy val benchmarks = project
.enablePlugins(JmhPlugin)
.settings(
libraryDependencies ++= Seq(
"org.slf4j" % "slf4j-simple" % versions.slf4j % "test"))
"org.slf4j" % "slf4j-simple" % versions.slf4j % Test))
.settings(noPublishSettings)
.dependsOn(
http,
Expand Down Expand Up @@ -624,7 +627,7 @@ lazy val http = project
}
).dependsOn(
jackson % "test->test;compile->compile",
injectRequestScope % "test",
injectRequestScope % Test,
injectSlf4j,
injectServer % "test->test;compile->compile",
httpclient % "test->test",
Expand All @@ -641,7 +644,7 @@ lazy val httpclient = project
"commons-codec" % "commons-codec" % versions.commonsCodec,
"com.twitter" %% "finagle-core" % versions.twLibVersion,
"com.twitter" %% "finagle-http" % versions.twLibVersion,
"com.twitter" %% "twitter-server" % versions.twLibVersion % "test",
"com.twitter" %% "twitter-server" % versions.twLibVersion % Test,
"com.twitter" %% "util-core" % versions.twLibVersion
),
publishArtifact in Test := true,
Expand Down Expand Up @@ -886,4 +889,14 @@ lazy val exampleWebDashboard = (project in file("examples/web-dashboard"))
injectCore % "test->test",
injectSlf4j)

lazy val exampleTwitterServer = (project in file("examples/example-twitter-server"))
.settings(exampleServerSettings)
.settings(noPublishSettings)
.settings(
name := "example-twitter-server",
moduleName := "example-twitter-server"
).dependsOn(
injectServer % "test->test;compile->compile",
injectSlf4j,
utils)
// END EXAMPLES
2 changes: 2 additions & 0 deletions doc/src/sphinx/user-guide/index.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ Logging
-------

- :doc:`logging/index`
- :doc:`logging/mdc`
- :doc:`logging/logback`

Injectable App
Expand Down Expand Up @@ -98,6 +99,7 @@ Testing
getting-started/futures
getting-started/examples
logging/index
logging/mdc
logging/logback
app/index
twitter-server/index
Expand Down
63 changes: 41 additions & 22 deletions doc/src/sphinx/user-guide/logging/logback.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,16 @@
Logback
=======

We highly recommend using `Logback <http://logback.qos.ch/>`__ as an SLF4J binding (logging implementation).
We highly recommend using `Logback <http://logback.qos.ch/>`__ as an SLF4J binding (logging
implementation).

.. admonition:: From the Logback documentation

Logback is intended as a successor to the popular log4j project, picking up where log4j leaves off.
Logback is intended as a successor to the popular log4j project, picking up where log4j leaves off.

Logback's architecture is sufficiently generic so as to apply under different circumstances. At present time, logback is divided into three modules, logback-core, logback-classic and logback-access.
Logback's architecture is sufficiently generic so as to apply under different circumstances. At
present time, logback is divided into three modules, logback-core, logback-classic and
logback-access.

See Logback's documentation on `reasons to switch from Log4j <https://logback.qos.ch/reasonsToSwitch.html>`__.

Expand All @@ -24,7 +27,7 @@ E.g., with `sbt <http://www.scala-sbt.org/>`__:

Where, `VERSION` represents the version of Logback you want to use.

Similarily, with `Maven <http://maven.apache.org/>`__:
Similarly, with `Maven <http://maven.apache.org/>`__:

.. code:: xml
Expand All @@ -34,12 +37,29 @@ Similarily, with `Maven <http://maven.apache.org/>`__:
<version>VERSION</version>
</dependency>
This will provide the `logback-classic` SLF4J implementation which when used in addition to the logging bridges provided transitively through the `finatra/inject-slf4j <https://github.com/twitter/finatra/tree/develop/inject/inject-slf4j>`__ module will configure your SLF4J logging implementation and `correctly bridge all other legacy APIs <https://www.slf4j.org/legacy.html>`__.
This will provide the `logback-classic` SLF4J implementation. And when used in addition to the
logging bridges provided transitively through the `finatra/inject-slf4j <https://github.com/twitter/finatra/tree/develop/inject/inject-slf4j>`__
module will configure your SLF4J logging implementation and `correctly bridge all other legacy
APIs <https://www.slf4j.org/legacy.html>`__.

.. note:: To configure dynamic changing log levels with `TwitterServer <https://twitter.github.io/twitter-server/>`__,
be sure to *also* include a dependency on the TwitterServer Logback library.

With `sbt <http://www.scala-sbt.org/>`__:

::

"com.twitter" % "twitter-server-logback-classic" % VERSION

See the TwitterServer `documentation <https://twitter.github.io/twitter-server/Features.html#dynamically-change-log-levels>`__
for more information on configuration for dynamically changing log levels.


Logback Configuration
---------------------

See the `Logback documentation on configuration <http://logback.qos.ch/manual/configuration.html>`__ for detailed information on configuring Logback logging.
See the Logback documentation on `configuration <http://logback.qos.ch/manual/configuration.html>`__
for detailed information on configuring Logback logging.

Examples
^^^^^^^^
Expand All @@ -56,32 +76,31 @@ configuration file makes use of `Logback's variable substitution <http://logback

.. code:: xml
${log.access.output:-access.log}
${log.service.output:-service.log}
${log.access.output:-access.log}
${log.service.output:-service.log}
for configuring the `<file/>` output of the appenders. A variable's value is specified at
runtime as a Java system property, e.g.,

.. code:: bash
-Dlog.service.output=myservice.log
-Dlog.service.output=myservice.log
If a value is not provided the inlined default will be used, i.e., what is on the other side of the `:-`, e.g., `access.log` or `service.log`.
If a value is not provided the inlined default will be used, i.e., what is on the other side of the
`:-`, e.g., `access.log` or `service.log`.

More information about Logback's variable substitution may be found `here <http://logback.qos.ch/manual/configuration.html#variableSubstitution>`__.

Rerouting `java.util.logging`
-------------------------------

Additional configuration is necessary to reroute the `java.util.logging` system. The reason is that the `jul-to-slf4j` bridge cannot replace classes in the `java.util.logging` package to do the redirection statically as it does for the other bridge implementations. Instead, it has to register a handler on the root logger and listen for logging statements like any other handler. It will then redirect those logging statements appropriately. This redirection is accomplished via SLF4J's `SLF4JBridgeHandler` [`documentation <http://www.slf4j.org/api/org/slf4j/bridge/SLF4JBridgeHandler.html>`__\ ]. `finatra/inject-modules <https://github.com/twitter/finatra/tree/develop/inject/inject-modules>`__ provides a module for initializing this bridge handler via the `LoggerModule <https://github.com/twitter/finatra/blob/develop/inject/inject-modules/src/main/scala/com/twitter/inject/modules/LoggerModule.scala>`__. This module is added by the framework to Http and Thrift servers by default.

Mapped Diagnostic Context Filter
--------------------------------

Finatra offers an integration with Logback's `Mapped Diagnostic Context <http://logback.qos.ch/manual/mdc.html>`__ (MDC) for consistent logging of useful information. For example, the `hello-world configuration <https://github.com/twitter/finatra/blob/c6e4716f082c0c8790d06d9e1664aacbd0c3fede/examples/hello-world/src/main/resources/logback.xml#L25>`__ references the contextual key "traceId" which will be logged with every statement sent to the appender. It is the responsibility of the application to populate the MDC with this contextual information. In this case, the "traceId" is added by including either the http `TraceIdMDCLoggingFilter <https://github.com/twitter/finatra/blob/develop/http/src/main/scala/com/twitter/finatra/http/filters/TraceIdMDCFilter.scala>`__ or thrift `TraceIdMDCLoggingFilter <https://github.com/twitter/finatra/blob/develop/thrift/src/main/scala/com/twitter/finatra/thrift/filters/TraceIdMDCFilter.scala>`__.

In order to use the Mapped Diagnostic Context, an application must first initialize the MDC `Finagle <https://twitter.github.io/finagle/>`__ adapter provided by Finatra. This initialization is provided by including either the http `LoggingMDCFilter <https://github.com/twitter/finatra/blob/develop/http/src/main/scala/com/twitter/finatra/http/filters/LoggingMDCFilter.scala>`__ or thrift `LoggingMDCFilter <https://github.com/twitter/finatra/blob/develop/thrift/src/main/scala/com/twitter/finatra/thrift/filters/LoggingMDCFilter.scala>`__ in your filter chain.

Make sure to place the `LoggingMDCFilter` before any other filters which will add MDC entries or expect MDC entries to be present.

See `Logback's documentation <http://logback.qos.ch/manual/mdc.html>`__ for more information the MDC functionality.
Additional configuration is necessary to reroute the `java.util.logging` system. The reason is that
the `jul-to-slf4j` bridge cannot replace classes in the `java.util.logging` package to do the
redirection statically as it does for the other bridge implementations. Instead, it has to register
a handler on the root logger and listen for logging statements like any other handler. It will then
redirect those logging statements appropriately. This redirection is accomplished via SLF4J's
`SLF4JBridgeHandler` [`documentation <http://www.slf4j.org/api/org/slf4j/bridge/SLF4JBridgeHandler.html>`__\ ].
`finatra/inject-modules <https://github.com/twitter/finatra/tree/develop/inject/inject-modules>`__
provides a module for initializing this bridge handler via the
`LoggerModule <https://github.com/twitter/finatra/blob/develop/inject/inject-modules/src/main/scala/com/twitter/inject/modules/LoggerModule.scala>`__.
This module is added by the framework to Http and Thrift servers by default.
Loading

0 comments on commit b56d939

Please sign in to comment.