Add task for http persistence (#34)

* Add and test (kind of) task for http persistence * Remove local network url from build.sbt * Update README * Update README * fix implicit without type annotation (scala 3 error) * set subproject names to nicer values * run tests with cross scala versions * correct plugin dep name in scripted tests * remove note about untested js implementation the app tests only verify that, when talking to an actual postgres database, the magic skunk strings are correct. those strings are shared, so the database interaction is tested in JS in one sense (barring any really weird skunk regressions that somehow aren't caught upstream -- this is almost an impossibility I think). also, testing the JS app is included in the ci-test command, it's just that there aren't currently any tests that get executed as part of it. if there are in the future though, they'll be tested, and invoking the test command even without any tests still at least cross-compiles the sjs code with scala 2.13 and scala 3. * Update README.md Co-authored-by: Juan Pedro Moreno <[email protected]> Co-authored-by: Juan Pedro Moreno <[email protected]>
47degrees · Jun 6, 2022 · 351df9a · 351df9a
1 parent 1922950
commit 351df9a
Show file tree

Hide file tree

Showing 13 changed files with 265 additions and 17 deletions.
diff --git a/README.md b/README.md
@@ -8,8 +8,7 @@ In order to use this plugin, you'll need to provide a jar containing `jRAPL` as
 
 ## Usage
 
-This plugin requires sbt 1.0.0+. It provides three tasks: `energyMonitorPreSample`, `energyMonitorPostSample`, and `energyMonitorPostSampleGitHub`.
-These tasks are controlled by two settings: `energyMonitorDisableSampling` and `energyMonitorOutputFile`.
+This plugin requires sbt 1.0.0+. It provides four tasks: `energyMonitorPreSample`, `energyMonitorPostSample`, `energyMonitorPostSampleHttp`, and `energyMonitorPostSampleGitHub` and appropriate settings for configuring them.
 
 ### `energyMonitorPreSample`
 
@@ -24,6 +23,37 @@ and prints calculated energy usage to the console.
 
 If `energyMonitorDisableSampling` is true, this task will do nothing.
 
+### `energyMonitorPostSampleHttp`
+
+This task works like `energyMonitorPostSample`, but instead of printing results to the console, sends
+them to an HTTP server with some metadata. The server should accept POSTs to the configured URL, with an HTTP body like:
+
+```json
+{
+  "joules": 23,
+  "seconds": 8,
+  "organization": "47degrees",
+  "repository": "sbt-energymonitor",
+  "branch": "abcde",
+  "run": 8,
+  "recordedAt": "2022-05-05T10:15:00Z"
+}
+```
+
+Its behavior is controlled by three settings: `energyMonitorPersistenceServerUrl`, and `energyMonitorPersistenceTag`.
+
+However, if `energyMonitorDisableSampling` is `true`, this task will do nothing.
+
+You can see an example usage of this task in the scripted test included in this repository under the `energy-monitor-plugin/src/sbt-test/http-store` directory.
+
+#### `energyMonitorPersistenceServerUrl`
+
+This setting determines where the task should send the information about energy consumption. Its default value is `http://localhost:8080`, which will be correct if you're experimenting locally and using the demo server provided in the `energyMonitorPersistenceApp` module. You _should not use a local server for real testing though_, since the server's energy consumption will also show up in the power consumed by the CPU / memory during your energy tests.
+
+#### `energyMonitorPersistenceTag`
+
+This setting determines whether some arbitrary string will be included with the energy consumption sample. You might want to do this if there's some significant change that you think should explain differences in energy, for instance, "upgrade to cats-effect 3", or "refine types to shrink validation in core business logic," or something similar. You can put whatever information you want in the tag, or nothing at all. Its default value is `None`.
+
 ### `energyMonitorPostSampleGitHub`
 
 This task works like `energyMonitorPostSample`, but instead of printing results to the console, sends them to a pull request comment.

diff --git a/build.sbt b/build.sbt
@@ -32,6 +32,17 @@ lazy val Version = new {
   val weaver = "0.7.12"
 }
 
+addCommandAlias(
+  "libraries-test",
+  List(
+    "+energyMonitorPersistenceCoreJS/test",
+    "+energyMonitorPersistenceCoreJVM/test",
+    "+energyMonitorPersistenceAppJVM/test",
+    "+energyMonitorPersistenceAppJVM/docker",
+    "+energyMonitorPersistenceAppJS/test"
+  ).mkString(";")
+)
+
 addCommandAlias(
   "ci-test",
   List(
@@ -40,12 +51,7 @@ addCommandAlias(
     "energyMonitorPlugin/test",
     "energyMonitorPlugin/publishLocal",
     "energyMonitorPlugin/scripted",
-    "energyMonitorPersistenceCoreJS/test",
-    "energyMonitorPersistenceCoreJVM/test",
-    "energyMonitorPersistenceAppJVM/test",
-    "energyMonitorPersistenceAppJVM/docker"
-    // the JS app implementation is untested for now, since there are some
-    // linking errors and it's not critical to the current scope of work
+    "libraries-test"
   ).mkString(";")
 )
 addCommandAlias("ci-publish", "github; ci-release")
@@ -94,7 +100,8 @@ lazy val energyMonitorPlugin =
       sbtPlugin := true,
       // set up 'scripted; sbt plugin for testing sbt plugins
       scriptedLaunchOpts ++=
-        Seq("-Xmx1024M", "-Dplugin.version=" + version.value)
+        Seq("-Xmx1024M", "-Dplugin.version=" + version.value),
+      name := "energy-monitor-plugin"
     )
 
 lazy val appSettings = Seq(
@@ -126,7 +133,8 @@ lazy val energyMonitorPersistenceCore =
         "org.http4s" %%% "http4s-circe" % Version.http4s,
         "org.http4s" %%% "http4s-core" % Version.http4s,
         "org.http4s" %%% "http4s-dsl" % Version.http4s
-      )
+      ),
+      name := "energy-monitor-persistence-core"
     )
 
 lazy val appImageName = "energy-monitor-persistence-app"
@@ -157,7 +165,8 @@ lazy val energyMonitorPersistenceApp =
         "org.typelevel" %%% "munit-cats-effect-3" % Version.munitCatsEffect % Test,
         "org.typelevel" %%% "scalacheck-effect-munit" % Version.scalacheckEffect % Test,
         "org.typelevel" %%% "squants" % Version.squants
-      )
+      ),
+      name := "energy-monitor-persistence-app"
     )
     .settings(
       appSettings: _*

diff --git a/...gy-monitor-persistence-core/shared/src/main/scala/energy-monitor-persistence/Routes.scala b/...gy-monitor-persistence-core/shared/src/main/scala/energy-monitor-persistence/Routes.scala
@@ -2,7 +2,7 @@ package energymonitor.persistence
 
 import cats.effect.Concurrent
 import cats.syntax.flatMap._
-import org.http4s.HttpRoutes
+import org.http4s.{EntityDecoder, HttpRoutes}
 import org.http4s.circe.CirceEntityEncoder._
 import org.http4s.circe.jsonOf
 import org.http4s.dsl.Http4sDsl
@@ -11,7 +11,8 @@ class Routes[F[_]: Concurrent](
     energyDiffRepository: EnergyDiffRepository[F]
 ) extends Http4sDsl[F] {
 
-  implicit val energyDiffCodec = jsonOf[F, EnergyDiff]
+  implicit val energyDiffCodec: EntityDecoder[F, EnergyDiff] =
+    jsonOf[F, EnergyDiff]
 
   private object BranchParam
       extends OptionalQueryParamDecoderMatcher[String]("branch")

diff --git a/...monitor-persistence-core/shared/src/test/scala/energy-monitor-persistence/Implicits.scala b/...monitor-persistence-core/shared/src/test/scala/energy-monitor-persistence/Implicits.scala
@@ -12,7 +12,9 @@ trait Implicits {
   private val genEnergyDiff: Gen[EnergyDiff] = for {
     joules <- Gen.double.filter(_.isFinite)
     seconds <- Gen.double.filter(_.isFinite)
-    recordedAt <- Gen.choose(0L, Int.MaxValue).map(Instant.ofEpochSecond(_))
+    recordedAt <- Gen
+      .choose(0L, Int.MaxValue.toLong)
+      .map(Instant.ofEpochSecond(_))
     run <- arbitrary[Int]
     organization <- Gen.alphaStr.filter(_.nonEmpty)
     repository <- Gen.alphaStr.filter(_.nonEmpty)

diff --git a/energy-monitor-plugin/src/main/scala/energymonitor/EnergyMonitorPlugin.scala b/energy-monitor-plugin/src/main/scala/energymonitor/EnergyMonitorPlugin.scala
@@ -23,18 +23,74 @@ import github4s.GithubConfig
 import github4s.http.HttpClient
 import github4s.interpreters.IssuesInterpreter
 import github4s.interpreters.StaticAccessToken
+import io.circe.Encoder
+import io.circe.syntax._
 import jRAPL.EnergyDiff
+import org.http4s.Method
+import org.http4s.Request
+import org.http4s.Uri
 import org.http4s.blaze.client.BlazeClientBuilder
 import sbt.Keys.streams
 import sbt._
 import sbt.plugins.JvmPlugin
 
 import java.nio.file.Paths
+import java.time.Instant
 
 import sRAPL.{postSample, preSample}
 
 object EnergyMonitorPlugin extends AutoPlugin {
 
+  /** Record an energy measurement with metadata
+    *
+    * This class mirrors the EnergyDiff class defined in the persistence
+    * submodules, but because plugins must use the same Scala version as that
+    * used to build sbt, adding a dependency would introduce additional
+    * cross-publication overhead in the other submodules.
+    *
+    * Additionally, this is a bit looser with the types (not using squants
+    * energy and time unit types), since it's private to the plugin so no one
+    * can accidentally depend on it, and to avoid adding an additional
+    * dependency to the plugin.
+    */
+  private case class EnergyMeasurement(
+      joules: Double,
+      seconds: Double,
+      recordedAt: Instant,
+      run: Int,
+      organization: String,
+      repository: String,
+      branch: String,
+      tag: Option[String]
+  )
+
+  private object EnergyMeasurement {
+    // This is definitely derivable, but I don't want to play with derivation
+    // in plugins.
+    implicit val encEnergyMeasurement: Encoder[EnergyMeasurement] =
+      Encoder.forProduct8(
+        "joules",
+        "seconds",
+        "recordedAt",
+        "run",
+        "organization",
+        "repository",
+        "branch",
+        "tag"
+      )(em =>
+        (
+          em.joules,
+          em.seconds,
+          em.recordedAt,
+          em.run,
+          em.organization,
+          em.repository,
+          em.branch,
+          em.tag
+        )
+      )
+  }
+
   override def trigger = allRequirements
   override def requires = JvmPlugin
 
@@ -45,6 +101,21 @@ object EnergyMonitorPlugin extends AutoPlugin {
     val energyMonitorDisableSampling = settingKey[Boolean](
       "Disable energy monitoring. The task keys will be available, but on invocation, they won't cause samples to be collected."
     )
+    val energyMonitorPersistenceServerUrl = settingKey[String](
+      """
+      | Server location to post sampling results to.
+      | For an example server implementation, see the energyMonitorPersistenceApp subproject
+      | By default, this is set to localhost:8080, assuming that the server is running on the same host
+      | as the tests. This is a very bad idea for getting reliable energy consumption results, since CPUs will consume
+      | energy for the server, the tests, and anything else running on the machine where you're running the tests. So,
+      | if you want to persist resluts to an HTTP server, you should run it somewhere else.""".trim.stripMargin
+    )
+    val energyMonitorPersistenceTag = settingKey[Option[String]](
+      """
+      | Tag to describe this energy monitoring run. If you're doing a lot of measurement, this can be useful
+      | to provide some kind of narrative about what changed that you think might cause some significant difference.
+      | """.trim.stripMargin
+    )
     val energyMonitorPreSample = taskKey[Unit](
       "Collect power consumption statistics before doing work. This task writes result to the value of energyMonitorOutputFile"
     )
@@ -57,10 +128,20 @@ object EnergyMonitorPlugin extends AutoPlugin {
       | Pull request, repository, and authentication information will be pulled from the environment.
       """.trim().stripMargin
     )
+    val energyMonitorPostSampleHttp = taskKey[Unit](
+      """
+      | Collect power consumption statistics after doing work, and send them to an http server matching the demo implementation
+      | provided in this repo. Metadata about the organization, repository, branch, and run number will be included in the request.
+      | The location of the server can be controlled with the energyMonitorPersistenceServerUrl setting key.
+      """.trim.stripMargin
+    )
+
   }
 
   import autoImport._
+
   implicit val runtime = IORuntime.global
+
   val disabledSamplingMessage =
     "Sampling disabled, not attempting to collect energy consumption stats"
 
@@ -108,6 +189,18 @@ object EnergyMonitorPlugin extends AutoPlugin {
     }
   }
 
+  private def postMeasurement(
+      measurement: EnergyMeasurement,
+      serverUrl: Uri
+  ): IO[Unit] =
+    BlazeClientBuilder[IO].resource.use { client =>
+      val request = Request[IO](
+        Method.POST,
+        serverUrl
+      ).withEntity(measurement.asJson.noSpaces)
+      client.run(request).use_
+    }
+
   def preSampleTask = Def.task[Unit] {
     val log = streams.value.log
     if (energyMonitorDisableSampling.value) {
@@ -162,18 +255,75 @@ object EnergyMonitorPlugin extends AutoPlugin {
     )(_.unsafeRunSync)
   }
 
+  val postSampleHttpTask = Def.task[Unit] {
+    val log = streams.value.log
+    val env = sys.env
+    (
+      env.get("GITHUB_REPOSITORY"),
+      env.get("GITHUB_RUN_ATTEMPT") map { _.toInt },
+      env.get("GITHUB_REF_NAME")
+    ).mapN { case (orgRepo, runAttempt, refName) =>
+      val persistenceTag = energyMonitorPersistenceTag.value
+      if (!energyMonitorDisableSampling.value) {
+        val owner :: repo :: Nil = orgRepo.split("/").toList
+        postSample(Paths.get(energyMonitorOutputFile.value)) flatMap { diff =>
+          val samples = diff.getPrimitiveSample()
+          val duration = diff.getTimeElapsed()
+          val totalJoules = samples.sum
+          val payload = EnergyMeasurement(
+            totalJoules,
+            duration.toMillis().toDouble / 1000d,
+            Instant.now(),
+            runAttempt,
+            owner,
+            repo,
+            refName,
+            persistenceTag
+          )
+          val serverLocation = energyMonitorPersistenceServerUrl.value
+          Uri.fromString(serverLocation) match {
+            case Left(e) =>
+              IO(
+                log.warn(
+                  s"Couldn't convert provided server url to a URI. You provided: $serverLocation. The error was: $e"
+                )
+              ) >> IO.raiseError(e)
+            case Right(uri) =>
+              postMeasurement(payload, uri)
+
+          }
+        }
+      } else {
+        IO {
+          log.info(
+            "Sampling is disabled, not attempting to POST an energy diff to GitHub"
+          )
+        }
+      }
+
+    }.fold(
+      log.warn(
+        "Could not obtain GitHub information from the environment. Check GITHUB_REF, GITHUB_REPOSITORY, and GITHUB_TOKEN env variables."
+      )
+    )(_.unsafeRunSync())
+  }
+
   override lazy val projectSettings = Seq(
     energyMonitorDisableSampling := energyMonitorDisableSampling.value || false,
     energyMonitorPreSample := preSampleTask.value,
     energyMonitorPostSample := postSampleTask.value,
     energyMonitorPostSampleGitHub := postSampleGitHubTask.value,
-    energyMonitorOutputFile := energyMonitorOutputFile.value
+    energyMonitorPostSampleHttp := postSampleHttpTask.value,
+    energyMonitorOutputFile := energyMonitorOutputFile.value,
+    energyMonitorPersistenceServerUrl := energyMonitorPersistenceServerUrl.value
   )
 
   override lazy val buildSettings = Seq()
 
   override lazy val globalSettings = Seq(
     energyMonitorOutputFile := "target/energy-sample",
-    energyMonitorDisableSampling := false
+    energyMonitorDisableSampling := false,
+    energyMonitorPersistenceServerUrl := "http://localhost:8080",
+    energyMonitorPersistenceTag := None
   )
 }
diff --git a/energy-monitor-plugin/src/sbt-test/sbt-energymonitor/http-store/build.sbt b/energy-monitor-plugin/src/sbt-test/sbt-energymonitor/http-store/build.sbt
@@ -0,0 +1,7 @@
+version := "0.1"
+scalaVersion := "2.13.8"
+
+lazy val httpTest = (project in file("."))
+  .settings(
+    libraryDependencies += "org.scalatest" %% "scalatest" % "3.2.12" % Test
+  )
diff --git a/energy-monitor-plugin/src/sbt-test/sbt-energymonitor/http-store/project/build.properties b/energy-monitor-plugin/src/sbt-test/sbt-energymonitor/http-store/project/build.properties
@@ -0,0 +1 @@
+sbt.version=1.6.2
diff --git a/energy-monitor-plugin/src/sbt-test/sbt-energymonitor/http-store/project/lib/jRAPL-1.0.jar b/energy-monitor-plugin/src/sbt-test/sbt-energymonitor/http-store/project/lib/jRAPL-1.0.jar
@@ -0,0 +1 @@
+../../../../../../lib/jRAPL-1.0.jar
diff --git a/energy-monitor-plugin/src/sbt-test/sbt-energymonitor/http-store/project/plugins.sbt b/energy-monitor-plugin/src/sbt-test/sbt-energymonitor/http-store/project/plugins.sbt
@@ -0,0 +1,9 @@
+{
+  val pluginVersion = System.getProperty("plugin.version")
+  if (pluginVersion == null)
+    throw new RuntimeException(
+      """|The system property 'plugin.version' is not defined.
+                                  |Specify this property using the scriptedLaunchOpts -D.""".stripMargin
+    )
+  else addSbtPlugin("com.47deg" % """energy-monitor-plugin""" % pluginVersion)
+}
diff --git a/energy-monitor-plugin/src/sbt-test/sbt-energymonitor/http-store/src/main/scala/Main.scala b/energy-monitor-plugin/src/sbt-test/sbt-energymonitor/http-store/src/main/scala/Main.scala
@@ -0,0 +1,18 @@
+package simple
+
+/** A simple class and objects to write tests against.
+  */
+class Main {
+  val default = "the function returned"
+  def method = default + " " + Main.function
+}
+
+object Main {
+
+  val constant = 1
+  def function = 2 * constant
+
+  def main(args: Array[String]): Unit = {
+    println(new Main().default)
+  }
+}