Skip to content

Commit

Permalink
Update README and provide DirectRunner for testing (#292)
Browse files Browse the repository at this point in the history
RustedBones authored Jun 20, 2024
1 parent 6fc33e2 commit 483709b
Showing 5 changed files with 24 additions and 17 deletions.
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
@@ -10,11 +10,11 @@ A [Giter8][g8] template for [Scio][scio] that includes a simple [WordCount][Word

## Running

1. Download and install the [Java Development Kit (JDK)](https://adoptopenjdk.net/index.html) version 8 or 11.
1. Download and install the [Java Development Kit (JDK)](https://adoptopenjdk.net/index.html) version 8 or higher
2. [Install sbt](http://www.scala-sbt.org/1.x/docs/Setup.html)
3. `sbt new spotify/scio.g8`
4. `sbt stage`
5. `target/universal/stage/bin/word-count --output=wc`
4. `cd <your-project-name>`
5. `sbt Test/runMain example.WordCount --runner=DirectRunner --output=wc`

⚠️ Check your project `README.md` for further details ⚠️

30 changes: 17 additions & 13 deletions src/main/g8/README.md
Original file line number Diff line number Diff line change
@@ -10,20 +10,12 @@ This project comes with number of preconfigured features, including:

### Running

* `sbt stage`
* `target/universal/stage/bin/word-count --output=wc`
The templates ships with beam `DirectRunner` in the `Test` scope. You can run locally with
`sbt Test/runMain example.WordCount --runner=DirectRunner --output=wc`

### Packaging

This template comes with [sbt-native-packager](https://sbt-native-packager.readthedocs.io) and it allows you to build **zips**, **docker** images, etc. Have a look at the documentation for more details.

```bash
sbt
# create a zip file
> universal:packageBin
# publish a docker image to your local registry
> docker:publishLocal
```
Run with the selected runner
`sbt runMain example.WordCount --runner=<runner> --output=<path>`
you may have to provide options depending on the runner

### Testing

@@ -45,6 +37,18 @@ sbt repl/run
$if(DataflowFlexTemplate.truthy)$
### Dataflow `Flex Template` usage

#### Packaging

This template comes with [sbt-native-packager](https://sbt-native-packager.readthedocs.io) and it allows you to build **zips**, **docker** images, etc. Have a look at the documentation for more details.

```bash
sbt
# create a zip file
> Universal/packageBin
# publish a docker image to your local registry
> Docker/publishLocal
```

#### Google Cloud Platform settings

```sbt
2 changes: 2 additions & 0 deletions src/main/g8/build.sbt
Original file line number Diff line number Diff line change
@@ -54,6 +54,7 @@ lazy val root: Project = project
description := "$name$",
publish / skip := true,
fork := true,
run / outputStrategy := Some(OutputStrategy.StdoutOutput),
libraryDependencies ++= Seq(
"com.spotify" %% "scio-core" % scioVersion,
"org.slf4j" % "slf4j-api" % slf4jVersion,
@@ -70,6 +71,7 @@ lazy val root: Project = project
"org.apache.spark" %% "spark-core" % sparkVersion % Runtime,
"org.apache.spark" %% "spark-streaming" % sparkVersion % Runtime,
$endif$
"org.apache.beam" % "beam-runners-direct-java" % beamVersion % Test,
"com.spotify" %% "scio-test" % scioVersion % Test,
"org.slf4j" % "slf4j-simple" % slf4jVersion % Test
),
1 change: 1 addition & 0 deletions src/main/g8/project/plugins.sbt
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
addDependencyTreePlugin
$if(DataflowFlexTemplate.truthy) $
addSbtPlugin("com.github.sbt" % "sbt-native-packager" % "1.9.16")
$endif$
Original file line number Diff line number Diff line change
@@ -30,7 +30,7 @@ object WordCount {
$if(DataflowFlexTemplate.truthy)$
sc.run()
$else$
val result = sc.run().waitUntilFinish()
val result = sc.run().waitUntilDone()
$endif$
}
}

0 comments on commit 483709b

Please sign in to comment.