improve testing framework #4962

RustedBones · 2023-08-22T15:55:36Z

Add runWitnOverrides to run pipeline unit-test, replacing some parts
Add JobTest io support for SMB
Add JobTest API for anonymous jobs (defined from context)

RustedBones · 2023-08-22T15:56:43Z

scio-test/src/main/scala/com/spotify/scio/testing/PipelineTestUtils.scala

@@ -36,15 +37,15 @@ trait PipelineTestUtils {
   * }
   * }}}
   */
-  def runWithContext[T](fn: ScioContext => T): ScioExecutionContext = {


This is breaking. type parameter was not really useful here

codecov · 2023-08-22T16:19:18Z

Codecov Report

Attention: 178 lines in your changes are missing coverage. Please review.

Comparison is base (1e0cb80) 63.42% compared to head (c69d279) 63.37%.

Files	Patch %	Lines
.../smb/syntax/SortMergeBucketScioContextSyntax.scala	19.42%	141 Missing ⚠️
...cala/com/spotify/scio/smb/SortMergeTransform.scala	45.71%	19 Missing ⚠️
.../main/scala/com/spotify/scio/testing/JobTest.scala	67.39%	15 Missing ⚠️
...sdk/extensions/smb/ParquetTypeSortedBucketIO.scala	0.00%	2 Missing ⚠️
.../smb/syntax/SortMergeBucketSCollectionSyntax.scala	90.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4962      +/-   ##
==========================================
- Coverage   63.42%   63.37%   -0.06%     
==========================================
  Files         288      291       +3     
  Lines       10805    10841      +36     
  Branches      778      781       +3     
==========================================
+ Hits         6853     6870      +17     
- Misses       3952     3971      +19

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

scio-test/src/main/scala/com/spotify/scio/testing/JobTest.scala

scio-smb/src/main/scala/com/spotify/scio/smb/SMBIO.scala

scio-smb/src/main/scala/com/spotify/scio/smb/syntax/SortMergeBucketScioContextSyntax.scala

scio-smb/src/test/scala/com/spotify/scio/smb/SortMergeBucketTest.scala

RustedBones · 2023-08-30T09:25:55Z

In order to make testing work with sortMergeTransform, I had to introduce some breaking API changes.
Rebased the branch on v0.14.x

scio-test/src/main/scala/com/spotify/scio/testing/RunEnforcementJobTest.scala

kellen

Nice work!

kellen · 2023-10-11T16:01:16Z

scio-test/src/main/scala/com/spotify/scio/testing/PipelineTestUtils.scala

+   * }
+   * }}}
+   */
+  def runWithOverrides(


Could this instead be chained onto ScioExecutionContext so we could avoid the distinct naming? e.g.

runWithContext { sc => // ... } .withOverrides( TransformOverride.of("operation", (v: Int) => v.toString) )

I tried but then it's not possible to decide when the pipeline should be run.

Ahh yeah that makes sense. Could we nest it somehow?

runWithContext { sc => sc.withOverrides( TransformOverride.of(...) ) }

looks nice and should be possible. Will try that.

This change is a bit risky because the transform must be defined before the withOverrides call or we need to store the override and call replaceAll just before run.

I tried to integrate the TestDataManager here but it is inconvenient

kellen

🙌

clairemcginty

amazing work!

scio-smb/src/test/scala/com/spotify/scio/smb/SortMergeBucketTest.scala

clairemcginty · 2024-01-05T17:59:58Z

scio-smb/src/main/scala/com/spotify/scio/smb/SortedBucketIO.scala

+import com.spotify.scio.io.{KeyedIO, TapOf, TapT, TestIO}
+import com.spotify.scio.util.ScioUtil
+
+final class SortedBucketIO[K, T](path: String, override val keyBy: T => K)(implicit


now that I think about it, this class name is a bit confusing as it conflicts with the beam packaged SortedBucketIO https://github.com/spotify/scio/blob/main/scio-smb/src/main/java/org/apache/beam/sdk/extensions/smb/SortedBucketIO.java. I was testing out a custom SortedBucketIO.Read override, but since I had import com.spotify.scio.smb._ in my job (in order to get the Scala API bindings for reads/writes), I had to rename the org.apache.beam.sdk.extensions.smb.SortedBucketIO import to be able to use it

Maybe SmbIO? SortedBucketTestIO?

Yes I noticed that. I tried to keep consistency with java naming but It's probably to better to avoid conflict with SmbIO

cool. I can make that change today!

RustedBones commented Aug 22, 2023

View reviewed changes

RustedBones force-pushed the testing-improvements branch 2 times, most recently from 3b481e3 to 434d558 Compare August 28, 2023 14:18

shnapz reviewed Aug 28, 2023

View reviewed changes

scio-test/src/main/scala/com/spotify/scio/testing/JobTest.scala Outdated Show resolved Hide resolved

shnapz reviewed Aug 28, 2023

View reviewed changes

scio-test/src/main/scala/com/spotify/scio/testing/JobTest.scala Outdated Show resolved Hide resolved

clairemcginty reviewed Aug 28, 2023

View reviewed changes

RustedBones force-pushed the testing-improvements branch from e051fc5 to ae86acf Compare August 30, 2023 09:22

RustedBones changed the base branch from main to v0.14.x August 30, 2023 09:23

RustedBones added this to the 0.14.0 milestone Aug 30, 2023

RustedBones mentioned this pull request Aug 30, 2023

Fix typeValidation in TransformOverride #4967

Merged

RustedBones marked this pull request as ready for review September 11, 2023 07:31

RustedBones force-pushed the testing-improvements branch from 99e71be to ad0e31d Compare October 11, 2023 15:50

RustedBones commented Oct 11, 2023

View reviewed changes

scio-test/src/main/scala/com/spotify/scio/testing/RunEnforcementJobTest.scala Outdated Show resolved Hide resolved

kellen approved these changes Nov 6, 2023

View reviewed changes

Base automatically changed from v0.14.x to main November 29, 2023 07:52

RustedBones force-pushed the testing-improvements branch from e8e36c2 to c19f88c Compare December 5, 2023 11:29

RustedBones added 12 commits December 8, 2023 16:47

Add runWithOverrides

c8b84a2

Add missing header

ec009fb

scalafix

8ca64b0

Fix typeValidation in TransformOverride

31b91c5

Skip TestData manager

f92e857

TestIO proposal for SMB

4eecb56

Add file headers

1b9e41e

Unify JobTest input with KeyedIO

01937b2

Run javafmt

61a6b50

Suppot KeyedIO TestStream input

e1c1fb5

Fix build

ffbc377

Fix 2.12 compilation issue

7ca5eaa

RustedBones added 13 commits December 8, 2023 16:48

Use consistent naming

c9128e9

Filter mima binary compatibility

b4d56ac

Add test support for SMB transform

3058a86

mock smb output

cb033ba

Compile scala 2.12

bae91d8

Make it serializable

75b7477

scalafix

a771819

Anonymous job test

50955d2

leverage jobTest in internall tests

129b381

Consistent JobTest constructor

c2faf1b

Add generated file to scalafmt exclude

a78fee0

Add documentation entry for runWithOverrides

e2117a6

Add documentation for anonymous JobTest

987f64d

RustedBones force-pushed the testing-improvements branch from c19f88c to 987f64d Compare December 8, 2023 15:49

kellen approved these changes Dec 12, 2023

View reviewed changes

clairemcginty mentioned this pull request Dec 12, 2023

Support Taps for SMB writes/transforms #5080

Closed

clairemcginty approved these changes Dec 12, 2023

View reviewed changes

scio-smb/src/test/scala/com/spotify/scio/smb/SortMergeBucketTest.scala Outdated Show resolved Hide resolved

Rename test

c69d279

RustedBones merged commit 5648e79 into main Dec 13, 2023
13 checks passed

RustedBones deleted the testing-improvements branch December 13, 2023 10:42

clairemcginty reviewed Jan 5, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

improve testing framework #4962

improve testing framework #4962

RustedBones commented Aug 22, 2023 •

edited

Loading

RustedBones Aug 22, 2023

codecov bot commented Aug 22, 2023 •

edited

Loading

RustedBones commented Aug 30, 2023 •

edited

Loading

kellen left a comment

kellen Oct 11, 2023

RustedBones Nov 6, 2023

kellen Nov 6, 2023

RustedBones Nov 7, 2023

RustedBones Dec 5, 2023

kellen left a comment

clairemcginty left a comment

clairemcginty Jan 5, 2024

RustedBones Jan 8, 2024

clairemcginty Jan 8, 2024

improve testing framework #4962

improve testing framework #4962

Conversation

RustedBones commented Aug 22, 2023 • edited Loading

Choose a reason for hiding this comment

codecov bot commented Aug 22, 2023 • edited Loading

Codecov Report

RustedBones commented Aug 30, 2023 • edited Loading

kellen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kellen left a comment

Choose a reason for hiding this comment

clairemcginty left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RustedBones commented Aug 22, 2023 •

edited

Loading

codecov bot commented Aug 22, 2023 •

edited

Loading

RustedBones commented Aug 30, 2023 •

edited

Loading