Skip to content

[FLINK-32695] [test-utils] Add flink-test-utils-source module for Source V2 tests. #26910

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

Poorvankbhatia
Copy link
Contributor

@Poorvankbhatia Poorvankbhatia commented Aug 15, 2025

What is the purpose of the change

Added a minimal, test-only utility module for FLIP-27 (Source V2) tests to reduce boilerplate when using Source V2 in Test cases. This is being done as a part of FLINK-32695 to migrate all the legacy test sources.

While migrating legacy SourceFunction–based tests to FLIP-27 (Source V2), a recurring pattern was noticed: many tests only need a mostly no-op scaffold (e.g., a source/enumerator that does nothing) or just one component—most often the SourceReader—to exercise a specific behavior. Hence this change is being made so that those patterns can be captured and test authors can plug in a tiny override (reader/enumerator/serializer) without re-implementing boilerplate.

Module path: flink-test-utils-parent/flink-test-utils-source

Brief change log

  • Add new module in flink-test-utils-parent (packaging: jar).
  • Introduced a reusable scaffolding under org.apache.flink.test.util.source:
    • AbstractTestSource — default BOUNDED Source V2 with sensible no-op implementations; easy to extend.
    • TestSourceReader — minimal SourceReader (default END_OF_INPUT), plumbs SourceReaderContext.
    • TestSplit — lightweight split with singleton-style serializer for simple tests.
    • TestSplitEnumerator — assigns a single split on request and signals no-more-splits to avoid hanging tests.
    • VoidSerializer — serializer for Void enumerator checkpoints.
  • Keep dependencies lean:
    • flink-core
  • No production modules changed.

Verifying this change

  • Module builds with the existing Flink parent
  • Unit tests added in SourcePatternExamplesTest exercise the new test-utils and cover complex Source V2 behaviors

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? no
  • If yes, how is the feature documented? not applicable, Javadoc added to new classes to describe intended usage and extension points.

@Poorvankbhatia Poorvankbhatia force-pushed the FLINK-32695-Common-Source-Test-Utils branch from 9da2a73 to 64236f3 Compare August 15, 2025 18:56
@flinkbot
Copy link
Collaborator

flinkbot commented Aug 15, 2025

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@Poorvankbhatia
Copy link
Contributor Author

@fapaul PTAL :)

@Poorvankbhatia Poorvankbhatia force-pushed the FLINK-32695-Common-Source-Test-Utils branch from 64236f3 to b076a81 Compare August 16, 2025 18:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants