Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(QTDI-653): Add a sample connector #986

Merged
merged 35 commits into from
Feb 3, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
880cd53
feat(QTDI-653): add annotation LastGroup for AfterGroup
yyin-talend Dec 3, 2024
6c1ceba
feat(QTDI-653): fix/changes proposal
Dec 5, 2024
d23e313
feat(TCOMP-2828): Add a sample connector to test @AfterGroup only once.
ypiel-talend Oct 23, 2024
61bbc80
feat(TCOMP-2828): Fix missing resources.
ypiel-talend Oct 24, 2024
8f2dd19
feat(TCOMP-2828): Fix API Test
acatoire Oct 24, 2024
6c4795c
feat(QTDI-653): add junit to test
yyin-talend Dec 10, 2024
2ef3df5
feat(QTDI-653): fix junit
yyin-talend Dec 13, 2024
bc8c79b
optimize some code
yyin-talend Dec 18, 2024
25c940a
Add validation rule for aftergroup method
yyin-talend Dec 19, 2024
82fb952
Revert "feat(QTDI-653): add junit to test"
yyin-talend Dec 19, 2024
66a449b
Merge branch 'master' into yyin/QTDI-653_AddLastToAfterGroup
yyin-talend Dec 19, 2024
3a48e70
fix merged conflicts.
yyin-talend Dec 19, 2024
27f6f15
fix conflict
yyin-talend Dec 19, 2024
aeac195
Update ModelVisitor.java
yyin-talend Dec 19, 2024
88d505a
remove empty line
yyin-talend Dec 19, 2024
ce55d0b
fix sonars
yyin-talend Dec 19, 2024
bb49901
add test coverage
yyin-talend Dec 20, 2024
7082d8d
Merge branch 'master' into yyin/QTDI-653_AddLastToAfterGroup
yyin-talend Dec 23, 2024
08ecec0
Add test case in ComponentBValidator
yyin-talend Dec 23, 2024
401c7c2
Add junit for AutoChunkProcessor
yyin-talend Dec 24, 2024
7922ac9
Add junit for auto chunk processor
yyin-talend Dec 25, 2024
2c2d10d
Add info for failed junit
yyin-talend Dec 27, 2024
9f67f80
fix filter action in junit
yyin-talend Dec 27, 2024
5bf41df
fix filter in junit
yyin-talend Dec 27, 2024
bc3d4d4
Merge branch 'master' into yyin/QTDI-653_AddLastToAfterGroup
yyin-talend Dec 30, 2024
42245e8
fix 3 small Sonars
yyin-talend Dec 30, 2024
595a3a2
Add junit case
yyin-talend Jan 3, 2025
9aac26b
Merge branch 'master' into yyin/QTDI-653_AddLastToAfterGroup
yyin-talend Jan 6, 2025
1ea41fd
Merge branch 'master' into yyin/QTDI-653_AddLastToAfterGroup
ypiel-talend Jan 15, 2025
2d4014f
feat(QTDI-653): Add a sample connector for @LastGRoup.
ypiel-talend Jan 15, 2025
dd05004
feat(QTDI-653): Fix artifact's name.
ypiel-talend Jan 15, 2025
444d260
feat(QTDI-653): Fix processor.
ypiel-talend Jan 16, 2025
979457e
Merge branch 'master' into ypiel/QTDI-653_AddASampleConnector
ypiel-talend Jan 16, 2025
049200b
feat(QTDI-653): Add a readme on @LastGroup sample.
ypiel-talend Jan 17, 2025
2903748
feat(QTDI-653): Immprove a readme of @LastGroup sample.
ypiel-talend Jan 17, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions sample-parent/sample-features/aftergroup-lastgroup/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
This is a TCK connector to validate thethe new `@LastGroup` annotation that is used in conjonctino with `@AfterGroup` within a processor.

# How to build & deploy in a studio
From the root of `component-runtime` workspace, run:
```shell
mvn clean sample -pl sample-parent/sample-features/aftergroup-lastgroup/ -am
```
Then deploy it into a studio:
```shell
java -jar sample-parent/sample-features/aftergroup-lastgroup/target/lastGroup-<version>.car studio-deploy --location <path to a studio> -f
```

# What is `@AfterGroup` / `@LastGroup` parameter for ?
Now, methods with annotated with `@AfterGroup` in a `@Processor` connector, can have a boolean parameter annotated `@LastGroup`. If this parameter is defined in the `@AfterGroup` method, it will be set to `true` for the last group, and only for it.

# LastGroupProcessor Sample explanation
This sample is a simple `@Processor` that uses the `@AfterGroup` annotation with a `@LastGroup` parameter, [as you can see here](./src/main/java/org/talend/sdk/component/feature/lastgroup/processor/LastGroupProcessor.java#L41).

This processor just buffers records, and, in the last call of `@AfterGroup` [method](./src/main/java/org/talend/sdk/component/feature/lastgroup/processor/LastGroupProcessor.java#L62), it will send all the records to the output. It also checks that the number of bufferized records is the same as the expected number set in its configuration [Config.java](./src/main/java/org/talend/sdk/component/feature/lastgroup/config/Config.java#L36).
30 changes: 30 additions & 0 deletions sample-parent/sample-features/aftergroup-lastgroup/pom.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
Copyright (C) 2006-2025 Talend Inc. - www.talend.com
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<parent>
<groupId>org.talend.sdk.component</groupId>
<artifactId>sample-features</artifactId>
<version>1.68.0-SNAPSHOT</version>
</parent>

<groupId>org.talend.sdk.component.sample.feature</groupId>
<artifactId>lastGroup</artifactId>
<packaging>jar</packaging>

<name>Component Runtime :: Sample Feature @AfterGroup / @LastGroup</name>
</project>
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
/**
* Copyright (C) 2006-2025 Talend Inc. - www.talend.com
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.talend.sdk.component.feature.lastgroup.config;

import java.io.Serializable;

import org.talend.sdk.component.api.configuration.Option;
import org.talend.sdk.component.api.configuration.constraint.Min;
import org.talend.sdk.component.api.configuration.constraint.Required;
import org.talend.sdk.component.api.configuration.ui.layout.GridLayout;
import org.talend.sdk.component.api.meta.Documentation;

import lombok.Data;

@Data
@GridLayout(value = { @GridLayout.Row({ "expectedNumberOfInputRecords" }) })
public class Config implements Serializable {

@Option
@Min(0)
@Required
@Documentation("The expected number of records.")
private int expectedNumberOfInputRecords;

}
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
/**
* Copyright (C) 2006-2025 Talend Inc. - www.talend.com
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
@Components(
family = "sampleLastGroup",
categories = "sample")
@Icon(value = Icon.IconType.CUSTOM, custom = "icon")
package org.talend.sdk.component.feature.lastgroup;

import org.talend.sdk.component.api.component.Components;
import org.talend.sdk.component.api.component.Icon;
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
/**
* Copyright (C) 2006-2025 Talend Inc. - www.talend.com
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.talend.sdk.component.feature.lastgroup.processor;

import java.io.Serializable;
import java.util.ArrayList;
import java.util.List;

import javax.annotation.PostConstruct;

import org.talend.sdk.component.api.component.Icon;
import org.talend.sdk.component.api.component.Version;
import org.talend.sdk.component.api.meta.Documentation;
import org.talend.sdk.component.api.processor.AfterGroup;
import org.talend.sdk.component.api.processor.ElementListener;
import org.talend.sdk.component.api.processor.Input;
import org.talend.sdk.component.api.processor.LastGroup;
import org.talend.sdk.component.api.processor.Output;
import org.talend.sdk.component.api.processor.OutputEmitter;
import org.talend.sdk.component.api.processor.Processor;
import org.talend.sdk.component.api.record.Record;
import org.talend.sdk.component.feature.lastgroup.config.Config;

@Version
@Processor(name = "Processor")
@Icon(value = Icon.IconType.CUSTOM, custom = "icon")
@Documentation("AfterGroup / LastGroup sample processor connector.")
public class LastGroupProcessor implements Serializable {

private final Config config;

private List<Record> buffer;

Check failure on line 45 in sample-parent/sample-features/aftergroup-lastgroup/src/main/java/org/talend/sdk/component/feature/lastgroup/processor/LastGroupProcessor.java

View check run for this annotation

sonar-eks / Component Runtime Sonarqube Results

sample-parent/sample-features/aftergroup-lastgroup/src/main/java/org/talend/sdk/component/feature/lastgroup/processor/LastGroupProcessor.java#L45

Make "buffer" transient or serializable.

public LastGroupProcessor(final Config config) {
this.config = config;
}

@PostConstruct
public void init() {
buffer = new ArrayList<>();
}

@ElementListener
public void process(@Input final Record input) {
buffer.add(input);
}

@AfterGroup
public void afterGroup(@Output final OutputEmitter<Record> main, @LastGroup final boolean lastGroup) {
if (lastGroup) {
if (buffer.size() != config.getExpectedNumberOfInputRecords()) {
throw new IllegalStateException("Expected " +
config.getExpectedNumberOfInputRecords() + " records but got " + buffer.size());
}
buffer.stream().forEach(main::emit);
buffer.clear();
}
}

}
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Copyright (C) 2006-2025 Talend Inc. - www.talend.com
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Here you can change all your configuration display names to use more explicit labels
# You can also translate your configuration by adding one file by local Messages_fr.properties for french for example

Config.expectedNumberOfInputRecords._displayName = <expectedNumberOfInputRecords>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we set a real display name here?

Config.expectedNumberOfInputRecords._placeholder =
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Copyright (C) 2006-2025 Talend Inc. - www.talend.com
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Here you can change all your configuration display names to use more explicit labels
# You can also translate your configuration by adding one file by local Messages_fr.properties for french for example

sampleLastGroup.Processor._displayName = Processor
76 changes: 76 additions & 0 deletions sample-parent/sample-features/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@

<modules>
<module>fixed-schema</module>
<module>aftergroup-lastgroup</module>
</modules>

<dependencies>
Expand All @@ -39,4 +40,79 @@
<scope>provided</scope>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
<groupId>org.talend.sdk.component</groupId>
<artifactId>talend-component-maven-plugin</artifactId>
<version>${project.version}</version>
<!-- Please look at the TCK documentation to understand the talend-component-maven-plugin -->
<!-- https://talend.github.io/component-runtime/main/latest/build-tools-maven.html -->
<executions>
<execution>
<id>talend-dependencies</id>
<goals>
<goal>dependencies</goal>
</goals>
<phase>process-classes</phase>
</execution>
<execution>
<id>talend-component-bundle</id>
<goals>
<goal>car</goal>
</goals>
<phase>package</phase>
<configuration>
<excludeArtifactsFilter>com.fasterxml.jackson.core:jackson-databind,com.fasterxml.jackson.core:jackson-core,com.fasterxml.jackson.core:jackson-annotations</excludeArtifactsFilter>
</configuration>
</execution>
<execution>
<id>talend-scan-descriptor</id>
<goals>
<goal>scan-descriptor</goal>
</goals>
</execution>
<execution>
<id>talend-svg2png</id>
<goals>
<goal>svg2png</goal>
</goals>
</execution>
<execution>
<id>talend-component-validate</id>
<goals>
<goal>validate</goal>
</goals>
<phase>process-classes</phase>
<configuration>
<validatePlaceholder>true</validatePlaceholder>
<validateSvg>true</validateSvg>
<validateLegacyIcons>false</validateLegacyIcons>
<validateFamily>true</validateFamily>
<validateSerializable>true</validateSerializable>
<validateInternationalization>true</validateInternationalization>
<validateModel>true</validateModel>
<validateMetadata>true</validateMetadata>
<validateComponent>true</validateComponent>
<validateDataStore>true</validateDataStore>
<validateDataSet>true</validateDataSet>
<validateActions>true</validateActions>
<validateDocumentation>true</validateDocumentation>
<validateWording>true</validateWording>
<validateLayout>true</validateLayout>
<validateOptionNames>true</validateOptionNames>
<validateOutputConnection>true</validateOutputConnection>
<validateLocalConfiguration>true</validateLocalConfiguration>
<validateNoFinalOption>true</validateNoFinalOption>
<validateExceptions>true</validateExceptions>
<failOnValidateExceptions>true</failOnValidateExceptions>
<validateRecord>true</validateRecord>
<validateSchema>true</validateSchema>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
Loading