Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[INLONG-11065][Sort] Provides a method to add openTelemetryAppender for the sort connector #11066

Merged
merged 3 commits into from
Sep 24, 2024

Conversation

qy-liuhuo
Copy link
Contributor

In order to more conveniently view the log output, using OpenTelemetry to achieve centralized log reporting

Motivation

In order to more conveniently view the sort task log, we need to achieve a centralized log reporting method.

Modifications

  • Added some Maven dependencies related to openTelemetry at inlong-sort/sort-flink/base/pom.xml and pom.xml
  • Designed and implemented an OpenTelemetryLogger class to add log reporting functionality for sort-connectors

Verifying this change

(Please pick either of the following options)

  • This change is a trivial rework/code cleanup without any test coverage.

  • This change is already covered by existing tests, such as:
    (please describe tests)

  • This change added tests and can be verified as follows:

    (example:)

    • Added integration tests for end-to-end deployment with large payloads (10MB)
    • Extended integration test for recovery after broker failure

Documentation

The way to integrate OpenTelemetryAppender for connectors that comply with the FLIP-27 standard is as follows:

  1. Create a new OpenTelemetryLogger object in the corresponding SourceReader construction method and configure relevant parameters(log-collector endpoint,logLevel,layout and so on)
  2. Call the install method of the openTelemetryLogger object in the start method of SourceReader
  3. Call the uninstall method of the openTelemetryLogger object in the close method of SourceReader

@qy-liuhuo
Copy link
Contributor Author

Just use it like this:

import org.apache.inlong.sort.base.util.OpenTelemetryLogger;

public class XXXSourceReader<T>
{

    private static final Logger LOG = LoggerFactory.getLogger(MySqlSourceReader.class);

    private final OpenTelemetryLogger openTelemetryLogger;

    public XXXSourceReader() {
        ...
        this.openTelemetryLogger = new OpenTelemetryLogger(); // initialize OpenTelemetryLogger
    }

    @Override
    public void start() {
        openTelemetryLogger.install(); // install OpenTelemetryLogger
        ...
    }

    @Override
    public void close() throws Exception {
        openTelemetryLogger.uninstall(); // uninstall OpenTelemetryLogger
        super.close();
    }
    
    ...
}

And you can configure the opentelemetryAppender by providing some parameters:

OpenTelemetryLogger(String serviceName, String endpoint, Layout<?> layout, Level logLevel)

@qy-liuhuo
Copy link
Contributor Author

A Demo at #11067

@vernedeng vernedeng merged commit c2e599a into apache:master Sep 24, 2024
11 checks passed
wohainilaodou pushed a commit to wohainilaodou/inlong that referenced this pull request Oct 8, 2024
…apache#11066)

* [INLONG-11065][Sort] Provides a method to add openTelemetryAppender for the sort connector

* [INLONG-11065][Sort] Optimize the code format of createOpenTelemetrySdk function
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature][Sort] Provides a method to add openTelemetryAppender for the sort connector
3 participants