Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(common.shim): Add batch to shim #16148

Open
wants to merge 15 commits into
base: master
Choose a base branch
from

Conversation

revilon1991
Copy link

Summary

I want to use batch with shim within the method Write.

Checklist

  • No AI generated code was used in this PR

Related issues

resolves #11902
resolves #10277

@telegraf-tiger
Copy link
Contributor

telegraf-tiger bot commented Nov 5, 2024

Thanks so much for the pull request!
🤝 ✒️ Just a reminder that the CLA has not yet been signed, and we'll need it before merging. Please sign the CLA when you get a chance, then post a comment here saying !signed-cla

@revilon1991
Copy link
Author

!signed-cla

@DStrand1
Copy link
Member

@revilon1991 Thanks for the PR! I see you commented for the CLA checker, but I don't see your username in the list. Could you double check that you submitted that? Thanks!

@DStrand1 DStrand1 self-assigned this Nov 13, 2024
@revilon1991
Copy link
Author

@DStrand1 which list? I don't know what I should do else

@revilon1991 revilon1991 changed the title add batch shim for output.execd feat(common.shim): Add batch to shim Nov 29, 2024
@telegraf-tiger telegraf-tiger bot added the feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin label Nov 29, 2024
@revilon1991
Copy link
Author

!signed-cla


"github.com/influxdata/telegraf"
"github.com/influxdata/telegraf/models"
"github.com/influxdata/telegraf/plugins/parsers/influx"
)

type batchMetrics struct {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should better be in its own file together with specific tests for each of the methods (add, clear, len)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done 962ae8e

metric := testutil.MustMetric("test_metric", nil, map[string]interface{}{"field1": 1}, time.Unix(0, 0))
bm.add(metric)

require.Equal(t, 1, bm.len())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe this can be used instead?

Suggested change
require.Equal(t, 1, bm.len())
require.Len(t, bm, 1)


bm.add(metric1)
bm.add(metric2)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
require.Len(t, bm.metrics, 2)


bm.add(metric)

testutil.RequireMetricsEqual(t, []telegraf.Metric{metric}, bm.metrics, testutil.IgnoreTime())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why ignoring the timestamp? It should stay the same?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought that would help me fix a problem with the test run.

=== RUN   TestCases/rfc3164_best_effort_udp
2025-01-07T15:20:38Z I! Loading config: testcases/rfc3164_best_effort_udp/telegraf.conf
2025-01-07T15:20:38Z I! [inputs.syslog] Listening on udp://127.0.0.1:58508
    syslog_test.go:274: []telegraf.Metric
        --- expected
        +++ actual
          []*testutil.metricDiff{
          	&{
          		Measurement: "syslog",
          		Tags:        {&{Key: "appname", Value: "app"}, &{Key: "facility", Value: "user"}, &{Key: "hostname", Value: "host"}, &{Key: "severity", Value: "notice"}, ...},
          		Fields: []*telegraf.Field{
          			&{Key: "facility_code", Value: int64(1)},
          			&{Key: "message", Value: string("Test")},
          			&{Key: "severity_code", Value: int64(5)},
          			&{
          				Key:   "timestamp",
        - 				Value: int64(1733157063000000000),
        + 				Value: int64(1764693063000000000),
          			},
          		},
          		Type: 3,
          		... // 1 ignored field
          	},
          }
--- FAIL: TestCases/rfc3164_best_effort_udp (0.11s)

I'll rollback it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is a test from a completely different plugin, which will be fixed when #16364 is merged..

mu: &mu,
}

metric := testutil.MustMetric("test_metric", map[string]string{"tag1": "value1"}, map[string]interface{}{"field1": 1}, time.Unix(0, 0))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe easier to just use testutil.TestMetric?

metric := testutil.TestMetric(101, "metric1")
bm.add(metric)

require.Len(t, bm.metrics, 1)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With this you are not testing your own Len method.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I cannot use the method require.Len and call bm.len() because Len requires a slice as an argument.
Can I use require.Equal like that?

require.Equal(t, bm.len(), 1)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@telegraf-tiger
Copy link
Contributor

Download PR build artifacts for linux_amd64.tar.gz, darwin_arm64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

⚠️ This pull request increases the Telegraf binary size by 1.42 % for linux amd64 (new size: 277.5 MB, nightly size 273.6 MB)

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB RPM TAR GZ ZIP
amd64.deb aarch64.rpm darwin_amd64.tar.gz windows_amd64.zip
arm64.deb armel.rpm darwin_arm64.tar.gz windows_arm64.zip
armel.deb armv6hl.rpm freebsd_amd64.tar.gz windows_i386.zip
armhf.deb i386.rpm freebsd_armv7.tar.gz
i386.deb ppc64le.rpm freebsd_i386.tar.gz
mips.deb riscv64.rpm linux_amd64.tar.gz
mipsel.deb s390x.rpm linux_arm64.tar.gz
ppc64el.deb x86_64.rpm linux_armel.tar.gz
riscv64.deb linux_armhf.tar.gz
s390x.deb linux_i386.tar.gz
linux_mips.tar.gz
linux_mipsel.tar.gz
linux_ppc64le.tar.gz
linux_riscv64.tar.gz
linux_s390x.tar.gz

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin
Projects
None yet
Development

Successfully merging this pull request may close these issues.

External plugin does not adhere to metric_batch_size shim trigger output to write single metric
3 participants