Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(metrics): deduplicate dimensions when serialising #1780

Merged
merged 7 commits into from
Nov 2, 2023
Merged

fix(metrics): deduplicate dimensions when serialising #1780

merged 7 commits into from
Nov 2, 2023

Conversation

am29d
Copy link
Contributor

@am29d am29d commented Nov 2, 2023

Description of your changes

This PR fixes the issue with duplicate dimensions when we serialise metrics. The quick fix is to filter unique keys for dimensions when serialising the metrics.

I have noticed that the implementation is a bit different compared to python version. In python we use dimension_set which has unique values from default dimension and dimensions. So every change of defaultDimension needs to be reflected in the dimension field. There is no clear easier approach, in both cases you need to clean up and merge values at some point, either in setDefaultDimension clearDefaultDimension or in serialise.

I have briefly considered to apply the same strategy, but it would require a more complex refactoring. I consider the Set operation performance wise negligible, since we are only doing it once during serialisation.

Related issues, RFCs

Issue number: closes #1773

Checklist

  • My changes meet the tenets criteria
  • I have performed a self-review of my own code
  • I have commented my code where necessary, particularly in areas that should be flagged with a TODO, or hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my change is effective and works
  • The PR title follows the conventional commit semantics

Breaking change checklist

Is it a breaking change?: NO

  • I have documented the migration process
  • I have added, implemented necessary warnings (if it can live side by side)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

@am29d am29d requested a review from a team as a code owner November 2, 2023 15:00
@boring-cyborg boring-cyborg bot added metrics This item relates to the Metrics Utility tests PRs that add or change tests labels Nov 2, 2023
@pull-request-size pull-request-size bot added the size/M PR between 30-99 LOC label Nov 2, 2023
@am29d am29d self-assigned this Nov 2, 2023
@github-actions github-actions bot added the bug Something isn't working label Nov 2, 2023
@am29d am29d requested a review from dreamorosi November 2, 2023 15:09
@sonarqubecloud SonarQubeCloud
Copy link

sonarqubecloud bot commented Nov 2, 2023

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
50.0% 50.0% Duplication

dreamorosi
dreamorosi approved these changes Nov 2, 2023
View reviewed changes
Copy link
Contributor

@dreamorosi dreamorosi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Integration tests are green, let's ship it!

@dreamorosi dreamorosi merged commit 8181b48 into main Nov 2, 2023
46 checks passed
@dreamorosi dreamorosi deleted the bug/1773-duplicate-metric-dimension branch November 2, 2023 16:42
dreamorosi pushed a commit that referenced this pull request Nov 20, 2023
@am29d @dreamorosi
fix(metrics): deduplicate dimensions when serialising (#1780)
9b9fc92 
* fix: deduplicate dimensions when serializing

* fix tests

* remove tsbuildinfo

* remove whitespace

* fix gitignore again

* play some sonar games

* fix test
dreamorosi pushed a commit that referenced this pull request Jan 27, 2024
@am29d @dreamorosi
fix(metrics): deduplicate dimensions when serialising (#1780)
4168a44 
* fix: deduplicate dimensions when serializing

* fix tests

* remove tsbuildinfo

* remove whitespace

* fix gitignore again

* play some sonar games

* fix test
dreamorosi pushed a commit that referenced this pull request Feb 5, 2024
@am29d @dreamorosi
fix(metrics): deduplicate dimensions when serialising (#1780)
8595dc8 
* fix: deduplicate dimensions when serializing

* fix tests

* remove tsbuildinfo

* remove whitespace

* fix gitignore again

* play some sonar games

* fix test
dreamorosi pushed a commit that referenced this pull request Feb 5, 2024
@am29d @dreamorosi
fix(metrics): deduplicate dimensions when serialising (#1780)
70e22e5 
* fix: deduplicate dimensions when serializing

* fix tests

* remove tsbuildinfo

* remove whitespace

* fix gitignore again

* play some sonar games

* fix test
dreamorosi pushed a commit that referenced this pull request Feb 20, 2024
@am29d @dreamorosi
fix(metrics): deduplicate dimensions when serialising (#1780)
748d3c9 
* fix: deduplicate dimensions when serializing

* fix tests

* remove tsbuildinfo

* remove whitespace

* fix gitignore again

* play some sonar games

* fix test
dreamorosi added a commit that referenced this pull request Feb 28, 2024
@dreamorosi @erikayao93
@antstanley @am29d @shdq @github-actions @dependabot
feat!: add ESM support & logger extensibility (#2117)
3b99263 
* feat(logger): Support for external observability providers (#1511)

* Updated formatAttributes for additional parameters and LogItem return type

* Updated the unit tests to pass with new formatter

* Updated Powertool named objects to Powertools

* Updated tests to match new naming consistency

* Updated for tests for new naming consistency

* Updated formatter for new design decisions

* Update Logger for ephemeral attributes

* Update bringYourOwnFormatter documentation to match new formatter

---------

Co-authored-by: erikayao93 <[email protected]>

* chore(logger): PowertoolsLogFormatter docstring and variable naming update (#1585)

* Updated formatAttributes for additional parameters and LogItem return type

* Updated the unit tests to pass with new formatter

* Updated Powertool named objects to Powertools

* Updated tests to match new naming consistency

* Updated for tests for new naming consistency

* Updated formatter for new design decisions

* Update Logger for ephemeral attributes

* Update bringYourOwnFormatter documentation to match new formatter

* Fixed incorrect return type, renamed variable for consistency

* feat(logger): Support for external observability providers (#1511)

* Updated formatAttributes for additional parameters and LogItem return type

* Updated the unit tests to pass with new formatter

* Updated Powertool named objects to Powertools

* Updated tests to match new naming consistency

* Updated for tests for new naming consistency

* Updated formatter for new design decisions

* Update Logger for ephemeral attributes

* Update bringYourOwnFormatter documentation to match new formatter

---------

Co-authored-by: erikayao93 <[email protected]>

* chore(logger): PowertoolsLogFormatter docstring and variable naming update (#1585)

* Updated formatAttributes for additional parameters and LogItem return type

* Updated the unit tests to pass with new formatter

* Updated Powertool named objects to Powertools

* Updated tests to match new naming consistency

* Updated for tests for new naming consistency

* Updated formatter for new design decisions

* Update Logger for ephemeral attributes

* Update bringYourOwnFormatter documentation to match new formatter

* Fixed incorrect return type, renamed variable for consistency

* chore(maintenance): bump dependencies & drop nodejs14x (#1687)

* chore: update release script to mark all utilities as alpha

* chore: restore version to ease conflicts

* chore: release version change

* chore: release version change

* chore(maintenance): remove `createLogger` and `createTracer` helpers (#1722)

* chore(maintenance): bump dependencies & drop nodejs14x (#1687)

* chore: add pre-release script

* chore: restore deps

* chore: added v2 shim

* chore(maintenance): remove logger and tracer helper function

* chore: remove imports

* chore: fix deps & versions

* tests: moved unit tests

* tests: move logger tests

* chore: added v2 shim

* chore: added v2 shim

* feat(logger): add esmodule support (#1734)

* feat(logger): add esm build output

* fix(Logger): Remove barrel files update references

* test(Logger): update jest/ts-jest to use ESM

* chore(Logger): remove unused lodash.merge

* fix(logger): reinstate lodash.merge

* chore(logger): revert TS assertion

* chore(logger): revert format changes

* chore(logger): update postbuild to remove incremental tsbuildinfo files

* fix(logger): correct reference to types output

* feat(logging): add middleware export

* chore(logger): replace postbuild script with echo statement

* feat(logger): add typesVersions property and barrel files to /types

* chore(logger): file not used, can be added back if needed

* chore(logger): add space back to README

* chore(logger): revert space in README

* feat(commons): add esmodule support (#1735)

* chore(logger): adapt logger to commons exports

* feat(commons): add esmodule support

* chore: address sonar findings

* chore(commons): exported version

* chore: fixed imports in examples

* chore(parameters): fixed imports

* chore(metrics): fixed imports

* chore(tracer): fixed imports

* chore(idempotency): fixed imports

* chore(commons): test coverage

* chore(batch): fix imports

* feat(parameters): add esmodule support (#1736)

* feat(batch): add esmodule support (#1737)

* feat(internal): add esmodule support (#1738)

* feat(testing): add esmodule support

* chore(all): update imports

* feat(metrics): add esmodule support (#1739)

* feat(tracer): add esmodule support (#1741)

* feat(tracer): add esmodule support

* chore(docs): update imports

* feat(idempotency): add esmodule support  (#1743)

* feat(idempotency): add esmodule support

* chore(metrics): fix import

* chore(ci): v2 release line

* chore(ci): fix alpha versioning pre-release

* docs(maintenance): add processes tab (#1747)

* docs(maintenance): update mkdocs to support tabs

* chore(ci): add parallel test npm script

* chore(ci): add jest command

* docs(maintenance): add testing page to navbar

* docs(maintenance): add contributing info

* chore: update roadmap

* chore: update release drafter workflow to allow for manual trigger

* fix formatting

* docs: maintainers handbook

* chore: link to new location

* fix links

* Update docs/maintainers.md

Co-authored-by: Alexander Schueren <[email protected]>

---------

Co-authored-by: Alexander Schueren <[email protected]>

* chore(docs): add invisible unicode char to decorator docs (#1755)

* chore: remove extra comma

* chore(docs): upgrade doc intro

* feat(logger): align sampling debug logs feature implementation with the other runtimes (#1744)

* test(logger): remove logsSampled field, add default sampleRateValue

* test(logger): add tests for sampling debug logs feature

* feat(logger): change implementation to make sampling decision at per-function level

* refactor(logger): remove redundant createLogger method

* refactor(logger): remove getSampleRateValue method

* test(logger): improve tests

* refactor(logger): return createLogger() back with the detailed comment of the method importance

* test(logger): add constructor/custom config/env var priority tests for sampling rate feature, improve description

* refactor(logger): address review comments

* feat(logger): add refreshSampleRateCalculation method and tests

* test(logger): adjust end-to-end tests

* chore(logger): refactor types and interfaces (#1758)

* chore(logger): refactor types and interfaces

* chore: grouped type files

* chore: fix code smell

* chore: fix ci

* chore: fix ci

* chore(maintenance): bump Middy v4 & run tests (#1760)

* chore(parameters): fix esm bundling

* chore(parameters): change declare client param

* chore(layers) widen version check in e2e

* chore(maintenance): enable `isolatedModules` and isolate cache (#1765)

* chore(layers) widen version check in e2e

* chore(maintenance): enable isolatedModules

* chore: remove redundant comments from tsconfig

* chore: changed path of tsbuild cache

* fix: idempotency types

* build(maintenance): bump aws sdk dev dependencies

* chore(parameters): add export types

* chore(logger): set default UTC timezone (#1775)

* chore(parameters): add export types

* chore(logger): set default utc timezone

* chore(logger): pass down envvarsservice to log formatter

* fix(metrics): deduplicate dimensions when serialising (#1780)

* fix: deduplicate dimensions when serializing

* fix tests

* remove tsbuildinfo

* remove whitespace

* fix gitignore again

* play some sonar games

* fix test

* chore(release): v1.14.2 [skip ci]

* feat(metrics): log directly to stdout  (#1786)

* chore(commons): move isDevMode to commons

* chore(logger): move isDev config out of logger to commons

* feat(metrics): use own console object by default

* tests(layers): fix unit tests

* feat(maintenance): add support for nodejs20.x runtime (#1790)

* feat(maintenance): support nodejs20.x runtime

* tests(metrics): fix object ordering in tests

* build(testing): bump cdk

* build(maintenance): revert aws-cdk-lib to support nodejs14

* tests(maintenance): set runtime with family

* chore(docs): patch runtime in cdk

* chore(docs): patch runtime in cdk

* chore(maintenance): increment version in commons ahead of release

* chore(commons): simplify config service interface

* chore(release): v1.15.0 [skip ci]

* feat(logger): add support for `AWS_LAMBDA_LOG_LEVEL` and `POWERTOOLS_LOG_LEVEL` (#1795)

* feat(logger): support advanced logging

* docs(logger): add alc info

* feat(logger): support alc

* docs: fix alc docs links

* tests(logger): add unit tests for the feature

* docs(logger): make POWERTOOLS_LOG_LEVEL default

* chore(release): v1.16.0 [skip ci]

* chore(maintenance): drop support for Node.js 14 (#1802)

* chore: package lock

* chore(docs): update docs url in comments & readme files (#1728)

* chore(ci): update v2 release workflow (#1745)

* docs(maintenance): add processes tab (#1747)

* docs(maintenance): update mkdocs to support tabs

* chore(ci): add parallel test npm script

* chore(ci): add jest command

* docs(maintenance): add testing page to navbar

* docs(maintenance): add contributing info

* chore: update roadmap

* chore: update release drafter workflow to allow for manual trigger

* fix formatting

* docs: maintainers handbook

* chore: link to new location

* fix links

* Update docs/maintainers.md

Co-authored-by: Alexander Schueren <[email protected]>

---------

Co-authored-by: Alexander Schueren <[email protected]>

* chore(maintenance): add --require-hashes flag to pip installs (#1827)

* bump version to 9.5.2, rerun pip-compile with correct deps (#1830)

* chore(ci): Dependabot fine tuning (#1862)

* ignore major updates for mike

* set versioning strategy for cdk, dependabot is running on auto upgrade strategy per default

* ignore middy major upgrades

* remove CodeQL, it's enough to have Sonar

* update package-lock

* chore(deps-dev): bump @aws-sdk/client-cloudwatch from 3.438.0 to 3.485.0 (#1857)

* chore(deps-dev): bump @types/node from 20.11.0 to 20.11.2 (#1912)

* chore(deps): fix dependencies and dependabot config (#1917)

* chore(deps-dev): bump @typescript-eslint/parser from 6.19.0 to 6.19.1 (#1946)

Bumps [@typescript-eslint/parser](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/parser) from 6.19.0 to 6.19.1.
- [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases)
- [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/parser/CHANGELOG.md)
- [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v6.19.1/packages/parser)

---
updated-dependencies:
- dependency-name: "@typescript-eslint/parser"
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps-dev): bump @typescript-eslint/eslint-plugin (#1948)

Bumps [@typescript-eslint/eslint-plugin](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/eslint-plugin) from 6.19.0 to 6.19.1.
- [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases)
- [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/eslint-plugin/CHANGELOG.md)
- [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v6.19.1/packages/eslint-plugin)

---
updated-dependencies:
- dependency-name: "@typescript-eslint/eslint-plugin"
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(ci): refactor workflows to scope permissions (#1978)

* chore(maintenance): rebase conflicts

* chore(maintenance): rebase conflicts

* chore: rebase conflicts

* chore: remove v2 specific release

* chore: release patch script

* docs: update docs & snippets

* chore: address SonarCloud findings

* chore: address SonarCloud findings

* chore: address SonarCloud findings

* tests: coverage

* chore: add layers v2

* chore: bump ua version

* docs: add install steps to batch

* chore: fix install command

* chore: remove unused dependency

* Update examples/cdk/package.json

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Erika Yao <[email protected]>
Co-authored-by: erikayao93 <[email protected]>
Co-authored-by: Ant Stanley <[email protected]>
Co-authored-by: Alexander Schueren <[email protected]>
Co-authored-by: Sergei Cherniaev <[email protected]>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
dreamorosi pushed a commit that referenced this pull request Mar 6, 2024
@am29d @dreamorosi
fix(metrics): deduplicate dimensions when serialising (#1780)
0910e03 
* fix: deduplicate dimensions when serializing

* fix tests

* remove tsbuildinfo

* remove whitespace

* fix gitignore again

* play some sonar games

* fix test
dreamorosi pushed a commit that referenced this pull request Mar 6, 2024
@am29d @dreamorosi
fix(metrics): deduplicate dimensions when serialising (#1780)
2931c5a 
* fix: deduplicate dimensions when serializing

* fix tests

* remove tsbuildinfo

* remove whitespace

* fix gitignore again

* play some sonar games

* fix test
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dreamorosi dreamorosi dreamorosi approved these changes

Assignees

@am29d am29d

Labels
bug Something isn't working metrics This item relates to the Metrics Utility size/M PR between 30-99 LOC tests PRs that add or change tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Bug: Override of default dimension with new value results into wrong CloudWatch dimension output
2 participants
@am29d @dreamorosi