Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(template): Introduce new File and InMemory Templates #1197

Merged
merged 5 commits into from
Oct 5, 2023

Conversation

UnseenWizzard
Copy link
Contributor

What this PR does / Why we need it:

The updated FileBasedTemplate loads file contents only when required, instead of at creation, reducing the long term in-use memory footprint
of configurations.

The new InMemoryTemplate replaces the Download specific template, as well as some dedicated test implementations of templates. It currently carries an optional filepath, which is used for writing converted templates to the desired target file. This may be split off.

Special notes for your reviewer:

  • exact memory impact during deployments still to be tested
  • the general InMemoryTemplate now used for Download, Convert as well as testing, may not be the best way to go.
    It might instead make sense to have dedicated template implementations in

Does this PR introduce a user-facing change?

No visible change, template files are loaded from disk later.

@UnseenWizzard UnseenWizzard added the run-e2e-test Manually trigger the E2E tests for reviewed PRs label Sep 22, 2023
@UnseenWizzard UnseenWizzard force-pushed the feat/defer-template-load branch from 5a647eb to 547de17 Compare September 22, 2023 12:30
@github-actions
Copy link

github-actions bot commented Sep 22, 2023

Test Results

       2 files  ±  0     220 suites  ±0   16s ⏱️ ±0s
1 504 tests  -   8  1 504 ✔️  -   8  0 💤 ±0  0 ±0 
3 008 runs   - 16  3 008 ✔️  - 16  0 💤 ±0  0 ±0 

Results for commit 99b1022. ± Comparison against base commit 44d82db.

This pull request removes 10 and adds 2 tests. Note that renamed tests count towards both.
github.com/dynatrace/dynatrace-configuration-as-code/v2/pkg/config/template ‑ TestCreateTemplateFromString
github.com/dynatrace/dynatrace-configuration-as-code/v2/pkg/config/template ‑ TestCreateTemplateFromString/simple_template_created
github.com/dynatrace/dynatrace-configuration-as-code/v2/pkg/config/template ‑ TestCreateTemplateFromString/works_on_empty_inputs
github.com/dynatrace/dynatrace-configuration-as-code/v2/pkg/config/template ‑ Test_fileBasedTemplate_Content_Returns_Content
github.com/dynatrace/dynatrace-configuration-as-code/v2/pkg/config/template ‑ Test_fileBasedTemplate_Filepath_Returns_Path
github.com/dynatrace/dynatrace-configuration-as-code/v2/pkg/config/template ‑ Test_fileBasedTemplate_Id_Returns_Path
github.com/dynatrace/dynatrace-configuration-as-code/v2/pkg/config/template ‑ Test_fileBasedTemplate_Name_Returns_Path
github.com/dynatrace/dynatrace-configuration-as-code/v2/pkg/delete ‑ TestSplitConfigsForDeletion/Id-fallback
github.com/dynatrace/dynatrace-configuration-as-code/v2/pkg/download/automation ‑ Test_createTemplateFromRawJSON/defaults_template_name_to_ID_if_title_is_not_found_-_but_returns_no_name
github.com/dynatrace/dynatrace-configuration-as-code/v2/pkg/download/automation ‑ Test_createTemplateFromRawJSON/sanitizes_template_as_expected_-_extracts_title_as_name
github.com/dynatrace/dynatrace-configuration-as-code/v2/pkg/delete ‑ TestSplitConfigsForDeletion/ID-fallback
github.com/dynatrace/dynatrace-configuration-as-code/v2/pkg/download/automation ‑ Test_createTemplateFromRawJSON/sets_template_ID_to_object_ID

♻️ This comment has been updated with latest results.

@Laubi Laubi force-pushed the feat/defer-template-load branch from ac502f0 to 6127a30 Compare September 25, 2023 13:28
@UnseenWizzard UnseenWizzard force-pushed the feat/defer-template-load branch from 6127a30 to d2a838a Compare September 27, 2023 10:13
@UnseenWizzard UnseenWizzard marked this pull request as ready for review September 27, 2023 10:13
@UnseenWizzard UnseenWizzard requested a review from a team as a code owner September 27, 2023 10:13
@UnseenWizzard UnseenWizzard added run-e2e-test Manually trigger the E2E tests for reviewed PRs and removed run-e2e-test Manually trigger the E2E tests for reviewed PRs labels Sep 27, 2023
@UnseenWizzard UnseenWizzard force-pushed the feat/defer-template-load branch from d2a838a to 2f21fe1 Compare September 29, 2023 07:56
@jskelin jskelin self-requested a review September 29, 2023 11:39
Copy link
Contributor

@jskelin jskelin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

having ID() method that returns differnet value for different implementation for me is problematic

pkg/config/template/inmemory.go Outdated Show resolved Hide resolved
pkg/config/template/inmemory.go Show resolved Hide resolved
pkg/config/template/template.go Show resolved Hide resolved
pkg/config/template/filebased.go Show resolved Hide resolved
pkg/config/template/filebased.go Show resolved Hide resolved
pkg/config/config.go Show resolved Hide resolved
@UnseenWizzard UnseenWizzard force-pushed the feat/defer-template-load branch 2 times, most recently from 7342ab1 to 99b1022 Compare October 3, 2023 07:06
@UnseenWizzard UnseenWizzard removed the run-e2e-test Manually trigger the E2E tests for reviewed PRs label Oct 3, 2023
@jskelin jskelin self-requested a review October 3, 2023 10:41
@UnseenWizzard UnseenWizzard force-pushed the feat/defer-template-load branch from 99b1022 to 7b87e56 Compare October 4, 2023 06:19
@github-actions
Copy link

github-actions bot commented Oct 4, 2023

Unit Test Results

       2 files  ±  0     220 suites  ±0   17s ⏱️ ±0s
1 504 tests  -   8  1 504 ✔️  -   8  0 💤 ±0  0 ±0 
3 008 runs   - 16  3 008 ✔️  - 16  0 💤 ±0  0 ±0 

Results for commit a4511cf. ± Comparison against base commit de70797.

This pull request removes 10 and adds 2 tests. Note that renamed tests count towards both.
github.com/dynatrace/dynatrace-configuration-as-code/v2/pkg/config/template ‑ TestCreateTemplateFromString
github.com/dynatrace/dynatrace-configuration-as-code/v2/pkg/config/template ‑ TestCreateTemplateFromString/simple_template_created
github.com/dynatrace/dynatrace-configuration-as-code/v2/pkg/config/template ‑ TestCreateTemplateFromString/works_on_empty_inputs
github.com/dynatrace/dynatrace-configuration-as-code/v2/pkg/config/template ‑ Test_fileBasedTemplate_Content_Returns_Content
github.com/dynatrace/dynatrace-configuration-as-code/v2/pkg/config/template ‑ Test_fileBasedTemplate_Filepath_Returns_Path
github.com/dynatrace/dynatrace-configuration-as-code/v2/pkg/config/template ‑ Test_fileBasedTemplate_Id_Returns_Path
github.com/dynatrace/dynatrace-configuration-as-code/v2/pkg/config/template ‑ Test_fileBasedTemplate_Name_Returns_Path
github.com/dynatrace/dynatrace-configuration-as-code/v2/pkg/delete ‑ TestSplitConfigsForDeletion/Id-fallback
github.com/dynatrace/dynatrace-configuration-as-code/v2/pkg/download/automation ‑ Test_createTemplateFromRawJSON/defaults_template_name_to_ID_if_title_is_not_found_-_but_returns_no_name
github.com/dynatrace/dynatrace-configuration-as-code/v2/pkg/download/automation ‑ Test_createTemplateFromRawJSON/sanitizes_template_as_expected_-_extracts_title_as_name
github.com/dynatrace/dynatrace-configuration-as-code/v2/pkg/delete ‑ TestSplitConfigsForDeletion/ID-fallback
github.com/dynatrace/dynatrace-configuration-as-code/v2/pkg/download/automation ‑ Test_createTemplateFromRawJSON/sets_template_ID_to_object_ID

♻️ This comment has been updated with latest results.

UnseenWizzard and others added 5 commits October 5, 2023 14:19
The updated FileBasedTemplate loads file contents only when required,
instead of at creation, reducing the long term in-use memory footprint
 of configurations.

The new InMemoryTemplate replaces the Download specific template, as
well as some dedicated test implementations of templates.
It currently carries an optional filepath, which is used for writing
converted templates to the desired target file.
Method signatures are refactored to return now possible teplate read/write
errors.
Additionally unused coordinate returns are removed.
%q does not only quote the variable's content, but also makes it 'safe'.
It's not clearly defined in the golang docs, but backslashes are escaped as well.
To properly render the error, we need to quote the string ourselves and print the string using %s
…er persisted

FileBasedTemplate is only used when a config was loaded from a file (in deployments), and
these types of Templates will never be passed to the config writer.
Only download and conversion will ever persist their configurations, and both create
InMemoryTemplates.

To simplify the persistence code and don't make it look like we ever persist FileBasedTemplates,
the codepath is simply removed.
If an unexpected type of Template still get's passed to the Writer, an error is returned.
If we ever extend Templates and don't implement persistence logic, we'll get an error.

As the interface already includes an error return and we won't notice this on compile,
an error was chosen over a hard-exit by panicking.
@UnseenWizzard UnseenWizzard force-pushed the feat/defer-template-load branch from 7b87e56 to a4511cf Compare October 5, 2023 12:19
@UnseenWizzard UnseenWizzard added the run-e2e-test Manually trigger the E2E tests for reviewed PRs label Oct 5, 2023
@sonarqubecloud
Copy link

sonarqubecloud bot commented Oct 5, 2023

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 2 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

@github-actions
Copy link

github-actions bot commented Oct 5, 2023

E2E Test Results

    3 files  ±0  330 suites  ±0   13m 37s ⏱️ - 5m 20s
106 tests ±0  105 ✔️ ±0  1 💤 ±0  0 ±0 
118 runs  ±0  117 ✔️ ±0  1 💤 ±0  0 ±0 

Results for commit a4511cf. ± Comparison against base commit de70797.

@UnseenWizzard UnseenWizzard merged commit 2c20206 into main Oct 5, 2023
14 checks passed
@UnseenWizzard UnseenWizzard deleted the feat/defer-template-load branch October 5, 2023 13:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
run-e2e-test Manually trigger the E2E tests for reviewed PRs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants