Cbang/simplify entry point2

[mrbangbricks]

Description

This PR introduces a new start_trace() decorator which simplifies the setup of logging setup inside entry point files of business applications: e.g. in measurements or wholesale. Further, a LoggingSettings() pydantic settings class is used to handle the automatic parsing of cli parameters and environment variables.

An implementation of this functionality is avaialble in this PR in the measurements repo: Energinet-DataHub/opengeh-measurements#180

Files changed:

• .docker/requirements.txt:
  • temporary change to point to own branch
• .github/workflows/create-release-tag.yml
  • Introducing a new version 4.2.0, as we are introducing a breaking change in logging_configuration method ’s parameters
• source/telemetry/release-notes/release-notes.md
  • Description of upgrade of version
• source/telemetry/telemetry_logging/decorators.py
  • Adition of start_trace decorator, which does the following:
    • Checks if an initial span name is provided from initial_span_name parameter. Uses this name if provided, and if not, the name of the function implmenting the decorator is used
    • Checks if logging has been configured in the Telemetery logging module. Raises NotImplementedError if not
    • Starts a tracer using the provided tracer name and logs an initial trace to application insights
    • Calls the function that is implementing the decorator in a try / except block, recording exceptions in Application Insights if they occour
• source/telemetry/telemetry_logging/logging_configuration.py
  • Implements a LoggingSettings class based on Pydantic BaseSettings which parses cli parameters and env variables
  • Changes to configure_logging:
    • Now just accepts a LoggingSettings object for configuration of logging
    • Automaticcaly adds orchestration_instance_id as an extra, as it is required for the Telemetry logs to show up in Application Insights
• source/telemetry/tests/conftest.py
  • Implements relative path to point to telemetery_folder_path, using Path Library
• Test files updated to accomodate the new LoggingSettings class and configure_logging() method:
  • source/telemetry/tests/integration_test/test_integration_azure_monitor.py
  • source/telemetry/tests/telemetry_logging/test_decorators.py
  • source/telemetry/tests/telemetry_logging/test_logging_configuration.py
  • source/telemetry/tests/telemetry_logging/test_span_recording.py

[github-actions]

Coverage report for 'geh_common'

File	Stmts	Miss	Cover	Missing
src/geh_common
__init__.py	0	0	100%
src/geh_common/domain
__init__.py	0	0	100%
src/geh_common/domain/types
__init__.py	14	14	0%	1–13, 15
charge_quality.py	6	6	0%	1, 4–8
charge_resolution.py	8	8	0%	1, 4–10
charge_type.py	5	5	0%	1, 4–7
charge_unit.py	6	6	0%	1, 4–8
metering_point_resolution.py	4	4	0%	1, 4–6
metering_point_sub_type.py	5	5	0%	1, 4–7
metering_point_type.py	18	18	0%	1, 4–20
net_settlement_group.py	5	5	0%	1, 4–7
orchestration_type.py	4	4	0%	1, 4–6
product.py	3	3	0%	1, 4–5
quantity_quality.py	6	6	0%	1, 4–8
quantity_unit.py	3	3	0%	1, 4–5
settlement_method.py	4	4	0%	1, 4–6
src/geh_common/migrations
__init__.py	4	0	100%
container.py	11	0	100%
migration_pipeline.py	6	0	100%
src/geh_common/migrations/constants
__init__.py	0	0	100%
migrations_constants.py	3	0	100%
src/geh_common/migrations/infrastructure
__init__.py	0	0	100%
apply_migration_scripts.py	22	3	86%	24–26
sql_file_executor.py	37	3	91%	29–31
uncommitted_migration_scripts.py	34	1	97%	60
src/geh_common/migrations/models
__init__.py	0	0	100%
configuration.py	9	6	33%	8–13
spark_sql_migrations_configuration.py	15	0	100%
table_version.py	4	2	50%	3–4
src/geh_common/migrations/schemas
__init__.py	0	0	100%
migrations_schema.py	3	0	100%
src/geh_common/migrations/utility
__init__.py	0	0	100%
delta_table_helper.py	34	10	70%	36–37, 41–42, 45, 48, 51–52, 58, 66
src/geh_common/parsing
__init__.py	2	0	100%
pydantic_settings_parsing.py	3	0	100%
src/geh_common/pyspark
__init__.py	0	0	100%
data_frame_wrapper.py	22	0	100%
read_csv.py	33	0	100%
read_csv_test.py	13	0	100%
transformations.py	31	11	64%	32, 47, 49–50, 52, 54–55, 57, 61, 65, 70
src/geh_common/telemetry
__init__.py	4	0	100%
decorators.py	41	3	92%	67–69
logger.py	22	0	100%
logging_configuration.py	53	0	100%
span_recording.py	5	0	100%
src/geh_common/testing
__init__.py	0	0	100%
src/geh_common/testing/container_test
__init__.py	2	2	0%	1, 3
databricks_api_client.py	36	36	0%	1, 3–4, 7–9, 11, 20–25, 27, 37–38, 40, 51, 53–56, 58–60, 62–64, 66–71, 73, 75
src/geh_common/testing/covernator
TestCases.py	47	3	93%	22, 56–57
__init__.py	2	0	100%
src/geh_common/testing/dataframes
__init__.py	5	0	100%
assert_dataframes.py	108	49	54%	37–39, 42–43, 72–75, 79–81, 83–85, 89–91, 93–95, 99–101, 103–108, 112–114, 116, 119, 127–128, 131–132, 144–145, 149–152, 154–156, 158
assert_schemas.py	63	4	93%	126, 159–160, 167
read_csv.py	16	0	100%
write_to_delta.py	14	2	85%	34–35
src/geh_common/testing/delta_lake
__init__.py	2	0	100%
delta_lake_operations.py	17	1	94%	29
src/geh_common/testing/scenario_testing
TestCases.py	31	2	93%	20, 60
__init__.py	3	0	100%
get_then_names.py	13	2	84%	25, 28
TOTAL	861	231	73%

[jpm-halfspace]

Sure this shouldn't just inherit from pydantic.BaseModel?

[mrbangbricks]

Yes, as we discussed with Bjarke that we want a generic Pydantic class where we control the most generic settings configurations, that meet our use cases with parameters and environment variables, such as:

cli_parse_args=True,
cli_kebab_case=True,
cli_ignore_unknown_args=True, (parsing will fail otherwise)
cli_implicit_flags=True (We have this some places)

[jpm-halfspace]

Why not just default this to an empty dict so you don't have to check for None later?

[mrbangbricks]

You are thinking about the extras dict right? I agree that it should just default to None

[BjarkeMeier]

Be aware of the default args caveats:
https://medium.com/@nebiyuelias1/be-careful-when-using-default-arguments-in-python-fd92df94efee

[jpm-halfspace]

Can you parameterize this so that these can be executed in parallel?

[mrbangbricks]

There is a general delay before the logs show up in application insights, so when the first query succeeds, the remaining three queries find the results instantly - so I don't think it will improve the speed of things to parameterize this :-)

[mrbangbricks]

We can do it for prettification though if you agree?