Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add schema collection to Postgres integration #15278

Closed
wants to merge 94 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
94 commits
Select commit Hold shift + click to select a range
c1e6be1
Add limited pool + LRU cache to MultiDatabaseConnectionPool
edengorevoy Jun 16, 2023
0bc9124
remove a rogue print
edengorevoy Jun 16, 2023
40da126
fixed formatting
edengorevoy Jun 16, 2023
c32e4b4
more helpful documentation
edengorevoy Jun 16, 2023
fe2b051
accidentally added an extraneous file
edengorevoy Jun 16, 2023
75bb16c
reverse the quotation mark replacements
edengorevoy Jun 16, 2023
09b8e99
style error
edengorevoy Jun 16, 2023
1e47389
add context managed connection, change eviction algo
edengorevoy Jun 20, 2023
a7efdba
push dependencies
edengorevoy Jun 20, 2023
1d7b89d
update docstring
edengorevoy Jun 20, 2023
185ee49
style
edengorevoy Jun 20, 2023
c9a6fec
remove unnecessary code, update references
edengorevoy Jun 20, 2023
b6a5ddd
update references
edengorevoy Jun 20, 2023
10f5237
init commit
edengorevoy Jun 13, 2023
6dfd9ed
update base branch
edengorevoy Jun 20, 2023
a0079e7
rebasing
edengorevoy Jun 20, 2023
ab5f458
discovery init
edengorevoy Jun 20, 2023
dd2a5e6
reverting quotation mark changes
edengorevoy Jun 20, 2023
629c5f6
cleaner code, and fixing references in other places
edengorevoy Jun 20, 2023
9995c7f
update from base, config changes, parsing includelist
edengorevoy Jun 21, 2023
1e9c1b0
Merge branch 'limited_db_pool' into pg_autodiscovery
edengorevoy Jun 21, 2023
e467d30
first test
edengorevoy Jun 21, 2023
47abea5
Merge branch 'master' into pg_autodiscovery
edengorevoy Jun 21, 2023
03b10fa
fixing test
edengorevoy Jun 21, 2023
64bc278
fix tests
edengorevoy Jun 21, 2023
5de7d8b
enable relations with autodiscovery
edengorevoy Jun 21, 2023
f382c03
updating with tests
edengorevoy Jun 23, 2023
1e9f32b
changing the dbs, testing discovery
edengorevoy Jun 23, 2023
87f53a9
tests
edengorevoy Jun 27, 2023
5fd5bf2
adding logs to cehck
edengorevoy Jun 27, 2023
3733f6e
more logs
edengorevoy Jun 27, 2023
a987a83
one more log
edengorevoy Jun 27, 2023
9734820
one more log
edengorevoy Jun 27, 2023
f649030
fixing print
edengorevoy Jun 27, 2023
8659ab1
removing extra logs
edengorevoy Jun 27, 2023
ab65c48
assert config validity
edengorevoy Jun 27, 2023
a7c0c92
styling
edengorevoy Jun 27, 2023
8243c6e
test change
edengorevoy Jun 27, 2023
b7f5ebc
license headers
edengorevoy Jun 27, 2023
26d7ff2
syncing files
edengorevoy Jun 27, 2023
70ee7da
fixing setup script
edengorevoy Jun 27, 2023
d4b9312
force only avaiallble in version >=13
edengorevoy Jun 27, 2023
813f24a
requires over 13 line
edengorevoy Jun 27, 2023
1e6c6b3
changed my mind about altering the tables at setup :,)
edengorevoy Jun 27, 2023
4786c58
commit didnt update?:
edengorevoy Jun 27, 2023
375c146
accdientally removed space
edengorevoy Jun 27, 2023
fa40504
style
edengorevoy Jun 27, 2023
d3edeca
cleaning up code, adding better logs, changing refresh default
edengorevoy Jun 29, 2023
7622508
tracking method
edengorevoy Jun 29, 2023
be97495
validate config
edengorevoy Jun 29, 2023
dcba329
fixed autodiscovery building
edengorevoy Jun 29, 2023
34a5607
added db config error
edengorevoy Jun 29, 2023
8aef4e1
fixing warning for max dbs
edengorevoy Jun 29, 2023
c9f5135
fixing warning for max dbs
edengorevoy Jun 29, 2023
54f22df
too many reprints of databases
edengorevoy Jun 29, 2023
1f56342
tracking
edengorevoy Jun 29, 2023
41065f7
style
edengorevoy Jun 29, 2023
f309a83
tracked method remove param
edengorevoy Jun 29, 2023
7dd3623
Merge branch 'master' into pg_autodiscovery
edengorevoy Jul 3, 2023
8d5b41e
instrument the autodicovery function
edengorevoy Jul 3, 2023
9e93399
comment changes
edengorevoy Jul 3, 2023
d786806
add tag to warnings
edengorevoy Jul 5, 2023
7397cc6
test change
edengorevoy Jul 5, 2023
fb7be22
fix tests from adding tags
edengorevoy Jul 5, 2023
8e80d54
remove accidental *1000 multiplier for collection time
edengorevoy Jul 5, 2023
16428df
Merge branch 'master' into pg_autodiscovery
edengorevoy Jul 6, 2023
e8a3f43
first commit
edengorevoy Jul 14, 2023
f68196b
All schema collection except ordering by metrics
edengorevoy Jul 17, 2023
2af2cac
Merge branch 'master' into collect-schema-metadata
edengorevoy Jul 17, 2023
ee524c5
merge master
edengorevoy Jul 17, 2023
d1aa666
Ordering tables by relation metrics
edengorevoy Jul 17, 2023
d040a9c
toast table changes
edengorevoy Jul 18, 2023
1f01f25
Adding some testing, and updating the payload
edengorevoy Jul 18, 2023
8c7725f
Add a debug log
edengorevoy Jul 19, 2023
8042b94
Updating last_schemas_query at correct time
edengorevoy Jul 19, 2023
414db6c
Actually return the metadata from _collect_schema_info
edengorevoy Jul 19, 2023
4c27dda
Convert db id from int to str
edengorevoy Jul 19, 2023
211f825
Correct json objects for indexes, foreign keys
edengorevoy Jul 20, 2023
465f214
Correct json objects for indexes, foreign keys
edengorevoy Jul 20, 2023
75aaddb
Updating queries and toast table getting
edengorevoy Aug 2, 2023
298cf1b
Config update
edengorevoy Aug 2, 2023
679f0ae
Updating to include OIDs in return, and don't calculate fk for tables…
edengorevoy Aug 3, 2023
727ea75
Update some comments
edengorevoy Aug 3, 2023
e6d3e88
Make test mor specific
edengorevoy Aug 3, 2023
a19b9a6
Update config to limit columns
edengorevoy Aug 3, 2023
4522c44
Formatting
edengorevoy Aug 3, 2023
65d28f8
Remove excess warnings and prints
edengorevoy Aug 3, 2023
2b84793
Merge branch 'master' into collect-schema-metadata
edengorevoy Aug 3, 2023
4dbf0ef
Fix schema id propagation
edengorevoy Aug 3, 2023
51c34fd
Clean up caching
edengorevoy Aug 3, 2023
5f90ab1
Formatting, updating to psycopg3
edengorevoy Aug 3, 2023
3e7b949
Conditionals for postgres version 9, no partitions
edengorevoy Aug 4, 2023
97dfe4e
Update table size test
edengorevoy Aug 4, 2023
441499e
Merge branch 'master' of github.com:DataDog/integrations-core into co…
edengorevoy Aug 4, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions postgres/assets/configuration/spec.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -513,6 +513,40 @@ files:
type: number
example: 600

- name: collect_schemas
description: |
Enable collection of database schemas. In order to collect schemas from all user databases,
enable `database_autodiscovery`. To collect from a single database, set `dbname` to collect
the schema for that database.
Relation metrics must be enabled for schema collection.
options:
- name: enabled
description: |
Enable collection of database schemas. Requires `dbm: true` and relation metrics must be enabled.
value:
type: boolean
example: false
- name: max_tables
description: |
Maximum amount of tables the agent will collect from the instance.
value:
type: number
example: 1000
display_default: 1000
- name: max_columns
description: |
Maximum amount of columns the agent will collect per table.
value:
type: number
example: 50
display_default: 50
- name: collection_interval
description: |
The database schema collection interval (in seconds).
value:
type: number
example: 600

- name: aws
description: |
This block defines the configuration for AWS RDS and Aurora instances.
Expand Down
6 changes: 6 additions & 0 deletions postgres/datadog_checks/postgres/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,12 @@ def __init__(self, instance):
self.pg_stat_activity_view = instance.get('pg_stat_activity_view', 'pg_stat_activity')
self.statement_samples_config = instance.get('query_samples', instance.get('statement_samples', {})) or {}
self.settings_metadata_config = instance.get('collect_settings', {}) or {}
self.schemas_metadata_config = instance.get('collect_schemas', {"enabled": False})
if not self.relations and self.schemas_metadata_config['enabled']:
raise ConfigurationError(
'In order to collect schemas on this database, you must enable relation metrics collection.'
)

self.resources_metadata_config = instance.get('collect_resources', {}) or {}
self.statement_activity_config = instance.get('query_activity', {}) or {}
self.statement_metrics_config = instance.get('query_metrics', {}) or {}
Expand Down
27 changes: 27 additions & 0 deletions postgres/datadog_checks/postgres/data/conf.yaml.example
Original file line number Diff line number Diff line change
Expand Up @@ -406,6 +406,33 @@ instances:
#
# collection_interval: 600

## Enable collection of database schemas. In order to collect schemas from all user databases,
## enable `database_autodiscovery`. To collect from a single database, set `dbname` to collect
## the schema for that database.
## Relation metrics must be enabled for schema collection.
#
# collect_schemas:

## @param enabled - boolean - optional - default: false
## Enable collection of database schemas. Requires `dbm: true` and relation metrics must be enabled.
#
# enabled: false

## @param max_tables - number - optional - default: 1000
## Maximum amount of tables the agent will collect from the instance.
#
# max_tables: 1000

## @param max_columns - number - optional - default: 50
## Maximum amount of columns the agent will collect per table.
#
# max_columns: 50

## @param collection_interval - number - optional - default: 600
## The database schema collection interval (in seconds).
#
# collection_interval: 600

## This block defines the configuration for AWS RDS and Aurora instances.
##
## Complete this section if you have installed the Datadog AWS Integration
Expand Down
Loading
Loading