Collect stored procedure metrics from dm_exec_procedure_stats (#15805)

* stored procs * fixes * fix query * Add config * lint * fix lint * debug test * fix tests * accelerate tests * two tests * tests * lint * fix lint: * fix tests and add more tests * Changelog and metadata update * fix changelog * fix spec * fix spec * fix headers * fix instance.py * oops * fix instance.py * feedback * lint * payload_type -> kind --------- Co-authored-by: Joel Marcotte <[email protected]>
DataDog · Sep 28, 2023 · b820284 · b820284
1 parent 242fe77
commit b820284
Show file tree

Hide file tree

Showing 16 changed files with 699 additions and 60 deletions.
diff --git a/sqlserver/CHANGELOG.md b/sqlserver/CHANGELOG.md
@@ -14,6 +14,7 @@
 * Add TempDB version store performance counters ([#15879](https://github.com/DataDog/integrations-core/pull/15879))
 * Add TempDB page counts metrics ([#15873](https://github.com/DataDog/integrations-core/pull/15873))
 * Add `index_name` tag to `.database.avg_fragmentation_in_percent`, `.database.fragment_count`, `.database.avg_fragment_size_in_pages` metrics. Also add a new metric `sqlserver.database.index_page_count`, tagged by `database_name`, `object_name`, `index_id` and `index_name`. ([#15721](https://github.com/DataDog/integrations-core/pull/15721))
+* When DBM is enabled, starts collecting stored procedure metrics from sys.dm_exec_procedure_stats at 60s interval (configurable). Also adds the corresponding `procedure_metrics` section to the config file. The new DBM-only metrics are `sqlserver.procedures.count`, `sqlserver.procedures.time`, `sqlserver.procedures.worker_time`, `sqlserver.procedures.physical_reads`, `sqlserver.procedures.logical_reads`, `sqlserver.procedures.logical_writes` and `sqlserver.procedures.spills`. ([#15805](https://github.com/DataDog/integrations-core/pull/15805))
 
 ***Added***:
 

diff --git a/sqlserver/assets/configuration/spec.yaml b/sqlserver/assets/configuration/spec.yaml
@@ -26,11 +26,11 @@ files:
       Note: All '%' characters must be escaped as '%%'.
     options:
     - name: host
-      description: | 
-        Host and port of your SQL server. If a port is ommitted, a default port of 1433 will be used. 
+      description: |
+        Host and port of your SQL server. If a port is ommitted, a default port of 1433 will be used.
         If you use Sql Server Browser Service or a similar port autodiscovery service, pass in a port
-        of 0 to omit port from your connection string. This should allow SSBS to autodiscover the 
-        correct database port. 
+        of 0 to omit port from your connection string. This should allow SSBS to autodiscover the
+        correct database port.
       required: true
       value:
         type: string
@@ -45,9 +45,9 @@ files:
         type: string
     - name: server_version
       description: |
-        Server version year of sqlserver the agent will connect to. 
+        Server version year of sqlserver the agent will connect to.
         Important for validating connection string attributes for older sqlserver versions.
-        
+
         This is required if connecting to a SQLServer instance older than 2014.
       value:
         type: string
@@ -66,7 +66,7 @@ files:
         type: string
     - name: database_autodiscovery
       description: |
-        Auto-discover and monitor databases. Supported for the metrics check. 
+        Auto-discover and monitor databases. Supported for the metrics check.
         If `true`, overrides `database` option.
         Can be combined with `autodiscovery_include` and `autodiscovery_exclude` options.
       value:
@@ -271,7 +271,7 @@ files:
             example: true
         - name: disable_secondary_tags
           description: |
-            Disable the secondary tags on query aggregate metrics, such as `user` and `database`. This is an optimization 
+            Disable the secondary tags on query aggregate metrics, such as `user` and `database`. This is an optimization
             for certain database workloads which result in long metric collection times.
           value:
             type: boolean
@@ -310,14 +310,48 @@ files:
           hidden: true
         - name: max_queries
           description: |
-            Limit the number of queries sent to the backend. 
+            Limit the number of queries sent to the backend.
             Note: The only time this value would need to be set is in special cases where the query limit
             applied on the backend is being modified.
           value:
             type: integer
             example: 250
             display_default: 250
           hidden: true
+    - name: procedure_metrics
+      description: Configure collection of procedure metrics
+      options:
+        - name: enabled
+          description: |
+            Enable collection of procedure metrics. Requires `dbm: true`.
+          value:
+            type: boolean
+            example: true
+        - name: collection_interval
+          description: |
+            Set the procedure metric collection interval (in seconds). Each collection involves one query to
+            sys.dm_exec_procedure_stats. If a non-default value is chosen then that exact same value must
+            be used for *every* check instance. Running different instances with different collection intervals
+            is not supported.
+          value:
+            type: number
+            example: 60
+        - name: dm_exec_procedure_stats_row_limit
+          description: |
+            Set the maximum number of procedure stats rows that can be retrieved in a single check run.
+          value:
+            type: integer
+            example: 10000
+        - name: max_procedures
+          description: |
+            Limit the number of procedures sent to the backend.
+            Note: The only time this value would need to be set is in special cases where the procedure limit
+            applied on the backend is being modified.
+          value:
+            type: integer
+            example: 250
+            display_default: 250
+          hidden: true
     - name: query_activity
       description: Configure collection of active sessions monitoring
       options:
@@ -339,20 +373,20 @@ files:
 
     - name: aws
       description: |
-        This block defines the configuration for AWS RDS and Aurora instances. 
-        
-        Complete this section if you have installed the Datadog AWS Integration 
-        (https://docs.datadoghq.com/integrations/amazon_web_services) to enrich instances 
+        This block defines the configuration for AWS RDS and Aurora instances.
+
+        Complete this section if you have installed the Datadog AWS Integration
+        (https://docs.datadoghq.com/integrations/amazon_web_services) to enrich instances
         with SQL Server integration telemetry.
-        
+
         These values are only applied when `dbm: true` option is set.
       options:
         - name: instance_endpoint
           description: |
-            Equal to the Endpoint.Address of the instance the agent is connecting to. 
+            Equal to the Endpoint.Address of the instance the agent is connecting to.
             This value is optional if the value of `host` is already configured to the instance endpoint.
-            
-            For more information on instance endpoints, 
+
+            For more information on instance endpoints,
             see the AWS docs https://docs.aws.amazon.com/AmazonRDS/latest/APIReference/API_Endpoint.html
           value:
             type: string
@@ -361,17 +395,17 @@ files:
     - name: gcp
       description: |
         This block defines the configuration for Google Cloud SQL instances.
-        
-        Complete this section if you have installed the Datadog GCP Integration 
-        (https://docs.datadoghq.com/integrations/google_cloud_platform) to enrich instances 
+
+        Complete this section if you have installed the Datadog GCP Integration
+        (https://docs.datadoghq.com/integrations/google_cloud_platform) to enrich instances
         with SQL Server integration telemetry.
 
         These values are only applied when `dbm: true` option is set.
       options:
         - name: project_id
           description: |
             Equal to the GCP resource's project ID.
-            
+
             For more information on project IDs,
             See the GCP docs https://cloud.google.com/resource-manager/docs/creating-managing-projects
           value:
@@ -380,7 +414,7 @@ files:
         - name: instance_id
           description: |
             Equal to the GCP resource's instance ID.
-            
+
             For more information on instance IDs,
             See the GCP docs https://cloud.google.com/sql/docs/sqlserver/instance-settings#instance-id-2ndgen
           value:
@@ -389,24 +423,24 @@ files:
 
     - name: azure
       description: |
-        This block defines the configuration for Azure Managed Instance, Azure SQL Database or 
+        This block defines the configuration for Azure Managed Instance, Azure SQL Database or
         SQLServer on Virtual Machines.
-        
-        Complete this section if you have installed the Datadog Azure Integration 
-        (https://docs.datadoghq.com/integrations/azure) to enrich instances 
+
+        Complete this section if you have installed the Datadog Azure Integration
+        (https://docs.datadoghq.com/integrations/azure) to enrich instances
         with SQL Server integration telemetry.
 
         These values are only applied when `dbm: true` option is set.
       options:
         - name: deployment_type
           description: |
-            Equal to the deployment type for the managed database. 
-            
+            Equal to the deployment type for the managed database.
+
             Acceptable values are:
-              - `sql_database` 
+              - `sql_database`
               - `managed_instance`
               - `virtual_machine`
-            
+
             For more information on deployment types, see the Azure
             docs https://docs.microsoft.com/en-us/azure/azure-sql/azure-sql-iaas-vs-paas-what-is-overview?view=azuresql
           value:
@@ -415,7 +449,7 @@ files:
         - name: fully_qualified_domain_name
           description: |
             Equal to the name of the SQL Database, Managed Instance (SQL MI) or virtual machine.
-            
+
             This value is optional if the value of `host` is already configured to the fully qualified domain name.
           value:
             type: string
@@ -424,10 +458,10 @@ files:
     - name: managed_identity
       description: |
         Configuration section used for Azure AD Authentication.
-        
+
         This supports using System or User assigned managed identities.
         If this section is set, then the `username` and `password` fields will be ignored.
-        
+
         For more information on Managed Identities, see the Azure docs
         https://learn.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/overview
       options:
@@ -440,7 +474,7 @@ files:
           description: |
             The permission scope from where to access the identity token. This value is optional if using the default
             identity scope for Azure managed databases.
-            
+
             For more information on scopes, see the Azure docs
             https://learn.microsoft.com/en-us/azure/active-directory/develop/scopes-oidc
           value:
@@ -552,7 +586,7 @@ files:
       hidden: true
       description: |
         Set the database instance collection interval (in seconds). The database instance collection sends
-        basic information about the database instance along with a signal that it still exists. 
+        basic information about the database instance along with a signal that it still exists.
         This collection does not involve any additional queries to the database.
       value:
         type: number

diff --git a/sqlserver/datadog_checks/sqlserver/config_models/instance.py b/sqlserver/datadog_checks/sqlserver/config_models/instance.py
@@ -96,6 +96,17 @@ class ObfuscatorOptions(BaseModel):
     replace_digits: Optional[bool] = None
 
 
+class ProcedureMetrics(BaseModel):
+    model_config = ConfigDict(
+        arbitrary_types_allowed=True,
+        frozen=True,
+    )
+    collection_interval: Optional[float] = None
+    dm_exec_procedure_stats_row_limit: Optional[int] = None
+    enabled: Optional[bool] = None
+    max_procedures: Optional[int] = None
+
+
 class QueryActivity(BaseModel):
     model_config = ConfigDict(
         arbitrary_types_allowed=True,
@@ -169,6 +180,7 @@ class InstanceConfig(BaseModel):
     password: Optional[str] = None
     proc_only_if: Optional[str] = None
     proc_only_if_database: Optional[str] = None
+    procedure_metrics: Optional[ProcedureMetrics] = None
     query_activity: Optional[QueryActivity] = None
     query_metrics: Optional[QueryMetrics] = None
     reported_hostname: Optional[str] = None

diff --git a/sqlserver/datadog_checks/sqlserver/data/conf.yaml.example b/sqlserver/datadog_checks/sqlserver/data/conf.yaml.example
@@ -37,10 +37,10 @@ init_config:
 instances:
 
     ## @param host - string - required
-    ## Host and port of your SQL server. If a port is ommitted, a default port of 1433 will be used. 
+    ## Host and port of your SQL server. If a port is ommitted, a default port of 1433 will be used.
     ## If you use Sql Server Browser Service or a similar port autodiscovery service, pass in a port
-    ## of 0 to omit port from your connection string. This should allow SSBS to autodiscover the 
-    ## correct database port. 
+    ## of 0 to omit port from your connection string. This should allow SSBS to autodiscover the
+    ## correct database port.
     #
   - host: <HOST>,<PORT>
 
@@ -55,7 +55,7 @@ instances:
     # password: <PASSWORD>
 
     ## @param server_version - string - optional - default: 2014
-    ## Server version year of sqlserver the agent will connect to. 
+    ## Server version year of sqlserver the agent will connect to.
     ## Important for validating connection string attributes for older sqlserver versions.
     ##
     ## This is required if connecting to a SQLServer instance older than 2014.
@@ -74,7 +74,7 @@ instances:
     # reported_hostname: <REPORTED_HOSTNAME>
 
     ## @param database_autodiscovery - boolean - optional - default: false
-    ## Auto-discover and monitor databases. Supported for the metrics check. 
+    ## Auto-discover and monitor databases. Supported for the metrics check.
     ## If `true`, overrides `database` option.
     ## Can be combined with `autodiscovery_include` and `autodiscovery_exclude` options.
     #
@@ -271,6 +271,28 @@ instances:
         #
         # samples_per_hour_per_query: 4
 
+    ## Configure collection of procedure metrics
+    #
+    # procedure_metrics:
+
+        ## @param enabled - boolean - optional - default: true
+        ## Enable collection of procedure metrics. Requires `dbm: true`.
+        #
+        # enabled: true
+
+        ## @param collection_interval - number - optional - default: 60
+        ## Set the procedure metric collection interval (in seconds). Each collection involves one query to
+        ## sys.dm_exec_procedure_stats. If a non-default value is chosen then that exact same value must
+        ## be used for *every* check instance. Running different instances with different collection intervals
+        ## is not supported.
+        #
+        # collection_interval: 60
+
+        ## @param dm_exec_procedure_stats_row_limit - integer - optional - default: 10000
+        ## Set the maximum number of procedure stats rows that can be retrieved in a single check run.
+        #
+        # dm_exec_procedure_stats_row_limit: 10000
+
     ## Configure collection of active sessions monitoring
     #
     # query_activity:
@@ -288,29 +310,29 @@ instances:
         #
         # collection_interval: 10
 
-    ## This block defines the configuration for AWS RDS and Aurora instances. 
+    ## This block defines the configuration for AWS RDS and Aurora instances.
     ##
-    ## Complete this section if you have installed the Datadog AWS Integration 
-    ## (https://docs.datadoghq.com/integrations/amazon_web_services) to enrich instances 
+    ## Complete this section if you have installed the Datadog AWS Integration
+    ## (https://docs.datadoghq.com/integrations/amazon_web_services) to enrich instances
     ## with SQL Server integration telemetry.
     ##
     ## These values are only applied when `dbm: true` option is set.
     #
     # aws:
 
         ## @param instance_endpoint - string - optional - default: mydb.cfxgae8cilcf.us-east-1.rds.amazonaws.com
-        ## Equal to the Endpoint.Address of the instance the agent is connecting to. 
+        ## Equal to the Endpoint.Address of the instance the agent is connecting to.
         ## This value is optional if the value of `host` is already configured to the instance endpoint.
         ##
-        ## For more information on instance endpoints, 
+        ## For more information on instance endpoints,
         ## see the AWS docs https://docs.aws.amazon.com/AmazonRDS/latest/APIReference/API_Endpoint.html
         #
         # instance_endpoint: mydb.cfxgae8cilcf.us-east-1.rds.amazonaws.com
 
     ## This block defines the configuration for Google Cloud SQL instances.
     ##
-    ## Complete this section if you have installed the Datadog GCP Integration 
-    ## (https://docs.datadoghq.com/integrations/google_cloud_platform) to enrich instances 
+    ## Complete this section if you have installed the Datadog GCP Integration
+    ## (https://docs.datadoghq.com/integrations/google_cloud_platform) to enrich instances
     ## with SQL Server integration telemetry.
     ##
     ## These values are only applied when `dbm: true` option is set.
@@ -333,22 +355,22 @@ instances:
         #
         # instance_id: foo-database
 
-    ## This block defines the configuration for Azure Managed Instance, Azure SQL Database or 
+    ## This block defines the configuration for Azure Managed Instance, Azure SQL Database or
     ## SQLServer on Virtual Machines.
     ##
-    ## Complete this section if you have installed the Datadog Azure Integration 
-    ## (https://docs.datadoghq.com/integrations/azure) to enrich instances 
+    ## Complete this section if you have installed the Datadog Azure Integration
+    ## (https://docs.datadoghq.com/integrations/azure) to enrich instances
     ## with SQL Server integration telemetry.
     ##
     ## These values are only applied when `dbm: true` option is set.
     #
     # azure:
 
         ## @param deployment_type - string - optional - default: sql_database
-        ## Equal to the deployment type for the managed database. 
+        ## Equal to the deployment type for the managed database.
         ##
         ## Acceptable values are:
-        ##   - `sql_database` 
+        ##   - `sql_database`
         ##   - `managed_instance`
         ##   - `virtual_machine`
         ##