diff --git a/.doc_gen/metadata/cloudwatch-logs_metadata.yaml b/.doc_gen/metadata/cloudwatch-logs_metadata.yaml index 3b7a2e80635..e923c3eeab2 100644 --- a/.doc_gen/metadata/cloudwatch-logs_metadata.yaml +++ b/.doc_gen/metadata/cloudwatch-logs_metadata.yaml @@ -315,6 +315,14 @@ cloudwatch-logs_GetQueryResults: - description: snippet_tags: - javascript.v3.cloudwatch-logs.actions.GetQueryResults + Python: + versions: + - sdk_version: 3 + github: python/example_code/cloudwatch-logs + excerpts: + - description: + snippet_tags: + - python.example_code.cloudwatch_logs.get_query_results services: cloudwatch-logs: {GetQueryResults} cloudwatch-logs_StartQuery: @@ -331,6 +339,14 @@ cloudwatch-logs_StartQuery: - description: snippet_tags: - javascript.v3.cloudwatch-logs.actions.StartQuery + Python: + versions: + - sdk_version: 3 + github: python/example_code/cloudwatch-logs + excerpts: + - description: + snippet_tags: + - python.example_code.cloudwatch_logs.start_query services: cloudwatch-logs: {StartQuery} cloudwatch-logs_Scenario_BigQuery: @@ -350,5 +366,16 @@ cloudwatch-logs_Scenario_BigQuery: - description: This is a class that splits queries into multiple steps if necessary. - snippet_files: - javascriptv3/example_code/cloudwatch-logs/scenarios/large-query/cloud-watch-query.js + Python: + versions: + - sdk_version: 3 + github: python/example_code/cloudwatch-logs/scenarios/large-query + excerpts: + - description: This file invokes an example module for managing CloudWatch queries exceeding 10,000 results. + snippet_files: + - python/example_code/cloudwatch-logs/scenarios/large-query/exec.py + - description: This module processes CloudWatch queries exceeding 10,000 results. + - snippet_files: + - python/example_code/cloudwatch-logs/scenarios/large-query/cloudwatch_query.py services: cloudwatch-logs: {StartQuery, GetQueryResults} diff --git a/python/example_code/cloudwatch-logs/scenarios/large-query/README.md b/python/example_code/cloudwatch-logs/scenarios/large-query/README.md new file mode 100644 index 00000000000..fd53fafc27c --- /dev/null +++ b/python/example_code/cloudwatch-logs/scenarios/large-query/README.md @@ -0,0 +1,50 @@ +# CloudWatch Logs large query + +## Overview + +This example shows how to use AWS SDKs to perform a query on Amazon CloudWatch Logs and get more than the maximum number of 10,000 logs back. + +The CloudWatch Logs API is capped at 10,000 records for requests that [read](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html) or [write](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_PutLogEvents.html). GetLogEvents returns tokens for pagination, but [GetQueryResults](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetQueryResults.html) does not. This example breaks down one query into multiple queries if more than the maximum number of records are returned from the query. + +The following components are used in this example: + +- [Amazon CloudWatch Logs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html) hosts the logs that are queried using the [Amazon CloudWatch Logs API](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/Welcome.html). + +## ⚠ Important + +- Running this code might result in charges to your AWS account. +- Running the tests might result in charges to your AWS account. +- We recommend that you grant your code least privilege. At most, grant only the minimum permissions required to perform the task. For more information, see [Grant least privilege](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#grant-least-privilege). +- This code is not tested in every AWS Region. For more information, see [AWS Regional Services](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services). + +## Scenario + +### Prerequisites + +For general prerequisites, see the [README](../../../../README.md) in the `python` folder. + +To run this example, you need a CloudWatch log group that contains over 10,000 logs. You can [create one yourself](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Working-with-log-groups-and-streams.html), or you can follow the steps in the [Infrastructure and data](#infrastructure-and-data) section. These steps require you to [install or update the latest version of the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) + +### Infrastructure and data + +Use the following steps to create the necessary resources in AWS CloudFormation and use the AWS CLI to upload the necessary logs. + +1. In your local terminal, change directories to [resources](../../../../../workflows/cloudwatch_logs_large_query/resources/). +1. Run `aws cloudformation deploy --template-file stack.yaml --stack-name CloudWatchLargeQuery` +1. Run `./make-log-files.sh`. This will output two timestamps for use in the following step. +1. Run `export QUERY_START_DATE=`. Replace `` with the output from the previous step. Repeat this for `QUERY_END_DATE`. +1. Run `./put-log-events.sh`. +1. Wait five minutes for logs to settle and to make sure you're not querying for logs that exist in the future. + +### Run the scenario + +1. `python exec.py` + +## Additional reading + +- [CloudWatch Logs Insights query syntax](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax.html) +- [CloudWatch Logs billing and cost](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/LogsBillingDetails.html) + +--- + +Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. SPDX-License-Identifier: Apache-2.0 diff --git a/python/example_code/cloudwatch-logs/scenarios/large-query/cloudwatch_query.py b/python/example_code/cloudwatch-logs/scenarios/large-query/cloudwatch_query.py new file mode 100644 index 00000000000..a356f236ba3 --- /dev/null +++ b/python/example_code/cloudwatch-logs/scenarios/large-query/cloudwatch_query.py @@ -0,0 +1,222 @@ +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 +import logging +import time +from datetime import datetime +import threading +import boto3 + +from date_utilities import DateUtilities + + +class DateOutOfBoundsError(Exception): + """Exception raised when the date range for a query is out of bounds.""" + + pass + + +class CloudWatchQuery: + """ + A class to query AWS CloudWatch logs within a specified date range. + + :ivar date_range: Start and end datetime for the query. + :vartype date_range: tuple + :ivar limit: Maximum number of log entries to return. + :vartype limit: int + """ + + def __init__(self, date_range): + self.lock = threading.Lock() + self.log_groups = "/workflows/cloudwatch-logs/large-query" + self.query_results = [] + self.date_range = date_range + self.query_duration = None + self.datetime_format = "%Y-%m-%d %H:%M:%S.%f" + self.date_utilities = DateUtilities() + self.limit = 10000 + + def query_logs(self, date_range): + """ + Executes a CloudWatch logs query for a specified date range and calculates the execution time of the query. + + :return: A batch of logs retrieved from the CloudWatch logs query. + :rtype: list + """ + start_time = datetime.now() + + start_date, end_date = self.date_utilities.normalize_date_range_format( + date_range, from_format="unix_timestamp", to_format="datetime" + ) + + logging.info( + f"Original query:" + f"\n START: {start_date}" + f"\n END: {end_date}" + ) + self.recursive_query((start_date, end_date)) + end_time = datetime.now() + self.query_duration = (end_time - start_time).total_seconds() + + def recursive_query(self, date_range): + """ + Processes logs within a given date range, fetching batches of logs recursively if necessary. + + :param date_range: The date range to fetch logs for, specified as a tuple (start_timestamp, end_timestamp). + :type date_range: tuple + :return: None if the recursive fetching is continued or stops when the final batch of logs is processed. + Although it doesn't explicitly return the query results, this method accumulates all fetched logs + in the `self.query_results` attribute. + :rtype: None + """ + batch_of_logs = self.perform_query(date_range) + # Add the batch to the accumulated logs + with self.lock: + self.query_results.extend(batch_of_logs) + if len(batch_of_logs) == self.limit: + logging.info(f"Fetched {self.limit}, checking for more...") + most_recent_log = self.find_most_recent_log(batch_of_logs) + most_recent_log_timestamp = next( + item["value"] + for item in most_recent_log + if item["field"] == "@timestamp" + ) + new_range = (most_recent_log_timestamp, date_range[1]) + midpoint = self.date_utilities.find_middle_time(new_range) + + first_half_thread = threading.Thread( + target=self.recursive_query, + args=((most_recent_log_timestamp, midpoint),), + ) + second_half_thread = threading.Thread( + target=self.recursive_query, args=((midpoint, date_range[1]),) + ) + + first_half_thread.start() + second_half_thread.start() + + first_half_thread.join() + second_half_thread.join() + + def find_most_recent_log(self, logs): + """ + Search a list of log items and return most recent log entry. + :param logs: A list of logs to analyze. + :return: log + :type :return List containing log item details + """ + most_recent_log = None + most_recent_date = "1970-01-01 00:00:00.000" + + for log in logs: + for item in log: + if item["field"] == "@timestamp": + logging.debug(f"Compared: {item['value']} to {most_recent_date}") + if ( + self.date_utilities.compare_dates( + item["value"], most_recent_date + ) + == item["value"] + ): + logging.debug(f"New most recent: {item['value']}") + most_recent_date = item["value"] + most_recent_log = log + logging.info(f"Most recent log date of batch: {most_recent_date}") + return most_recent_log + + # snippet-start:[python.example_code.cloudwatch_logs.start_query] + def perform_query(self, date_range): + """ + Performs the actual CloudWatch log query. + + :param date_range: A tuple representing the start and end datetime for the query. + :type date_range: tuple + :return: A list containing the query results. + :rtype: list + """ + client = boto3.client("logs") + try: + try: + start_time = round( + self.date_utilities.convert_iso8601_to_unix_timestamp(date_range[0]) + ) + end_time = round( + self.date_utilities.convert_iso8601_to_unix_timestamp(date_range[1]) + ) + response = client.start_query( + logGroupName=self.log_groups, + startTime=start_time, + endTime=end_time, + queryString="fields @timestamp, @message | sort @timestamp asc", + limit=self.limit, + ) + query_id = response["queryId"] + except client.exceptions.ResourceNotFoundException as e: + raise DateOutOfBoundsError(f"Resource not found: {e}") + while True: + time.sleep(1) + results = client.get_query_results(queryId=query_id) + if results["status"] in [ + "Complete", + "Failed", + "Cancelled", + "Timeout", + "Unknown", + ]: + return results.get("results", []) + except DateOutOfBoundsError: + return [] + + def _initiate_query(self, client, date_range, max_logs): + """ + Initiates the CloudWatch logs query. + + :param date_range: A tuple representing the start and end datetime for the query. + :type date_range: tuple + :param max_logs: The maximum number of logs to retrieve. + :type max_logs: int + :return: The query ID as a string. + :rtype: str + """ + try: + start_time = round( + self.date_utilities.convert_iso8601_to_unix_timestamp(date_range[0]) + ) + end_time = round( + self.date_utilities.convert_iso8601_to_unix_timestamp(date_range[1]) + ) + response = client.start_query( + logGroupName=self.log_groups, + startTime=start_time, + endTime=end_time, + queryString="fields @timestamp, @message | sort @timestamp asc", + limit=max_logs, + ) + return response["queryId"] + except client.exceptions.ResourceNotFoundException as e: + raise DateOutOfBoundsError(f"Resource not found: {e}") + + # snippet-end:[python.example_code.cloudwatch_logs.start_query] + + # snippet-start:[python.example_code.cloudwatch_logs.get_query_results] + def _wait_for_query_results(self, client, query_id): + """ + Waits for the query to complete and retrieves the results. + + :param query_id: The ID of the initiated query. + :type query_id: str + :return: A list containing the results of the query. + :rtype: list + """ + while True: + time.sleep(1) + results = client.get_query_results(queryId=query_id) + if results["status"] in [ + "Complete", + "Failed", + "Cancelled", + "Timeout", + "Unknown", + ]: + return results.get("results", []) + + # snippet-end:[python.example_code.cloudwatch_logs.get_query_results] diff --git a/python/example_code/cloudwatch-logs/scenarios/large-query/date_utilities.py b/python/example_code/cloudwatch-logs/scenarios/large-query/date_utilities.py new file mode 100644 index 00000000000..ceb9f4cc85b --- /dev/null +++ b/python/example_code/cloudwatch-logs/scenarios/large-query/date_utilities.py @@ -0,0 +1,222 @@ +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 +from datetime import datetime, timezone + + +class DateUtilities: + """A class to help mutate dates in Python.""" + + def __init__(self): + """Initialize the DateUtilities class with default datetime format.""" + self.datetime_format = "%Y-%m-%d %H:%M:%S" + + @staticmethod + def is_datetime(date_string, format_string): + """ + Checks if the given date string matches the specified format. + + :param date_string: The date string to be checked. + :type date_string: str + :param format_string: The format string to check against. + :type format_string: str + :return: True if the date_string matches the format_string, False otherwise. + :rtype: bool + """ + try: + datetime.strptime(date_string, format_string) + return True + except ValueError: + return False + + def find_middle_time(self, date_range) -> tuple: + """ + Find the middle time between two timestamps in ISO8601 format. + Returns: + - str: The middle time in ISO8601 format. + """ + # Parse the ISO8601 formatted strings into datetime objects + dt1 = datetime.fromisoformat(date_range[0]) + dt2 = datetime.fromisoformat(date_range[1]) + + # Ensure dt1 is the earlier datetime + if dt1 > dt2: + dt1, dt2 = dt2, dt1 + + # Calculate the difference between the two datetime objects + difference = dt2 - dt1 + + # Find the halfway duration + halfway = difference / 2 + + # Calculate the middle time + middle_time = dt1 + halfway + + return middle_time.isoformat() + + @staticmethod + def format_iso8601(date_str): + # Parse the ISO8601 date string + dt = datetime.fromisoformat(date_str) + + # Format date without microseconds + date_without_microseconds = dt.strftime("%Y-%m-%dT%H:%M:%S") + + # Format microseconds to remove trailing zeros, ensuring at least 3 digits + microseconds = f".{dt.microsecond:06}".rstrip("0")[:4] + + # Construct the final date string + formatted_date = date_without_microseconds + microseconds + + return formatted_date + + # + @staticmethod + def divide_date_range(date_range): + """ + Splits a date range into two equal halves. + + :param date_range: Start and end datetime in a tuple. + :type date_range: tuple + :return: List of tuples with two split date ranges. + :rtype: list of tuples + """ + midpoint = (date_range[0] + date_range[1]) / 2 + return [(date_range[0], round(midpoint)), (round(midpoint), date_range[1])] + + @staticmethod + def convert_unix_timestamp_to_iso8601( + unix_timestamp, iso8601_format="%Y-%m-%d %H:%M:%S" + ): + """ + Converts a UNIX timestamp in milliseconds to a date string in the specified format. + + :param unix_timestamp: UNIX timestamp in milliseconds. + :type unix_timestamp: int + :param iso8601_format: The format string for the output date string, defaults to "%Y-%m-%d %H:%M:%S.%f". + :type iso8601_format: str + :return: The formatted date string. + :rtype: str + """ + in_seconds = unix_timestamp / 1000.0 + date_time = datetime.utcfromtimestamp(in_seconds) + iso8601 = date_time.strftime(iso8601_format) + return iso8601 + + @staticmethod + def convert_iso8601_to_datetime(iso8601, iso8601_format="%Y-%m-%d %H:%M:%S"): + """ + Converts a date string in ISO 8601 format to a Python datetime object. + + :param iso8601: The ISO 8601 formatted date string. + :type iso8601: str + :param iso8601_format: The format string of the input date, defaults to ISO 8601 format. + :type iso8601_format: str + :return: The corresponding Python datetime object. + :rtype: datetime + """ + # date = datetime.strptime(iso8601, iso8601_format) + date = datetime.fromisoformat(iso8601) + return date + + @staticmethod + def convert_datetime_to_unix_timestamp(dt): + """ + Converts a Python datetime object to a UNIX timestamp in milliseconds. + + :param dt: The datetime object to be converted. + :type dt: datetime + :return: UNIX timestamp in milliseconds. + :rtype: int + """ + unix_timestamp = dt.replace(tzinfo=timezone.utc).timestamp() + return unix_timestamp * 1000 + + def convert_unix_timestamp_to_datetime(self, unix_timestamp): + """ + Converts a UNIX timestamp in milliseconds to a Python datetime object. + + :param unix_timestamp: UNIX timestamp in milliseconds. + :type unix_timestamp: int + :return: The corresponding Python datetime object. + :rtype: datetime + """ + ts = self.convert_unix_timestamp_to_iso8601(unix_timestamp) + dt = self.convert_iso8601_to_datetime(ts) + return dt + + def convert_iso8601_to_unix_timestamp(self, iso8601): + """ + Converts a date string in ISO 8601 format to a UNIX timestamp in milliseconds. + + :param iso8601: The ISO 8601 formatted date string. + :type iso8601: str + :return: UNIX timestamp in milliseconds. + :rtype: int + """ + dt = self.convert_iso8601_to_datetime(iso8601) + unix_timestamp = dt.replace(tzinfo=timezone.utc).timestamp() + return unix_timestamp * 1000 + + def convert_datetime_to_iso8601(self, datetime_obj): + """ + Converts a Python datetime object to ISO 1806 format. + + :param dt: The datetime object to be converted. + :type dt: datetime + :return: ISO 1806. + :rtype: str + """ + unix_timestamp = datetime_obj.replace(tzinfo=timezone.utc).timestamp() + iso8601 = self.convert_unix_timestamp_to_iso8601(round(unix_timestamp * 1000)) + return iso8601 + + def compare_dates(self, date_str1, date_str2): + """ + Compares two dates in ISO 8601 format and returns the later one. + + :param date_str1: The first date string in ISO 8601 format. + :type date_str1: str + :param date_str2: The second date string in ISO 8601 format. + :type date_str2: str + :return: The later of the two dates. + :rtype: str + """ + date1 = datetime.fromisoformat(date_str1) + date2 = datetime.fromisoformat(date_str2) + + if date1 > date2: + return date_str1 + else: + return date_str2 + + def normalize_date_range_format(self, date_range, from_format=None, to_format=None): + """ + Normalizes date ranges received in variable formats to a specified format. + + :param date_range: The date range to be normalized. + :type date_range: tuple + :param from_format: The current format of the date range. + :type from_format: str, optional + :param to_format: The target format for the date range. + :type to_format: str, optional + :return: The normalized date range. + :rtype: tuple + :raises Exception: If required parameters are missing. + """ + if not (to_format, from_format): + raise Exception( + "This function requires a date range, a starting format, and a target format" + ) + if "unix_timestamp" in to_format and "datetime" in from_format: + if not self.is_datetime(date_range[0], self.datetime_format): + start_date = self.convert_unix_timestamp_to_datetime(date_range[0]) + else: + start_date = date_range[0] + + if not self.is_datetime(date_range[1], self.datetime_format): + end_date = self.convert_unix_timestamp_to_datetime(date_range[1]) + else: + end_date = date_range[1] + else: + return date_range + return start_date, end_date diff --git a/python/example_code/cloudwatch-logs/scenarios/large-query/exec.py b/python/example_code/cloudwatch-logs/scenarios/large-query/exec.py new file mode 100644 index 00000000000..402e6346563 --- /dev/null +++ b/python/example_code/cloudwatch-logs/scenarios/large-query/exec.py @@ -0,0 +1,125 @@ +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 +import logging +import os +import sys + +import boto3 +from botocore.config import Config + +from cloudwatch_query import CloudWatchQuery +from date_utilities import DateUtilities + +# Configure logging at the module level. +logging.basicConfig( + level=logging.INFO, + format="%(asctime)s - %(levelname)s - %(filename)s:%(lineno)d - %(message)s", +) + + +class CloudWatchLogsQueryRunner: + def __init__(self): + """ + Initializes the CloudWatchLogsQueryRunner class by setting up date utilities + and creating a CloudWatch Logs client with retry configuration. + """ + self.date_utilities = DateUtilities() + self.cloudwatch_logs_client = self.create_cloudwatch_logs_client() + + def create_cloudwatch_logs_client(self): + """ + Creates and returns a CloudWatch Logs client with a specified retry configuration. + + :return: A CloudWatch Logs client instance. + :rtype: boto3.client + """ + try: + return boto3.client("logs", config=Config(retries={"max_attempts": 10})) + except Exception as e: + logging.error(f"Failed to create CloudWatch Logs client: {e}") + sys.exit(1) + + def fetch_environment_variables(self): + """ + Fetches and validates required environment variables for query start and end dates. + + :return: Tuple of query start date and end date as integers. + :rtype: tuple + :raises SystemExit: If required environment variables are missing or invalid. + """ + try: + query_start_date = int(os.environ["QUERY_START_DATE"]) + query_end_date = int(os.environ["QUERY_END_DATE"]) + except KeyError: + logging.error( + "Both QUERY_START_DATE and QUERY_END_DATE environment variables are required." + ) + sys.exit(1) + except ValueError as e: + logging.error(f"Error parsing date environment variables: {e}") + sys.exit(1) + + return query_start_date, query_end_date + + def convert_dates_to_iso8601(self, start_date, end_date): + """ + Converts UNIX timestamp dates to ISO 8601 format using DateUtilities. + + :param start_date: The start date in UNIX timestamp. + :type start_date: int + :param end_date: The end date in UNIX timestamp. + :type end_date: int + :return: Start and end dates in ISO 8601 format. + :rtype: tuple + """ + start_date_iso8601 = self.date_utilities.convert_unix_timestamp_to_iso8601( + start_date + ) + end_date_iso8601 = self.date_utilities.convert_unix_timestamp_to_iso8601( + end_date + ) + return start_date_iso8601, end_date_iso8601 + + def execute_query( + self, + start_date_iso8601, + end_date_iso8601, + log_group="/workflows/cloudwatch-logs/large-query", + ): + """ + Creates a CloudWatchQuery instance and executes the query with provided date range. + + :param start_date_iso8601: The start date in ISO 8601 format. + :type start_date_iso8601: str + :param end_date_iso8601: The end date in ISO 8601 format. + :type end_date_iso8601: str + :param log_group: Log group to search: "/workflows/cloudwatch-logs/large-query" + :type log_group: str + """ + cloudwatch_query = CloudWatchQuery( + [start_date_iso8601, end_date_iso8601], + ) + cloudwatch_query.query_logs((start_date_iso8601, end_date_iso8601)) + logging.info("Query executed successfully.") + logging.info( + f"Queries completed in {cloudwatch_query.query_duration} seconds. Total logs found: {len(cloudwatch_query.query_results)}" + ) + + +def main(): + """ + Main function to start a recursive CloudWatch logs query. + Fetches required environment variables, converts dates, and executes the query. + """ + logging.info("Starting a recursive CloudWatch logs query...") + runner = CloudWatchLogsQueryRunner() + query_start_date, query_end_date = runner.fetch_environment_variables() + start_date_iso8601 = DateUtilities.convert_unix_timestamp_to_iso8601( + query_start_date + ) + end_date_iso8601 = DateUtilities.convert_unix_timestamp_to_iso8601(query_end_date) + runner.execute_query(start_date_iso8601, end_date_iso8601) + + +if __name__ == "__main__": + main() diff --git a/python/example_code/cloudwatch-logs/scenarios/large-query/requirements.txt b/python/example_code/cloudwatch-logs/scenarios/large-query/requirements.txt new file mode 100644 index 00000000000..1db657b6b33 --- /dev/null +++ b/python/example_code/cloudwatch-logs/scenarios/large-query/requirements.txt @@ -0,0 +1 @@ +boto3 \ No newline at end of file diff --git a/python/example_code/cloudwatch-logs/scenarios/large-query/test/test_large_query.py b/python/example_code/cloudwatch-logs/scenarios/large-query/test/test_large_query.py new file mode 100644 index 00000000000..c62c07d4e99 --- /dev/null +++ b/python/example_code/cloudwatch-logs/scenarios/large-query/test/test_large_query.py @@ -0,0 +1,21 @@ +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 +import sys +from datetime import datetime, timedelta +import pytest + +sys.path.append("../") +from date_utilities import DateUtilities +from exec import CloudWatchLogsQueryRunner + +date_utility = DateUtilities() +runner = CloudWatchLogsQueryRunner() + + +@pytest.mark.integ +def test_run_successfully(): + now = datetime.utcnow() + ten_days_ago = now - timedelta(days=10) + query_start_date = date_utility.convert_datetime_to_iso8601(ten_days_ago) + query_end_date = date_utility.convert_datetime_to_iso8601(now) + runner.execute_query(query_start_date, query_end_date) diff --git a/workflows/cloudwatch_logs_large_query/README.md b/workflows/cloudwatch_logs_large_query/README.md index d4d7a47ed3f..0c1fe5f0b7c 100644 --- a/workflows/cloudwatch_logs_large_query/README.md +++ b/workflows/cloudwatch_logs_large_query/README.md @@ -2,7 +2,7 @@ ## Overview -This example shows how to use AWS SDKs to perform a query on CloudWatch logs and get more than the maximum number of 10,000 logs back. +This example shows how to use AWS SDKs to perform a query on Amazon CloudWatch Logs and get more than the maximum number of 10,000 logs back. The CloudWatch Logs API is capped at 10,000 records for requests that [read](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html) or [write](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_PutLogEvents.html). This example breaks down one query into multiple queries if more than the maximum number of records are returned from the query. @@ -48,6 +48,7 @@ A lot of logs are needed to make a robust example. If you happen to have a log g This example is implemented in the following languages: - [JavaScript](../../javascriptv3/example_code/cloudwatch-logs/scenarios/large-query/README.md) +- [Python](../../python/example_code/cloudwatch-logs/scenarios/large-query/README.md) ## Additional reading