AmazonCloudWatchLogsClient.GetLogEvents does not return any event

[gerdb42]

Describe the bug

I use the following code to retrieve LogEvents from CloudWatch (shortened for brevity):

var request = new GetLogEventsRequest
{
	LogGroupName = <valid log group name>,
	LogStreamName = <valid log stream name>
};

var response = await amazonCloudWatchLogsClient.GetLogEventsAsync(request);

response.Events.ForEach(t =>
{
    Console.Write ($"{t.Timestamp}: {t.Message}");
});

However, response.Events always contains 0 items. And despite response.NextForwardToken being non-empty, the request never returns any events and may be repeated forever. And I know the connection works and the log stream name is valid, because the name comes from an enumeration of log streams done just before. And the log steam contains events since I can see them in CloudWatch console.

Expected Behavior

The returned response contains a reasonable number of events together with a non-null NextForwardToken if more events are available or NextForwardToken being null if there are no more events to be retrieved.

Current Behavior

In the returned response, the Events collection contains 0 elements and the NextForwardToken is set. Repeating the request with NextToken set returns 0 events again.

Reproduction Steps

See code sample above

Possible Solution

No response

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

AWSSDK.CloudWatchLogs 3.7.304.8

Targeted .NET Platform

.NET 8

Operating System and version

Windows 10

[dscpinheiro]

The pagination for this operation is odd, if I try aws logs get-log-events --log-group-name "my-log-group" --log-stream-name "my-log-stream" from the CLI (which is the exact example from the CloudWatch documentation), I get no events either.

I checked the console and it sets up an extra property in GetLogEventsRequest:

 startFromHead
    If the value is true, the earliest log events are returned first. If the value is false, the latest log events are returned first. The default value is false.
    If you are using a previous nextForwardToken value as the nextToken in this operation, you must specify true for startFromHead.
    Type: Boolean
    Required: No

Another interesting thing is that the NextForwardToken won't be null after you reached the end of the stream:

 nextForwardToken
    The token for the next set of items in the forward direction. The token expires after 24 hours. If you have reached the end of the stream, it returns the same token you passed in.
    Type: String

With the following snippet, I can get all the events but we'll follow up internally as this isn't a great customer experience... And this behavior also means our built-in paginator doesn't work as expected.

var logs = new AmazonCloudWatchLogsClient();
var request = new GetLogEventsRequest
{
    LogGroupName = "my-log-group",
    LogStreamName = "my-log-stream",
    Limit = 200, // Default console limit, but can be higher
    StartFromHead = true
};

while (true)
{
    var response = await logs.GetLogEventsAsync(request);
    if (response.NextForwardToken == request.NextToken)
    {
        break;
    }

    Console.WriteLine(response.Events.Count); // Outputs greater than 0
    request.NextToken = response.NextForwardToken;
}

[gerdb42]

Yes, setting StartFromHead to true does the Trick! So this means we can only get events in ascending order (oldest first). But I still get several results with 0 events before NextForwardToken equals the previous token.

It would be nice if StartFromHead would work as described and NextForwardToken was set to null upon delivery of the last chunk of events (to avoid unnecessary roundtrips).

[peterrsongg]

@gerdb42 If tailing a log is what you want, CloudWatchLogs added a StartLiveTail API recently, and there is an example of how to configure it here. I helped write the example so I know that the code in there works. It might feel more natural than the current API you're using.

Example in docs

[dscpinheiro]

Today's release of the AWSSDK.CloudWatchLogs package (3.7.305.24) includes a fix so that the SDK built-in paginator works as expected with this API. That means you can use the following snippet (instead of the example I sent earlier):

var logs = new AmazonCloudWatchLogsClient();
var getLogsRequest = new GetLogEventsRequest
{
    LogGroupName = "my-log-group",
    LogStreamName = "my-log-stream",
    StartFromHead = true,
};

var paginator = logs.Paginators.GetLogEvents(getLogsRequest);
await foreach (var logEvent in paginator.Events)
{
   Console.WriteLine(logEvent.Message);
}

And as Peter suggested, check out the StartLiveTail API as it may be more applicable to your use case.

[github-actions]

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.