Skip to content

Latest commit

 

History

History
237 lines (178 loc) · 12.5 KB

standalone-gc-eventing.md

File metadata and controls

237 lines (178 loc) · 12.5 KB
Raw

Standalone GC Eventing Design

Author: Sean Gillespie (@swgillespie) - 2017

This document aims to provide a specification for how a standalone GC fires events that can be collected by trace collectors. Such a feature is highly desirable for a standalone GC since it is the primary way that is used to reason about GC performance. Since a standalone GC is not permitted to link against the rest of the runtime, all communication between the runtime and the GC most pass through dynamically-dispatched interfaces.

Definitions

  • An event is some unit of information that the runtime can issue if requested. In general, this is used for lightweight tracing. Managed code can issue events using System.Diagnostics.Tracing.EventSource. Native code (i.e. the runtime) issues events by calling macros that delegate the issuing of events to autogenerated code that is generated to interface with the underlying event implementation. Events are only issued if they are turned on; the mechanism by which events are turned on is not in the scope of this document
  • The payload of an event is some amount of data that is delivered with the event itself. Its size may be variable. Most events that are fired by the runtime have a schema (predefined layout), but it is not a requirement.

Goals

The goal of this document is to describe a system that allows for the efficient firing of performance events by a standalone GC. This system must have three properties in order to be acceptable:

  1. It must be efficient to query whether or not a particular event is turned on. It is not acceptable to perform an indirection (i.e. cross the GC/EE interface boundary) in order to get this information.
  2. The cost of firing an event by a standalone GC should be comparable to the cost of firing an event without using a standalone GC.
  3. A standalone GC must be able to add new events without having to recompile the EE.

Querying Whether Events Are Enabled

It is not acceptable to perform an indirection when querying whether or not a requested event is enabled. Therefore, it follows that the GC must maintain some state about what events are currently enabled. Events are enabled through keywords and levels on a particular provider; a particular event is enabled if the provider to which the event belongs has the event's keyword and level enabled.

The GC fires events from two providers: the "main" provider, Microsoft-Windows-DotNETRuntime, and the "private" provider, Microsoft-Windows-DotNETRuntimePrivate. The GC must track the enabled keyword and level status of each provider separately. To accomplish this, the GC will contain a class with this signature:

enum GCEventProvider
{
    GCEventProvider_Default = 0,
    GCEventProvider_Private = 1
};


class GCEventStatus
{
public:
    // Returns true if the given keyword and level are enabled for the given provider,
    // false otherwise.
    static bool IsEnabled(GCEventProvider provider, int keyword, int level);

    // Enables events with the given keyword and level on the given provider.
    static bool Enable(GCEventProvider provider, int keyword, int level);

    // Disables events with the given keyword and level on the given provider.
    static bool Disable(GCEventProvider provider, int keyword, int level);
};

The GC will use GCEventStatus::IsEnabled to query whether or not a particular event is enabled. Whenever the EE observes a change in what keywords or levels are enabled for a particular provider, it must inform the GC of the change so that it can update GCEventStatus using Enable and Disable. The exact mechanism by which the EE observes a change in the event state is described further below. ("Getting Informed of Changes to Event State").

When the EE does observe a change in event state, it must inform the GC of these changes so that it can update its state accordingly. The following additions are made to the IGCHeap API surface area:

class IGCHeap
{
    // Enables or disables events with the given keyword and level on the default provider.
    virtual void ControlEvents(bool enable, int keyword, int level) = 0;

    // Enables or disables events with the given keyword and level on the private provider.
    virtual void ControlPrivateEvents(bool enable, int keyword, int level) = 0;
};

The currently enabled keywords and levels are encoded as bit vectors so that querying whether an event is enabled is efficient:

uint32_t enabledLevels[2];
uint32_t enabledKeywords[2];

bool GCEventStatus::IsEnabled(GCEventProvider provider, int keyword, int level)
{
    size_t index = static_cast<size_t>(provider);
    return (enabledLevels[index] & level) && (enabledKeywords[index] & keyword);
}

Firing Events

In order to fire an event, the GC will need to communicate with the EE in some way. The EE is ultimately responsible for routing the event to any appropriate subsystems (ETW, LTTNG, EventPipe) and the GC has no knowledge of what it is going to do with events that we give it.

Events are divided into two categories: known events and dynamic (or custom) events. Known events are known to the EE and correspond one-to-one to individual event types fired by the underlying platform loggers. Dynamic events are events not known to the EE; their use and description is below in the "Dynamic Events" section.

All events are fired through the IGCToCLREventSink interface, which is accessed through the IGCToCLR interface given to the GC on startup:

class IGCToCLREventSink
{

};

class IGCToCLR
{
    virtual IGCToCLREventSink* EventSink() = 0;
};

Every known event is fired through its own dedicated callback on IGCToCLREventSink. For example, the GCEnd event is fired through a callback like this:

class GCToCLREventSink : public IGCToCLREventSink { ... }

GCToCLREventSink::FireGCEnd(uint32_t count, uint16_t depth)
{
    // ...
}

GCTOCLREventSink::FireGCEnd is responsible for constructing and dispatching the event to platform loggers if the event is enabled. The principal advantage of having one callback per known event is that known events can reach into EE internals and add data to events that the GC otherwise would not be aware of. Concrete examples of this would be:

  • The addition of ClrInstanceId to the payload of many events fired by the GC
  • Getting human-readable type names for objects allocated on the heap
  • Correlating GCStart events with collections induced via ETW.

Defining and Firing Dynamic Events

It is useful for a standalone GC to be able to fire events that the EE was not previously aware of. For example, it is useful for a GC developer to add some event-based instrumentation to the GC, especially when testing new features and ensuring that they work as expected. Furthermore, it is desirable for GCs that are shipped from this repository (the "CLR GC") to interopate seamlessly with future versions of the .NET Core EE, which implies that it should be possible for the GC within this repository to add new events without having to recompile the runtime.

While it is possible for some eventing implementations to receive events that are created at runtime, not all eventing implementations (particularly LTTNG) are not flexible enough for this. In order to accommodate new events, another method is added to IGCToCLREventSink:

void IGCToCLREventSink::FireDynamicEvent(
    /* IN */ const char* eventName,
    /* IN */ void* payload,
    /* IN */ size_t payloadSize
);

A runtime implementing this callback will implement it by having a "catch-all" GC event whose schema is an arbitrary sequence of bytes. Tools can parse the (deliberately unspecified) binary format provided by GC events that use this mechanism in order to recover the data within the payload.

Dynamic events will by fired by the GC whenever developers want to add a new event but don't want to force users to get a new version of the runtime in order to utilize the new event.

Getting Informed of Changes to Event State

There are three mechanisms by which CoreCLR is able to log events: EventPipe, ETW, and LTTNG. When it comes to changing the state of events, EventPipe and ETW both allow users to attach callbacks that are invoked whenever events are enabled or disabled. For these two mechanisms, it is sufficient to use this callback mechanism to call IGCEventController::{Enable/Disable}Events from within such a callback in order to inform the GC of changes to tracing state.

LTTNG does not have such a mechanism. In order to observe changes in the eventing state, LTTNG must be polled periodically. Other eventing components in CoreCLR must already poll LTTNG for changes in the eventing state, so this design can utilize that same poll to inform the GC of changes.

Implementation Concerns

An implementation of this spec must take care not to perturb the existing GC code base too much. The GC fires events through the use of macros generated by the ETW message compiler and carefully mocked by code generation for the other platform logging implementations. The eventing scheme in this document will need to provide implementations for all eventing macros used by the GC (1).

These headers are often auto-generated. We will need to take care to re-use existing code generators if possible - after all, we do want it to be easy to add new events. It will likely be difficult to balance auto-generated code with the need for subtle custom modifications to event dispatch code.

Tools (specifically PerfView) will want to be enlightened about dynamic events fired by the runtime. The extent to which we want to make this experience nice is up to us, but we will most likely want to ship PerfView support for any dynamic event that we end up shipping with the CLR GC.

Concrete Example: Porting a single known event to IGCToCLREventSink

The following steps illustrate what needs to be done to bring a single event over to IGCToCLREventSink:

Two things are needed by the GC in order to fire an event: a way to determine if the event is on, and a way to fire the event. Events can be fired even if they aren't enabled (they occasionally are); the platform loggers will ignore the event if it is not enabled. However, the GC generally avoids doing expensive eventing-related operations if an event is not on.

The GC generally uses the ETW_EVENT_ENABLED macro to query whether an event is on. ETW_EVENT_ENABLED will be implemented in terms of GCEventStatus::IsEnabled above, so you will need to define appropriate macros in order for your event to work here. This will likely mean that you will need to define a macro that turns the name of your event into a pair of a level and a keyword, which will determine whether or not your event is enabled.

To fire the event, you can add your event's callback to IGCToCLREventSink:

class IGCToCLREventSink
{
    virtual void FireYourEvent(
        /* your event arguments here... */
    ) = 0;
};

You can then define the FireYourEvent macro in src/gc/env/etmdummy.h to point to your new method on IGCToCLREventSink Your implementation of FireYourEvent in the EE will need to calculate any EE-specific data (e.g. ClrInstanceId) and then forward the event arguments onto the platform logger, which can be done with the Fire macros that the EE has access to (that we implemented for the GC).

Concrete Example: Adding a dynamic event

The following steps illustrate what needs to be done to add a new dynamic event:

There are two things that need to be written: the ETW_EVENT_ENABLED support for your new event and the macro responsible for firing your event. ETW_EVENT_ENABLED can be implemented in the same manner as a known event, by introducing a macro for your event name that expands to your event's level and keyword.

Firing of a dynamic event ultimately must call FireDynamicEvent with an event name and a serialized payload. Therefore, it is the responsibility of the GC to format an event's payload into a binary format. It is ultimately up to you(1) to write a function that serializes your event's arguments into a buffer and sends the buffer to FireDynamicEvent. This code in turn can be wired up to the GC codebase by defining a new FireMyCustomEvent macro whose arguments are forwarded onto the event serialization function.

  1. C++ has the ability to auto-generate large swaths of this code. The implementation of this spec will provide a series of composable helper functions that automate the serialization of arguments so that they do not have to be written by the developer adding a new event.