Skip to content

Latest commit

 

History

History
113 lines (88 loc) · 8.2 KB

Debugging.md

File metadata and controls

113 lines (88 loc) · 8.2 KB
Raw

Debugging

This article details how to troubleshoot and debug your deployed instance of the IoMT FHIR Connector for Azure. Below will cover troubleshooting your deployed instance as well as how to debug locally.

Troubleshooting Deployed Services

An instance of Application Insights is deployed as part of the IoMT FHIR Connector for Azure. From here you can see errors and telemetry for logical components of the connector. In addition to the information contained in Application Insights each component deployed (Event Hub & Stream Analytics) has its own telemetry you can use as well. Some important information includes the number of messages received and sent on the Event Hubs and the watermark delay along with input and output events of the Stream Analytic job.

Telemetry Collected

All telemetry below can be viewed in Application Insights. Select the metrics section. The telemetry documented will fall under the custom category.

  • MeasurementIngestionLatency: The difference between the ingestion time on the event hub end point and received by the FHIR conversion function. Measured in milliseconds.

  • MeasurementGroup: The number of unique groupings of measurements across type, device, patient, and configured time period generated by the FHIR conversion function. Should directly correlate to the number of observations being created or updated.

  • Measurement: The number of raw value readings received by the FHIR conversion function.

  • DeviceEvent: The number of raw events received by the connector prior to any normalization.

  • NormalizedEvent: The number of mapped normalized values outputted from the normalization step.

  • DeviceEventProcessingLatency: The time between an event's ingestion and being processed for normalization. A large value here indicates a backlog of events being processed for normalization. Measured in milliseconds.

  • Handled Exceptions: The number of handled exceptions for a given category. In order to not stall the pipeline certain non recoverable errors are categorized and then recorded as telemetry. The current exceptions handled are:

    Exception Description
    PatientDeviceMismatchException In Create mode, if both the device and patient exist but the device is linked to another patient this exception will be thrown.
    ResourceIdentityNotDefinedException Thrown when a resource identifier is expected and required for the current mode (Create, Lookup, Lookup w/Encounter). The actual telemetry event will be specific to the type of resource identifier missing (i.e. Patient, Device, or Encounter).
    NotSupportedException Typically thrown if their isn't a supported template or value type during processing.
    FhirResourceNotFoundException Thrown when the identifier provided in the message isn't found. The actual telemetry recorded will be specific the type of resource not found (i.e. Patient, Device, or Encounter).

Common Issues

  • Messages are being sent but I don't see any output on the normalized data Event Hub.

    First, ensure you have properly deployed the configuration with your template(s) into the Storage Blob template container with the name devicecontent.json. If there is a problem loading templates the issue will surface in Application Insights as ArgumentNullExceptions.

    Another potential cause is your device content templates aren't matching the payloads being sent. If possible you can try debugging locally. Otherwise you can try retrieving messages from the device data Event Hub and investigating the payload. The are example unit tests in the solution you can modify to match your template and payload so you can check the template is being applied as intended.

  • Messages are processed and job output are being sent to the FHIR conversion but I don't see any observations created.

    Check that you have properly created a device resource that is linked to a patient resource in the FHIR server. The device id extracted by the template should match the device identifier on device resource in FHIR. If not setup correctly you will see PatientNotFoundException or DeviceNotFoundException metrics in Applications Insights.

Run and Debug Locally

The core logic of the IoMT FHIR Connector for Azure is handled by two dotnet applications. You can run the projects locally but the projects have dependencies on resources deployed in Azure (see below).

  • Microsoft.Health.Fhir.Ingest.Console.Normalization

    • Azure Event Hub Namespace
    • 2 Azure Event Hubs (1 for device data and 1 for normalized data)
    • Storage Account

  • Microsoft.Health.Fhir.Ingest.Console.FhirTransformation

    • Azure Event Hub Namespace
    • 1 Azure Event Hubs (for normalized data)
    • Storage Account
    • Health Data Services workspace
    • Fhir Service

  • Ensure the project that you want to debug is set as the startup project in the solution.

  • Upload your device content and/or FHIR mapping to the Storage Account.

  • Complete the local.settings.json file in the root of the project.

    • NOTE - If settings fail to load make sure the Copy to Output Directory property of the local.settings.json is set to Copy always or Copy if newer.

Example Microsoft.Health.Fhir.Ingest.Console.Normalization local.settings.json

{
  "EventBatching:FlushTimespan": 10,
  "EventBatching:MaxEvents": 1,
  "Checkpoint:CheckpointBatchCount": 1,
  "Checkpoint:BlobPrefix": "devicedata",
  "CheckpointStorage:AuthenticationType": "ManagedIdentity",
  "CheckpointStorage:BlobStorageContainerUri": "<<Storage Container Checkpoint URI>>",
  "TemplateStorage:AuthenticationType": "ManagedIdentity",
  "TemplateStorage:BlobStorageContainerUri": "<<Storage Container Template URI>>",
  "InputEventHub:AuthenticationType": "ManagedIdentity",
  "InputEventHub:EventHubNamespaceFQDN": "<<Device data Event Hub Namespace FQDN>>",
  "InputEventHub:EventHubConsumerGroup": "$Default",
  "InputEventHub:EventHubName": "<<Device data Event Hub Name>>",
  "NormalizationEventHub:AuthenticationType": "ManagedIdentity",
  "NormalizationEventHub:EventHubNamespaceFQDN": "<<Normalized data Event Hub Namespace FQDN>>",
  "NormalizationEventHub:EventHubConsumerGroup": "$Default",
  "NormalizationEventHub:EventHubName": "<<Normalized data Event Hub Name>>",
  "Template:DeviceContent": "devicecontent.json"
}

Example Microsoft.Health.Fhir.Ingest.Console.FhirTransformation local.settings.json

{
  "EventBatching:FlushTimespan": 10,
  "EventBatching:MaxEvents": 1,
  "Checkpoint:CheckpointBatchCount": 1,
  "Checkpoint:BlobPrefix": "normalizeddata",
  "CheckpointStorage:AuthenticationType": "ManagedIdentity",
  "CheckpointStorage:BlobStorageContainerUri": "<<Storage Container Checkpoint URI>>",
  "TemplateStorage:AuthenticationType": "ManagedIdentity",
  "TemplateStorage:BlobStorageContainerUri": "<<Storage Container Template URI>>",
  "FhirClient:UseManagedIdentityCredential": "true",
  "FhirService:Url": "<<FHIR service URL>>",
  "NormalizationEventHub:AuthenticationType": "ManagedIdentity",
  "NormalizationEventHub:EventHubNamespaceFQDN": "<<Normalized data Event Hub Namespace FQDN>>",
  "NormalizationEventHub:EventHubConsumerGroup": "$Default",
  "NormalizationEventHub:EventHubName": "<<Normalized data Event Hub Name>>",
  "ResourceIdentity:ResourceIdentityServiceType": "Create",
  "ResourceIdentity:DefaultDeviceIdentifierSystem": "",
  "Template:FhirMapping": "fhirmapping.json"
}

Once you have completed the prerequisites you can run the projects.

Microsoft.Health.Fhir.Ingest.Console.Normalization will read events from the InputEventHub defined in the local.settings.json and output normalized data to the NormalizationEventHub defined in the local.settings.json.

Microsoft.Health.Fhir.Ingest.Console.FhirTransformation will read events from the NormalizationEventHub defined in the local.settings.json and write FHIR resources to the FhirService defined in the local.settings.json.