Logs sent to OTEL Collector are not being recorded

[sergiojrdotnet]

Bug description

I am currently working on forwarding access logs from Azure Application Gateway to Signoz. To achieve this, I have configured an Azure Event Hub and an OTEL Collector to receive the events and export them to Signoz's Collector.

However, despite the exporter not showing any errors, the logs are not being stored in the logs_v2 table in ClickHouse, nor are they visible in the frontend.

To troubleshoot, I performed some manual tests by taking one of the requests and sending it directly to the endpoint. The behavior remains the same:

Expected behavior

Log records should be stored in ClickHouse and made visible on the frontend.

How to reproduce

1. Update timeUnixNano field to current time and send the following request to OTLP HTTP endpoint (e.g., http://hostname:4318/v1/logs)

{
    "resourceLogs": [
        {
            "resource": {
                "attributes": [
                    {
                        "key": "azure.resource.id",
                        "value": {
                            "stringValue": "/SUBSCRIPTIONS/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXX-XXXXXX/RESOURCEGROUPS/MY-RESOURCE-GROUP/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/MY-GATEWAY"
                        }
                    }
                ]
            },
            "scopeLogs": [
                {
                    "scope": {
                        "name": "github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/azure",
                        "version": "0.114.0"
                    },
                    "logRecords": [
                        {
                            "timeUnixNano": "1732668380000000000",
                            "body": {},
                            "attributes": [
                                {
                                    "key": "azure.category",
                                    "value": {
                                        "stringValue": "ApplicationGatewayAccessLog"
                                    }
                                },
                                {
                                    "key": "azure.operation.name",
                                    "value": {
                                        "stringValue": "ApplicationGatewayAccess"
                                    }
                                },
                                {
                                    "key": "azure.properties",
                                    "value": {
                                        "kvlistValue": {
                                            "values": [
                                                {
                                                    "key": "serverStatus",
                                                    "value": {
                                                        "stringValue": "200"
                                                    }
                                                },
                                                {
                                                    "key": "host",
                                                    "value": {
                                                        "stringValue": "contoso.com"
                                                    }
                                                },
                                                {
                                                    "key": "sslCipher",
                                                    "value": {
                                                        "stringValue": ""
                                                    }
                                                },
                                                {
                                                    "key": "sslProtocol",
                                                    "value": {
                                                        "stringValue": ""
                                                    }
                                                },
                                                {
                                                    "key": "clientPort",
                                                    "value": {
                                                        "doubleValue": 48344
                                                    }
                                                },
                                                {
                                                    "key": "requestUri",
                                                    "value": {
                                                        "stringValue": "/v1/the-endpoint"
                                                    }
                                                },
                                                {
                                                    "key": "requestQuery",
                                                    "value": {
                                                        "stringValue": ""
                                                    }
                                                },
                                                {
                                                    "key": "httpStatus",
                                                    "value": {
                                                        "doubleValue": 200
                                                    }
                                                },
                                                {
                                                    "key": "WAFPolicyID",
                                                    "value": {
                                                        "stringValue": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxx-xxxxxx/resourceGroups/my-resource-group/providers/Microsoft.Network/ApplicationGatewayWebApplicationFirewallPolicies/waf-policy"
                                                    }
                                                },
                                                {
                                                    "key": "sslEnabled",
                                                    "value": {
                                                        "stringValue": ""
                                                    }
                                                },
                                                {
                                                    "key": "sslClientVerify",
                                                    "value": {
                                                        "stringValue": ""
                                                    }
                                                },
                                                {
                                                    "key": "originalRequestUriWithArgs",
                                                    "value": {
                                                        "stringValue": "/v1/the-endpoint"
                                                    }
                                                },
                                                {
                                                    "key": "timeTaken",
                                                    "value": {
                                                        "doubleValue": 0.008
                                                    }
                                                },
                                                {
                                                    "key": "receivedBytes",
                                                    "value": {
                                                        "doubleValue": 249
                                                    }
                                                },
                                                {
                                                    "key": "clientResponseTime",
                                                    "value": {
                                                        "doubleValue": 0
                                                    }
                                                },
                                                {
                                                    "key": "WAFEvaluationTime",
                                                    "value": {
                                                        "stringValue": "0.004"
                                                    }
                                                },
                                                {
                                                    "key": "transactionId",
                                                    "value": {
                                                        "stringValue": "c9dd4ffaf1d013d5358bddbfdaf80a4e"
                                                    }
                                                },
                                                {
                                                    "key": "clientIP",
                                                    "value": {
                                                        "stringValue": "186.209.113.137"
                                                    }
                                                },
                                                {
                                                    "key": "sslClientCertificateIssuerName",
                                                    "value": {
                                                        "stringValue": ""
                                                    }
                                                },
                                                {
                                                    "key": "serverRouted",
                                                    "value": {
                                                        "stringValue": "10.4.1.4:80"
                                                    }
                                                },
                                                {
                                                    "key": "serverResponseLatency",
                                                    "value": {
                                                        "stringValue": "0.004"
                                                    }
                                                },
                                                {
                                                    "key": "httpMethod",
                                                    "value": {
                                                        "stringValue": "GET"
                                                    }
                                                },
                                                {
                                                    "key": "sentBytes",
                                                    "value": {
                                                        "doubleValue": 3534
                                                    }
                                                },
                                                {
                                                    "key": "noOfConnectionRequests",
                                                    "value": {
                                                        "doubleValue": 1
                                                    }
                                                },
                                                {
                                                    "key": "WAFMode",
                                                    "value": {
                                                        "stringValue": "Prevention"
                                                    }
                                                },
                                                {
                                                    "key": "originalHost",
                                                    "value": {
                                                        "stringValue": "contoso.com"
                                                    }
                                                },
                                                {
                                                    "key": "instanceId",
                                                    "value": {
                                                        "stringValue": "appgw_0"
                                                    }
                                                },
                                                {
                                                    "key": "error_info",
                                                    "value": {
                                                        "stringValue": "ERRORINFO_NO_ERROR"
                                                    }
                                                },
                                                {
                                                    "key": "httpVersion",
                                                    "value": {
                                                        "stringValue": "HTTP/1.1"
                                                    }
                                                },
                                                {
                                                    "key": "sslClientCertificateFingerprint",
                                                    "value": {
                                                        "stringValue": ""
                                                    }
                                                },
                                                {
                                                    "key": "upstreamSourcePort",
                                                    "value": {
                                                        "stringValue": "39116"
                                                    }
                                                },
                                                {
                                                    "key": "userAgent",
                                                    "value": {
                                                        "stringValue": ""
                                                    }
                                                },
                                                {
                                                    "key": "contentType",
                                                    "value": {
                                                        "stringValue": "text/json"
                                                    }
                                                },
                                                {
                                                    "key": "connectionSerialNumber",
                                                    "value": {
                                                        "doubleValue": 12345678
                                                    }
                                                }
                                            ]
                                        }
                                    }
                                },
                                {
                                    "key": "cloud.provider",
                                    "value": {
                                        "stringValue": "azure"
                                    }
                                }
                            ],
                            "traceId": "",
                            "spanId": ""
                        }
                    ]
                }
            ]
        }
    ]
}

The response indicates 200 OK

Version information

• Signoz version: v0.59.0 (Self Hosted - Docker)

Additional context

OTEL Collector Config

receivers:
  azureeventhub:
    connection: <connection>

processors:
  batch:

exporters:
  debug:
  otlphttp:
    endpoint: http://<hostname>:4318

service:

  pipelines:

    logs:
      receivers: [azureeventhub]
      processors: [batch]
      exporters: [debug,otlphttp]

The behavior is the same when using the gRPC endpoint, which I actually plan to set as the default once the tests are completed.

By analyzing the content of other tables in ClickHouse, I can see that some information has been created, such as the resource key azure.resource.id in logs_resource_keys and the attribute key azure.category in logs_attribute_keys. Indicating that the records are at least being received.

[welcome]

Thanks for opening this issue. A team member should give feedback soon. In the meantime, feel free to check out the contributing guidelines.

[pranay01]

@sergiojrdotnet Are you seeing this issue only with access logs from Azure Application Gateway or you are not able to see any data from Azure Event Hub receiver?

[sergiojrdotnet]

@pranay01 I can see the data normally from Azure Event Hub, and I’m also able to export the data from my OTEL collector to other platforms (e.g., Seq) using OTLP

[sergiojrdotnet]

@Calm-Rock Is there any procedure or log from SigNoz itself that I could use to investigate if there's any exception in the ingestion? The response currently returned by the API doesn’t indicate any error.