update submission received event and delete v1 azure events (#15990)

* remove ReportRouteEvent.kt

* remove events and update md

* update to include queryParameter

* update unit test

* transition to config based filtering and allow one query parameter to have multiple values

* account for empty parameter filter list

* transition to application.yml

prime-router/docs/observability/azure-events.md

@@ -45,115 +45,6 @@ class MyService(
 }
 ```
 
-Under the hood, it will serialize your event class and push the event
-to the configured Microsoft AppInsights instance.
-
-## Event Glossery
-
-### ReportCreatedEvent
-This event is emitted by the convert step when a report is successfully translated into a FHIR bundle.
-- reportId
-    - The ID assigned to the created report
-- topic
-    - The topic of the created report
-
-
-### ReportAcceptedEvent
-This event is emitted by the destination filter step, _before_ any filters are evaluated
-- reportId
-  - The report ID from the preceding function (convert step)
-- submittedReportId
-  - The report ID submitted by the sender
-- topic
-  - The topic of the report
-- sender
-  - The full sender name
-- observations
-  - A list of observations each containing a list of its mapped conditions
-- bundleSize
-  - Length of the bundle JSON string
-- messageId
-  - From the bundle.identifier value and system. If ingested as HL7 this comes from MSH-10
-
-
-### ReportNotRoutedEvent
-This is event is emitted by the destination filter step if a bundle not routed to any receivers.
-
-- reportId
-    - The ID of the empty report that terminated this lineage
-- parentReportId
-    - The report ID from the preceding function (convert step)
-- submittedReportId
-    - The report ID submitted by the sender
-- topic
-    - The topic of the report
-- sender
-    - The full sender name
-- bundleSize
-    - Length of the bundle JSON string
-- failingFilters
-  - A list of all the filters that failed causing this report not the be routed
-- messageId
-    - From the bundle.identifier value and system. If ingested as HL7 this comes from MSH-10
-
-
-### ReportRouteEvent
-This event is emitted by the receiver filter step, _after_ all filters have passed and a report has been
-routed to a receiver. Many `ReportRouteEvent` can correspond to a `ReportAcceptedEvent` and can be "joined" on:
-
-`ReportAcceptedEvent.reportId == ReportRouteEvent.parentReportId`
-
-- reportId
-  - The ID of the report routed to the receiver
-- parentReportId
-  - The report ID from the preceding function (destination filter step)
-- submittedReportId
-  - The report ID submitted by the sender
-- topic
-  - The topic of the report
-- sender
-  - The full sender name
-- receiver
-  - The full receiver name. (deprecated: When a report does not get routed to a receiver this value will be `"null"`)
-- observations
-  - A list of observations each containing a list of its mapped conditions
-- (deprecated) originalObservations
-  - (deprecated) A list of observations in the originally submitted report, before any filters were run
-- filteredObservations
-  - A list of observations that were filtered from the bundle during filtering
-- bundleSize
-  - Length of the bundle JSON string
-- messageId
-  - From the bundle.identifier value and system. If ingested as HL7 this comes from MSH-10
-
-
-### ReceiverFilterFailedEvent
-This event is emitted by the receiver filter step if a bundle fails a receiver filter.
-
-- reportId
-    - The ID of the empty report that terminated this lineage
-- parentReportId
-    - The report ID from the preceding function (destination filter step)
-- submittedReportId
-    - The report ID submitted by the sender
-- topic
-    - The topic of the report
-- sender
-    - The full sender name
-- receiver
-    - The full receiver name.
-- observations
-    - A list of observations each containing a list of its mapped conditions
-- failingFilters
-    - A list of all the filters that failed for this report
-- failingFilterType
-    - The type of filter that failed this report
-- bundleSize
-    - Length of the bundle JSON string
-- messageId
-    - From the bundle.identifier value and system. If ingested as HL7 this comes from MSH-10
-
-
 ## How to query for events
 
 Events that are pushed to Azure can be found in the `customEvents` table in the log explorer. The properties defined in
@@ -176,24 +67,24 @@ customEvents
 ### Distinct senders
 ```
 customEvents
-| where name == "ReportAcceptedEvent"
-| extend sender = tostring(customDimensions.sender)
-| distinct sender
+| where name == "REPORT_RECEIVED"
+| extend senderName = tostring(parse_json(tostring(customDimensions.params)).senderName)
+| distinct senderName
 ```
 
 ### Get report count sent by sender
 ```
 customEvents
-| where name == "ReportAcceptedEvent"
-| extend sender = tostring(customDimensions.sender)
-| summarize count() by sender 
+| where name == "REPORT_RECEIVED"
+| extend senderName = tostring(parse_json(tostring(customDimensions.params)).senderName)
+| summarize count() by senderName 
 | order by count_
 ```
 
 ### Get report count sent by topic
 ```
 customEvents
-| where name == "ReportAcceptedEvent"
+| where name == "REPORT_RECEIVED"
 | extend topic = tostring(customDimensions.topic)
 | summarize count() by topic 
 | order by count_
@@ -202,55 +93,43 @@ customEvents
 ### Get reportable conditions count for all reports sent to Report Stream
 ```
 customEvents
-| where name == "ReportAcceptedEvent"
-| extend observations = parse_json(tostring(customDimensions.observations))
-| mv-expand observations
-| extend conditions = parse_json(tostring(observations.conditions))
-| mv-expand conditions
-| extend conditionDisplay = tostring(conditions.display)
+| where name == "ITEM_ACCEPTED"
+| extend conditionDisplay = tostring(parse_json(tostring(parse_json(tostring(parse_json(tostring(parse_json(tostring(parse_json(tostring(customDimensions.params)).bundleDigest)).observationSummaries))[0].testSummary))[0].conditions))[0].display)
 | summarize count() by conditionDisplay
 | order by count_
 ```
 
 ### Distinct receivers
 ```
 customEvents
-| where name == "ReportRouteEvent"
-| extend receiver = tostring(customDimensions.receiver)
-| where receiver != "null"
-| distinct receiver
+| where name == "ITEM_ROUTED"
+| extend receiverName = tostring(parse_json(tostring(customDimensions.params)).receiverName)
+| distinct receiverName
 ```
 
 ### Get report count routed to a receiver
 ```
 customEvents
-| where name == "ReportRouteEvent"
-| extend receiver = tostring(customDimensions.receiver)
-| where receiver != "null"
-| summarize count() by receiver 
+| where name == "ITEM_ROUTED"
+| extend receiverName = tostring(parse_json(tostring(customDimensions.params)).receiverName)
+| summarize count() by receiverName 
 | order by count_
 ```
 
 ### Get report count routed by topic
 ```
 customEvents
-| where name == "ReportRouteEvent"
-| extend topic = tostring(customDimensions.topic), receiver = tostring(customDimensions.receiver)
-| where receiver != "null"
+| where name == "ITEM_ROUTED"
+| extend topic = tostring(customDimensions.topic)
 | summarize count() by topic 
 | order by count_
 ```
 
 ### Get reportable conditions count for all reports routed to receivers
 ```
 customEvents
-| where name == "ReportRouteEvent"
-| extend observations = parse_json(tostring(customDimensions.observations)), receiver = tostring(customDimensions.receiver)
-| where receiver != "null"
-| mv-expand observations
-| extend conditions = parse_json(tostring(observations.conditions))
-| mv-expand conditions
-| extend conditionDisplay = tostring(conditions.display)
+| where name == "ITEM_ROUTED"
+| extend conditionDisplay = tostring(parse_json(tostring(parse_json(tostring(parse_json(tostring(parse_json(tostring(parse_json(tostring(customDimensions.params)).bundleDigest)).observationSummaries))[0].testSummary))[0].conditions))[0].display)
 | summarize count() by conditionDisplay
 | order by count_
 ```

prime-router/docs/universal-pipeline/destination-filter.md

@@ -186,10 +186,10 @@ This filter will log messages to the console when:
 
 This step emits one of two events below _once_ each time it runs.
 
-| Event                                                                         | Trigger                                                          |
-|-------------------------------------------------------------------------------|------------------------------------------------------------------|
-| [ReportAcceptedEvent](../observability/azure-events.md#reportacceptedevent)   | when a report is received by this step (precludes all filtering) |
-| [ReportNotRoutedEvent](../observability/azure-events.md#reportnotroutedevent) | when a report is not valid for any receivers                     |
+| Event           | Trigger                                                          |
+|-----------------|------------------------------------------------------------------|
+| ITEM_ROUTED     | when a report is received by this step (precludes all filtering) |
+| ITEM_NOT_ROUTED | when a report is not valid for any receivers                     |
 
 ## Retries
 

prime-router/docs/universal-pipeline/receiver-filter.md

@@ -294,8 +294,7 @@ This step emits one of the below events _per receiver_ each time it runs.
 
 | Event                                                                                   | Trigger                                                    |
 |-----------------------------------------------------------------------------------------|------------------------------------------------------------|
-| [ReportRouteEvent](../observability/azure-events.md#reportrouteevent)                   | When a report is routed to a receiver (all filters passed) |
-| [ReceiverFilterFailedEvent](../observability/azure-events.md#receiverfilterfailedevent) | When a report fails receiver filters                       |
+| ITEM_FILTER_FAILED | When a report fails receiver filters                       |
 
 ## Retries
 

submissions/build.gradle.kts

@@ -30,6 +30,7 @@ dependencies {
     testImplementation("org.jetbrains.kotlin:kotlin-test-junit5")
     testImplementation("org.mockito.kotlin:mockito-kotlin:5.4.0")
     testImplementation("org.apache.commons:commons-compress:1.27.1")
+    testImplementation("org.springframework.security:spring-security-test")
     testRuntimeOnly("org.junit.platform:junit-platform-launcher")
     implementation("org.jetbrains.kotlinx:kotlinx-coroutines-reactor:1.9.0")
     implementation(project(":shared"))

submissions/src/main/kotlin/gov/cdc/prime/reportstream/submissions/SubmissionReceivedEvent.kt

@@ -8,9 +8,17 @@ data class SubmissionReceivedEvent(
     val reportId: UUID,
     val parentReportId: UUID,
     val rootReportId: UUID,
-    val headers: Map<String, String>,
-    val sender: String,
-    val senderIP: String,
-    val fileSize: String,
+    val requestParameters: SubmissionDetails,
+    val method: String,
+    val url: String,
+    val senderName: String,
+    val senderIp: String,
+    val fileLength: String,
     val blobUrl: String,
+    val pipelineStepName: String,
+)
+
+data class SubmissionDetails(
+    val headers: Map<String, String>,
+    val queryParameters: Map<String, List<String>>,
 )