[BUG] Error when creating a dector. An index and alias that doesn't exist is referrenced in the error message returned from the API endpoint.

[Outrun207]

What is the bug?
I am receiving the following error message:

{"level":"INFO","location":"create_analytics:130","message":"HTTP Status Code: 500","timestamp":"2023-10-25 18:38:24,972+0000","service":"service_undefined","xray_trace_id":"1-6539609d-29663438314d189b5ed560a4"}

{"level":"INFO","location":"create_analytics:131","message":{"error":{"root_cause":[{"type":"security_analytics_exception","reason":"Failed to get write index for queryIndex alias:.opensearch-sap-cloudtrail-detectors-queries"}],"type":"security_analytics_exception","reason":"Failed to get write index for queryIndex alias:.opensearch-sap-cloudtrail-detectors-queries","caused_by":{"type":"exception","reason":"org.opensearch.alerting.util.AlertingException: Failed to get write index for queryIndex alias:.opensearch-sap-cloudtrail-detectors-queries"}},"status":500},"timestamp":"2023-10-25 18:38:24,972+0000","service":"service_undefined","xray_trace_id":"1-6539609d-29663438314d189b5ed560a4"}

How can one reproduce the bug?
Steps to reproduce the behavior:

Code:

def create_analytics(os_endpoint_url, os_pass):
    analytic_dir = "os_configuration_templates/security_analytics"
    url = f"{os_endpoint_url}/_plugins/_security_analytics/detectors"
    for analyticBody in os.listdir(analytic_dir):
        if analyticBody.endswith(".json"):
            try:
                with open(os.path.join(analytic_dir, analyticBody)) as f:
                    analytic_data = f.read()

                response = requests.post(
                    url,
                    data=analytic_data,
                    headers={"Content-Type": "application/json"},
                    auth=(os.environ["MASTER_USER_NAME"], os_pass),
                    timeout=30,
                    cookies=get_cookie(os_endpoint_url, os_pass),
                )
                logger.info(f"Adding {analyticBody} as {url}")
                if response.status_code == 200 or 201:
                    logger.info(
                        f"Analytic '{analyticBody}' uploaded successfully.",
                    )
                    logger.info(f"HTTP Status Code: {response.status_code}")
                    logger.info(response.text)

                else:
                    logger.info(
                        f"Failed to configure analytic '{analyticBody}'.",
                    )
                    logger.info(f"HTTP Status Code: {response.status_code}")
                    logger.info(response.text)
            except (OSError, requests.exceptions.RequestException) as e:
                logger.info(
                    f"Error configuring '{analyticBody}': {e}",
                )

Detector definition json:

{
   "type": "detector",
   "detector_type": "cloudtrail",
   "name": "CloudTrail Security Analytics Detector",
   "enabled": true,
   "createdBy": "",
   "schedule": { "period": { "interval": 6, "unit": "MINUTES" } },
   "inputs": [
       {
           "detector_input": {
               "description": "",
               "indices": ["ocsf-3005*"],
               "pre_packaged_rules": [
                   { "id": "abae8fec-57bd-4f87-aff6-6e3db989843d" },
                   { "id": "b056de1a-6e6e-4e40-a67e-97c9808cf41b" },
                   { "id": "25cb1ba1-8a19-4a23-a198-d252664c8cef" },
                   { "id": "97fbabf8-8e1b-47a2-b8d5-a418d2b95b3d" },
                   { "id": "78b3756a-7804-4ef7-8555-7b9024a02d2d" },
                   { "id": "3940b5f1-3f46-44aa-b746-ebe615b879e0" },
                   { "id": "8ad1600d-e9dc-4251-b0ee-a65268f29add" },
                   { "id": "0a5177f4-6ca9-44c2-aacf-d3f3d8b6e4d2" },
                   { "id": "e9c14b23-47e2-4a8b-8a63-d36618e33d70" },
                   { "id": "7c797da2-9cf2-4523-ba64-33b06339f0cc" },
                   { "id": "33d50d03-20ec-4b74-a74e-1e65a38af1c0" },
                   { "id": "97fbabf8-8e1b-47a2-b7d5-a418d2b95e3d" },
                   { "id": "055fb148-60f8-462d-ad16-26926ce050f1" },
                   { "id": "b45ab1d2-712f-4f01-a751-df3826969807" },
                   { "id": "905d389b-b853-46d0-9d3d-dea0d3a3cd49" },
                   { "id": "a607e1fe-74bf-4440-a3ec-b059b9103157" },
                   { "id": "4990c2e3-f4b8-45e3-bc3c-30b14ff0ed26" },
                   { "id": "d914951b-52c8-485f-875e-86abab710c0b" },
                   { "id": "6e61ee20-ce00-4f8d-8aee-bedd8216f7e3" },
                   { "id": "f43f5d2f-3f2a-4cc8-b1af-81fde7dbaf0e" },
                   { "id": "91f6a16c-ef71-437a-99ac-0b070e3ad221" },
                   { "id": "4ae68615-866f-4304-b24b-ba048dfa5ca7" },
                   { "id": "6a7ba45c-63d8-473e-9736-2eaabff79964" },
                   { "id": "16124c2d-e40b-4fcc-8f2c-5ab7870a2223" },
                   { "id": "07330162-dba1-4746-8121-a9647d49d297" },
                   { "id": "4db60cc0-36fb-42b7-9b58-a5b53019fb74" }
               ],
               "custom_rules": []
           }
       }
   ],
   "triggers": [
       {
           "name": "CloudTrailSecurityAnalyticsTrigger",
           "sev_levels": [],
           "tags": [],
           "actions": [
               {
                   "id": "",
                   "name": "Triggered alert condition:  - Severity: 1 (Highest) - Threat detector: DNS Resolver Logs",
                   "destination_id": "sns-security-channel",
                   "subject_template": {
                       "source": "Triggered alert condition:  - Severity: 1 (Highest) - Threat detector: DNS Resolver Logs",
                       "lang": "mustache"
                   },
                   "message_template": {
                       "source": "Triggered alert condition: \nSeverity: 1 (Highest)\nThreat detector: CloudTrail Detector\nDescription: \nCloudTrail data sources:\n\tocsf-3005*",
                       "lang": "mustache"
                   },
                   "throttle_enabled": false,
                   "throttle": { "value": 10, "unit": "MINUTES" }
               }
           ],
           "types": ["cloudtrail"],
           "severity": "1",
           "ids": []
       }
   ]
}

What is the expected behavior?
I expect a detector to be created.

What is your host/environment?
Lambda Python 3.11

[engechas]

This exception occurs when there's no write index for the query indices system alias. This should only occur when the index does not exist, which shouldn't happen due to this check: https://github.com/opensearch-project/alerting/blob/main/alerting/src/main/kotlin/org/opensearch/alerting/transport/TransportIndexMonitorAction.kt#L574-L577

I am not able to reproduce the issue when creating a detector with the provided config on a fresh cluster running from the current code.

Could you either add the OpenSearch version where you are facing this issue or try again with the latest release of OpenSearch to see if the issue still occurs?

[engechas]

Closing as I am unable to reproduce. Please reopen the issue if it is still occurring