Opensearch running abnormally #489
Comments
|
During initialization there are normally various warnings/error messages coming from opensearch while things are starting up and initializing that usually settle out once everything's initialized. Besides the error messages, what is actually not working? Does opensearch die? Do the other containers that use opensearch (dashboards, etc.) report that it is not available? These messages in and of themselves don't constitute a bug. As far as the "deleting the opensearch pod" results in an error in your last paragraph... I mean, yeah, I would expect that deleting the opensearch pod would cause an error. |
Because we encountered several server restarts, we redeployed Malcolm, but there was an error when starting opensearch. Only by clearing the mounted volume of opensearch and restarting it can it work |
|
Glad you got it working. |
Describe the bug
After running opensearch for a period of time, an exception log appears
To Reproduce
Steps to reproduce the behavior:
Expected behavior
A clear and concise description of what you expected to happen.
**Screenshots and/or Logs **
OpenSearch Security Plugin does not exist, disable by default
OpenSearch Performance Analyzer Plugin does not exist, disable by default
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.8.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
WARNING: System::setSecurityManager will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.8.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
WARNING: System::setSecurityManager will be removed in a future release
[2024-11-05T07:34:30,529][WARN ][o.o.g.DanglingIndicesState] [opensearch-deployment-58cfc9b467-g5f5z] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2024-11-05T07:34:32,261][WARN ][o.o.b.BootstrapChecks ] [opensearch-deployment-58cfc9b467-g5f5z] initial heap size [2147483648] not equal to maximum heap size [17179869184]; this can cause resize pauses and prevents memory locking from locking the entire heap
[2024-11-05T07:42:11,437][WARN ][o.o.c.m.MetadataIndexTemplateService] [opensearch-deployment-58cfc9b467-g5f5z] index template [malcolm_template] has index patterns [arkime_sessions3-] matching patterns from existing older templates [arkime_sessions3_ecs_template,arkime_sessions3_template] with patterns (arkime_sessions3_ecs_template => [arkime_sessions3-],arkime_sessions3_template => [arkime_sessions3-*]); this template [malcolm_template] will take precedence during new index creation
[2024-11-05T07:46:02,724][WARN ][o.o.a.t.RCFResultTransportAction] [opensearch-deployment-58cfc9b467-g5f5z] Anomaly Detector 7O9J-5IBqyhIJ63MfcWI org.opensearch.ad.common.exception.ResourceNotFoundException: No checkpoints found for model id 7O9J-5IBqyhIJ63MfcWI_model_rcf_0
[2024-11-05T07:46:02,725][ERROR][o.o.a.t.AnomalyResultTransportAction] [opensearch-deployment-58cfc9b467-g5f5z] Received an error from node MqBesElaSwyy9126QZoWEQ while doing model inference for 7O9J-5IBqyhIJ63MfcWI
org.opensearch.transport.RemoteTransportException: [opensearch-deployment-58cfc9b467-g5f5z][10.32.0.51:9300][cluster:admin/opendistro/adinternal/rcf/result]
Caused by: org.opensearch.ad.common.exception.ResourceNotFoundException: No checkpoints found for model id 7O9J-5IBqyhIJ63MfcWI_model_rcf_0
at org.opensearch.ad.ml.ModelManager.processRestoredTRcf(ModelManager.java:302) ~[?:?]
at org.opensearch.ad.ml.ModelManager.lambda$getTRcfResult$1(ModelManager.java:185) ~[?:?]
at org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:80) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.ad.ml.CheckpointDao.lambda$getTRCFModel$15(CheckpointDao.java:688) [opensearch-anomaly-detection-2.8.0.0.jar:2.8.0.0]
at org.opensearch.action.ActionListener$1.onFailure(ActionListener.java:88) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.ad.util.ClientUtil.lambda$asyncRequest$3(ClientUtil.java:128) [opensearch-anomaly-detection-2.8.0.0.jar:2.8.0.0]
at org.opensearch.action.ActionListener$1.onFailure(ActionListener.java:88) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.support.TransportAction$1.onFailure(TransportAction.java:122) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:224) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) [opensearch-index-management-2.8.0.0.jar:2.8.0.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.indexmanagement.controlcenter.notification.filter.IndexOperationActionFilter.apply(IndexOperationActionFilter.kt:39) [opensearch-index-management-2.8.0.0.jar:2.8.0.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:476) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.client.support.AbstractClient.get(AbstractClient.java:572) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.ad.util.ClientUtil.asyncRequest(ClientUtil.java:126) [opensearch-anomaly-detection-2.8.0.0.jar:2.8.0.0]
at org.opensearch.ad.ml.CheckpointDao.getTRCFModel(CheckpointDao.java:679) [opensearch-anomaly-detection-2.8.0.0.jar:2.8.0.0]
at org.opensearch.ad.ml.ModelManager.getTRcfResult(ModelManager.java:181) [opensearch-anomaly-detection-2.8.0.0.jar:2.8.0.0]
at org.opensearch.ad.transport.RCFResultTransportAction.doExecute(RCFResultTransportAction.java:77) [opensearch-anomaly-detection-2.8.0.0.jar:2.8.0.0]
at org.opensearch.ad.transport.RCFResultTransportAction.doExecute(RCFResultTransportAction.java:36) [opensearch-anomaly-detection-2.8.0.0.jar:2.8.0.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) [opensearch-index-management-2.8.0.0.jar:2.8.0.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.indexmanagement.controlcenter.notification.filter.IndexOperationActionFilter.apply(IndexOperationActionFilter.kt:39) [opensearch-index-management-2.8.0.0.jar:2.8.0.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:102) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:98) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:113) [opensearch-index-management-2.8.0.0.jar:2.8.0.0]
at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:106) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.transport.TransportService.sendLocalRequest(TransportService.java:1058) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.transport.TransportService$3.sendRequest(TransportService.java:152) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:996) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.transport.TransportService.sendRequest(TransportService.java:883) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.transport.TransportService.sendRequest(TransportService.java:826) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.ad.transport.AnomalyResultTransportAction.lambda$onFeatureResponseForSingleEntityDetector$10(AnomalyResultTransportAction.java:604) [opensearch-anomaly-detection-2.8.0.0.jar:2.8.0.0]
at org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:80) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.ad.feature.FeatureManager.updateUnprocessedFeatures(FeatureManager.java:219) [opensearch-anomaly-detection-2.8.0.0.jar:2.8.0.0]
at org.opensearch.ad.feature.FeatureManager.lambda$getCurrentFeatures$1(FeatureManager.java:165) [opensearch-anomaly-detection-2.8.0.0.jar:2.8.0.0]
at org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:80) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.ad.feature.SearchFeatureDao.lambda$getFeatureSamplesForPeriods$14(SearchFeatureDao.java:606) [opensearch-anomaly-detection-2.8.0.0.jar:2.8.0.0]
at org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:80) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.ActionListener$6.onResponse(ActionListener.java:299) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:113) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:107) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.search.TransportSearchAction.lambda$executeRequest$0(TransportSearchAction.java:399) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:80) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.ActionListener$5.onResponse(ActionListener.java:266) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.search.AbstractSearchAsyncAction.sendSearchResponse(AbstractSearchAsyncAction.java:658) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.search.ExpandSearchPhase.run(ExpandSearchPhase.java:132) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:427) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:421) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.search.FetchSearchPhase.moveToNextPhase(FetchSearchPhase.java:299) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.search.FetchSearchPhase.lambda$innerRun$1(FetchSearchPhase.java:139) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:151) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:123) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
[2024-11-05T07:47:02,808][ERROR][o.o.a.t.ADTaskManager ] [opensearch-deployment-58cfc9b467-g5f5z] Failed to update realtime task for detector 7O9J-5IBqyhIJ63MfcWI
org.opensearch.ad.common.exception.ResourceNotFoundException: can't find latest task
at org.opensearch.ad.task.ADTaskManager.lambda$updateLatestADTask$80(ADTaskManager.java:1976) [opensearch-anomaly-detection-2.8.0.0.jar:2.8.0.0]
at org.opensearch.ad.task.ADTaskManager.lambda$getAndExecuteOnLatestADTask$21(ADTaskManager.java:943) [opensearch-anomaly-detection-2.8.0.0.jar:2.8.0.0]
at org.opensearch.ad.task.ADTaskManager.lambda$getAndExecuteOnLatestADTasks$22(ADTaskManager.java:1016) [opensearch-anomaly-detection-2.8.0.0.jar:2.8.0.0]
at org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:80) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:113) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:107) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.search.TransportSearchAction.lambda$executeRequest$0(TransportSearchAction.java:399) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:80) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.ActionListener$5.onResponse(ActionListener.java:266) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.search.AbstractSearchAsyncAction.sendSearchResponse(AbstractSearchAsyncAction.java:658) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.search.ExpandSearchPhase.run(ExpandSearchPhase.java:132) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:427) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:421) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.search.FetchSearchPhase.moveToNextPhase(FetchSearchPhase.java:299) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.search.FetchSearchPhase.lambda$innerRun$1(FetchSearchPhase.java:139) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:151) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:123) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
[2024-11-05T07:47:02,810][ERROR][o.o.a.ExecuteADResultResponseRecorder] [opensearch-deployment-58cfc9b467-g5f5z] Can't find latest realtime task of detector 7O9J-5IBqyhIJ63MfcWI
[2024-11-05T08:48:46,676][WARN ][o.o.m.f.FsHealthService ] [opensearch-deployment-58cfc9b467-g5f5z] health check of [/usr/share/opensearch/data/nodes/0] took [10403ms] which is above the warn threshold of [5s]
[2024-11-05T19:34:38,287][ERROR][o.o.a.a.AlertIndices ] [opensearch-deployment-58cfc9b467-g5f5z] info deleteOldIndices
[2024-11-05T19:34:38,287][ERROR][o.o.a.a.AlertIndices ] [opensearch-deployment-58cfc9b467-g5f5z] info deleteOldIndices
[2024-11-05T19:34:38,314][ERROR][o.o.s.i.DetectorIndexManagementService] [opensearch-deployment-58cfc9b467-g5f5z] info deleteOldIndices
[2024-11-05T19:34:38,314][ERROR][o.o.s.i.DetectorIndexManagementService] [opensearch-deployment-58cfc9b467-g5f5z] info deleteOldIndices
Malcolm Version:
How are you running Malcolm?
k8s
Additional context
There is another issue in k8s where manually deleting the opensearch pod results in the opensearch startup report org.onsearch.action.search SearchPhaseExecutionException: all shards failed
The text was updated successfully, but these errors were encountered: