Elasticsearch queries failing

[arikuukka]

Hello,

For some reason the queries made by Graph Explorer to the Elasticsearch seem to fail. I have tried installing GE both from PyPI with pip and from latest source using git, both result in the same problem.

For example, when making a query "list" in the GE web ui, GE returns: "Targets matching 0/229 (limit 500), Graphs matching 0/0" on the web ui, and the following is logged by GE:

2014-05-27 03:48:29,234 - app - DEBUG - making sure index exists..
2014-05-27 03:48:29,277 - app - DEBUG - making sure shard is started..
2014-05-27 03:48:29,278 - app - DEBUG - shard is ready!
2014-05-27 03:48:29,278 - app - DEBUG - querying up to 500 metrics from ES...
2014-05-27 03:48:29,279 - app - DEBUG - making sure index exists..
2014-05-27 03:48:29,320 - app - DEBUG - making sure shard is started..
2014-05-27 03:48:29,322 - app - DEBUG - shard is ready!
2014-05-27 03:48:29,323 - app - DEBUG - Sending query to ES: {'filtered': {'filter': {'and': [{'prefix': {'tags': 'target_type='}}, {'prefix': {'tags': 'unit='}}]}, 'query': {'match_all': {}}}}
2014-05-27 03:48:29,366 - app - DEBUG - got 4 metrics!
2014-05-27 03:48:29,366 - app - DEBUG - making sure index exists..
2014-05-27 03:48:29,411 - app - DEBUG - making sure shard is started..
2014-05-27 03:48:29,412 - app - DEBUG - shard is ready!

If I make that same logged query manually to ES, I get zero hits:

curl -XGET 'http://127.0.0.1:9200/graphite_metrics2/_search?pretty=1' -d '{"query" : {"filtered": {"filter": {"and": [{"prefix": {"tags": "target_type="}}, {"prefix": {"tags": "unit="}}]}, "query": {"match_all": {}}}}}'

{
"took" : 8,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 0,
"max_score" : null,
"hits" : [ ]
}
}

I noticed that if I leave the "=" characters away from the query (from "target_type=" and "unit=", then I do get some real hits from ES;

curl -XGET 'http://127.0.0.1:9200/graphite_metrics2/_search?pretty=1' -d '{"query" : {"filtered": {"filter": {"and": [{"prefix": {"tags": "target_type"}}, {"prefix": {"tags": "unit"}}]}, "query": {
"match_all": {}}}}}'

{
"took" : 9,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 228,
"max_score" : 1.0,
"hits" : [ {
"_index" : "graphite_metrics2",
"_type" : "metric",
"_id" : "servers.server1.cpu.cpu1.idle",
"_score" : 1.0, "_source" : {"tags": ["core=cpu1", "what=cpu_usage", "plugin=cpu", "target_type=gauge_pct", "server=server1", "type=idle", "unit=Jiff"]}
}, {
"_index" : "graphite_metrics2",
"_type" : "metric",
"_id" : "servers.server1.memory.Active",
"_score" : 1.0, "_source" : {"tags": ["plugin=memory", "type=active", "target_type=gauge", "server=server1", "where=system_memory", "unit=B"]}
}, {
"_index" : "graphite_metrics2",
"_type" : "metric",
"_id" : "servers.server2.physicaldisk.z-drive.diskwritequeuelength",
"_score" : 1.0, "_source" : {"tags": ["plugin=catchall_diamond", "target_type=unknown", "server=server2", "source=diamond", "n1=physicaldisk", "n2=z-drive", "n3=diskwritequeuelength", "unit=unknown"]}
}, {
"_index" : "graphite_metrics2",
"_type" : "metric",
"_id" : "servers.server2.networkinterface.localareaconnection_11.packetssentnon-unicast-sec",
"_score" : 1.0, "_source" : {"tags": ["plugin=catchall_diamond", "target_type=unknown", "server=server2", "source=diamond", "n1=networkinterface", "n2=localareaconnection_11", "n3=packetssentnon-unicast-sec", "unit=unknown"]}
}, {
"_index" : "graphite_metrics2",
"_type" : "metric",
"_id" : "servers.server1.iostat.xvda.io_milliseconds_weighted",
"_score" : 1.0, "_source" : {"tags": ["plugin=iostat", "target_type=gauge", "server=server1", "device=xvda", "type=io", "unit=ms_weighted"]}
}, {
"_index" : "graphite_metrics2",
"_type" : "metric",
"_id" : "carbon.agents.server1.cache.queries",
"_score" : 1.0, "_source" : {"tags": ["unit=queries", "target_type=gauge", "agent=server1", "plugin=carbon"]}
}, {
"_index" : "graphite_metrics2",
"_type" : "metric",
"_id" : "servers.server1.iostat.xvdb.await",
"_score" : 1.0, "_source" : {"tags": ["plugin=iostat", "target_type=gauge", "server=server1", "device=xvdb", "type=iop service time", "unit=ms"]}
}, {
"_index" : "graphite_metrics2",
"_type" : "metric",
"_id" : "servers.server1.iostat.xvda1.read_byte_per_second",
"_score" : 1.0, "_source" : {"tags": ["plugin=iostat", "target_type=rate", "server=server1", "device=xvda1", "type=read", "unit=B"]}
}, {
"_index" : "graphite_metrics2",
"_type" : "metric",
"_id" : "servers.server1.memory.SwapCached",
"_score" : 1.0, "_source" : {"tags": ["plugin=memory", "type=swap_cached", "target_type=gauge", "server=server1", "where=system_memory", "unit=B"]}
}, {
"_index" : "graphite_metrics2",
"_type" : "metric",
"_id" : "servers.server1.iostat.xvda.service_time",
"_score" : 1.0, "_source" : {"tags": ["plugin=iostat", "target_type=gauge", "server=server1", "device=xvda", "type=service time", "unit=ms"]}
} ]
}
}

[arikuukka]

Elasticsearc version is:
[2014-05-20 09:56:23,254][INFO ][node ] [Jackhammer] version[1.1.1], pid[4944], build[f1585f0/2014-04-16T14:27:12Z]

[Dieterbe]

weird.