PrometheusDataReceiver does not receive any metrics while running performance benchmark test

[rohitprg]

I am doing performance benchmark testing(Scenario10kItemsPerSecond) against Opentelemetry collector using datasender-
httpotlp and datareceiver - prometheus. Below is the testcase which I am trying to run. but I always see received as 0 items,even after increasing cpu,ram and prometheus ScrapeInterval and ScrapeTimeout in prometheusDataReceiver ("github.com/open-telemetry/opentelemetry-collector-contrib/testbed/datareceivers") I have added this test in class (https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/4061bf6d6d3198c395668a0a08373e9084d0477a/testbed/tests/metric_test.go)

{
	name:     "OTLP-HTTP-Prometheus-Metric",
	sender:   testbed.NewOTLPHTTPMetricDataSender(testbed.DefaultHost, testbed.GetAvailablePort(t)),
	receiver: datareceivers.NewPrometheusDataReceiver(testbed.GetAvailablePort(t)),
	resourceSpec: testbed.ResourceSpec{
		ExpectedMaxCPU: 100,
		ExpectedMaxRAM: 1000,
	},
}

Below is the result

=== RUN   TestMetric10kDPS
--- FAIL: TestMetric10kDPS (115.45s)
=== RUN   TestMetric10kDPS/OTLP-HTTP
2023/02/08 11:10:39 Starting mock backend...
2023/02/08 11:10:39 Starting Agent (opentelemetry-collector-contrib/bin/oteltestbedcol_darwin_arm64)
2023/02/08 11:10:39 Writing Agent log to opentelemetry-collector-contrib/testbed/tests/results/TestMetric10kDPS/OTLP-HTTP/agent.log
2023/02/08 11:10:39 Agent running, pid=73576
2023/02/08 11:10:39 Starting load generator at 10000 items/sec.
2023/02/08 11:10:42 Agent RAM (RES):   0 MiB, CPU:84.6% | Sent:    185507 items | Received:         0 items (0/sec)
2023/02/08 11:10:45 Agent RAM (RES): 240 MiB, CPU:33.9% | Sent:    395500 items | Received:         0 items (0/sec)
2023/02/08 11:10:48 Agent RAM (RES): 432 MiB, CPU:43.0% | Sent:    604800 items | Received:         0 items (0/sec)
2023/02/08 11:10:51 Agent RAM (RES): 658 MiB, CPU:43.0% | Sent:    814800 items | Received:         0 items (0/sec)
2023/02/08 11:10:54 Agent RAM (RES): 791 MiB, CPU:56.5% | Sent:   1024800 items | Received:         0 items (0/sec)
2023/02/08 11:10:54 Stopped generator. Sent:   1049300 items
2023/02/08 11:10:57 Agent RAM (RES):1089 MiB, CPU:83.5% | Sent:   1049300 items | Received:         0 items (0/sec)
2023/02/08 11:11:00 Agent RAM (RES):1089 MiB, CPU: 1.8% | Sent:   1049300 items | Received:         0 items (0/sec)
2023/02/08 11:11:03 Agent RAM (RES):1089 MiB, CPU: 0.0% | Sent:   1049300 items | Received:         0 items (0/sec)
2023/02/08 11:11:06 Agent RAM (RES):1089 MiB, CPU: 0.0% | Sent:   1049300 items | Received:         0 items (0/sec)
2023/02/08 11:11:09 Agent RAM (RES):1089 MiB, CPU: 0.0% | Sent:   1049300 items | Received:         0 items (0/sec)
2023/02/08 11:11:12 Agent RAM (RES):1089 MiB, CPU: 0.0% | Sent:   1049300 items | Received:         0 items (0/sec)
2023/02/08 11:11:15 Agent RAM (RES):1089 MiB, CPU: 0.0% | Sent:   1049300 items | Received:         0 items (0/sec)
2023/02/08 11:11:18 Agent RAM (RES):1089 MiB, CPU: 0.0% | Sent:   1049300 items | Received:         0 items (0/sec)
2023/02/08 11:11:21 Agent RAM (RES):1089 MiB, CPU: 0.0% | Sent:   1049300 items | Received:         0 items (0/sec)
2023/02/08 11:11:24 Agent RAM (RES):1089 MiB, CPU: 0.0% | Sent:   1049300 items | Received:         0 items (0/sec)
2023/02/08 11:11:27 Agent RAM (RES):1089 MiB, CPU: 0.0% | Sent:   1049300 items | Received:         0 items (0/sec)
2023/02/08 11:11:30 Agent RAM (RES):1089 MiB, CPU: 0.0% | Sent:   1049300 items | Received:         0 items (0/sec)
2023/02/08 11:11:33 Agent RAM (RES):1089 MiB, CPU: 0.0% | Sent:   1049300 items | Received:         0 items (0/sec)
2023/02/08 11:11:36 Agent RAM (RES):1089 MiB, CPU: 0.0% | Sent:   1049300 items | Received:         0 items (0/sec)
2023/02/08 11:11:39 Agent RAM (RES):1089 MiB, CPU: 0.0% | Sent:   1049300 items | Received:         0 items (0/sec)
2023/02/08 11:11:42 Agent RAM (RES):1089 MiB, CPU: 0.0% | Sent:   1049300 items | Received:         0 items (0/sec)
2023/02/08 11:11:45 Agent RAM (RES):1089 MiB, CPU: 0.0% | Sent:   1049300 items | Received:         0 items (0/sec)
2023/02/08 11:11:48 Agent RAM (RES):1089 MiB, CPU: 0.0% | Sent:   1049300 items | Received:         0 items (0/sec)
2023/02/08 11:11:51 Agent RAM (RES):1089 MiB, CPU: 0.0% | Sent:   1049300 items | Received:         0 items (0/sec)
2023/02/08 11:11:54 Agent RAM (RES):1089 MiB, CPU: 0.0% | Sent:   1049300 items | Received:         0 items (0/sec)
2023/02/08 11:11:57 Agent RAM (RES):1089 MiB, CPU: 0.0% | Sent:   1049300 items | Received:         0 items (0/sec)
2023/02/08 11:12:00 Agent RAM (RES):1089 MiB, CPU: 0.0% | Sent:   1049300 items | Received:         0 items (0/sec)
2023/02/08 11:12:03 Agent RAM (RES):1089 MiB, CPU: 0.0% | Sent:   1049300 items | Received:         0 items (0/sec)
2023/02/08 11:12:06 Agent RAM (RES):1089 MiB, CPU: 0.0% | Sent:   1049300 items | Received:         0 items (0/sec)
2023/02/08 11:12:09 Agent RAM (RES):1089 MiB, CPU: 0.0% | Sent:   1049300 items | Received:         0 items (0/sec)
2023/02/08 11:12:12 Agent RAM (RES):1089 MiB, CPU: 0.0% | Sent:   1049300 items | Received:         0 items (0/sec)
2023/02/08 11:12:15 Agent RAM (RES):1089 MiB, CPU: 0.0% | Sent:   1049300 items | Received:         0 items (0/sec)
2023/02/08 11:12:18 Agent RAM (RES):1089 MiB, CPU: 0.0% | Sent:   1049300 items | Received:         0 items (0/sec)
2023/02/08 11:12:21 Agent RAM (RES):1089 MiB, CPU: 0.0% | Sent:   1049300 items | Received:         0 items (0/sec)
2023/02/08 11:12:24 Agent RAM (RES):1089 MiB, CPU: 0.0% | Sent:   1049300 items | Received:         0 items (0/sec)
2023/02/08 11:12:27 Agent RAM (RES):1089 MiB, CPU: 0.0% | Sent:   1049300 items | Received:         0 items (0/sec)
2023/02/08 11:12:30 Agent RAM (RES):1089 MiB, CPU: 0.0% | Sent:   1049300 items | Received:         0 items (0/sec)
2023/02/08 11:12:33 Agent RAM (RES):1089 MiB, CPU: 0.0% | Sent:   1049300 items | Received:         0 items (0/sec)
    test_case.go:287: Time out waiting for all data items received
2023/02/08 11:12:34 Gracefully terminating Agent pid=73576, sending SIGTEM...
2023/02/08 11:12:34 Stopping process monitor.
2023/02/08 11:12:34 Agent process stopped, exit code=0
    validator.go:42: 
        	Error Trace:	/opentelemetry-collector-contrib/testbed/tests/validator.go:42
        	            				/opentelemetry-collector-contrib/testbed/tests/test_case.go:248
        	            				/opentelemetry-collector-contrib/testbed/tests/scenarios.go:190
        	            				/opentelemetry-collector-contrib/testbed/tests/metric_test.go:94
        	Error:      	Not equal: 
        	            	expected: 1049300
        	            	actual  : 0
        	Test:       	TestMetric10kDPS/OTLP-HTTP
        	Messages:   	Received and sent counters do not match.
2023/02/08 11:12:34 Stopping mock backend...
2023/02/08 11:12:34 Stopped backend. Received:         0 items (0/sec)
    --- FAIL: TestMetric10kDPS/OTLP-HTTP (115.44s)


Expected :1049300
Actual   :0
<Click to see difference>



FAIL

Can please somebody help why prometheus receiver is not receiving any data. Or do we already have any existing test case in opentelemetry collector contrib gihub?

[rohitprg]

@atoulme Can you please help

[atoulme]

Sorry, I don't know of this part of the code enough to help. Please try Slack or open an issue if you see a bug.

[rohitprg]

Thanks.