Send data to self hosted API

[umnibot]

I have tried to find information about sending the data to my own API, but cannot find information on what request to expect, so I can build my logic.

Is there any documentation as of what the data,php should look like? Or at least is the request GET or POST

Thanks

[umnibot]

Okay, I found out that it is a simple POST. However, when I try to just send it to my server, it says Connection refused.
But I can access the data.php from any other server.

Do is this option Send data to custom API even works?

[ricki-z]

This option is working. Is the sensor in the same network as this server or in a guest wifi?
You also need to activate the option "Send data to custom API"(but I think you did this and only the screenshot was made with the option disabled)

[umnibot]

Hello, the sensor is within my home network, while the API is on my webserver.
Yes, I did check the option "Send data to custom API".
I have tried all possible scenarios that I could think of (HTTP, https, different ports, different pats, combined paths). Also, install a logger script where the API is, to see if there is a connection attempt.

if I execute the URL via the browser it logs the activity, but no attempts if I try it via the sensor.

[umnibot]

This is what I have in device status

[ricki-z]

So your webserver is in your local network (192.168.234.1)?
Can you see any access in your webserver logs?

[umnibot]

No, my server is not in my local network. It is an outside network

[ricki-z]

Then please enter the correct IP of the server (192.168.234.1 is your router I think) or the hostname in the field "server".

[umnibot]

Well, I don't know where you are getting this IP address.
in the server field, I have the hostname of my server

[umnibot]

Oh, you got it from the initial screenshot. That's the configuration before me editing it.

[ricki-z]

remove the http:// at the start f the line. the protocol is choosen by port. There must be only the server name or IP in this field.

[umnibot]

Same result with a configuration like this.

[ricki-z]

Really?
Server: Only the server name or server IP !!!!
Path: the complete path to the custom API script, in your case /App/AirQData/data.php

[umnibot]

Tried these ways

Now, there is no "Connection refused" error in the device status

However, there is also no attempt to send any data to my server. The logs are not showing any attempt.

[umnibot]

is there anything else I can try. Still don't receive any data from the sensor to my server

[ricki-z]

@umnibot can you send me the hostname and script path to rajko (at) sensor.community? So I can have a look at that.

[ricki-z]

The server is forwarding to https. Could you please activate https for this API and check again? It's possible that the firmware isn't following the 30x redirect but also doesn't show an error (http error codes are greater or equal 400).

[umnibot]

Just tried with https and got the connection refused

[ricki-z]

Our firmware only supports the following ciphers (minimum set of ciphers):
TLS_RSA_WITH_AES_128_CBC_SHA256,
TLS_RSA_WITH_AES_256_CBC_SHA256,
TLS_RSA_WITH_AES_128_CBC_SHA,
TLS_RSA_WITH_AES_256_CBC_SHA
We can't enable all possible ciphers as this would need too much flash and RAM.

Your server supports the following ciphers:

• TLS 1.3 (server has no preference)
  TLS_AES_128_GCM_SHA256
  TLS_AES_256_GCM_SHA384
  TLS_CHACHA20_POLY1305_SHA256

• TLS 1.2 (server has no preference)
  TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
  TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
  TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
  TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
  TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256

[umnibot]

So, the only option, if I cant support these ciphers, would be to make it HTTP:. Right?

[umnibot]

Just activated TLS_RSA_WITH_AES_128_CBC_SHA256 but still can't make it work

[ricki-z]

At the moment: yes.
We have to check the memory consumption of additional ciphers. My last info was that we need to activate all cipher if we need others. That would really need too much flash.
Another factor is the time for de- and encoding. Other ciphers are so complex that the NodeMCU needs too long.

[umnibot]

Okay, I will set the folder to http to see if it will work.

[umnibot]

Ok, I have managed to add make it working. I have a question about a few data keys marked in red. What is that measurement?

Also in the device status, I see read timeout. Is that normal?

[ricki-z]

The measurements "samples", "min_micro" and "max_micro" are used to check the overall performance of the firmware. "samples" is the count of loops the main() function was running, "min_micro" and "max_micro" are the minimum and the maximum time the main() loop was running (in microseconds).

To the screenshot: Only the last errors are shown, not the result of the lastest transmission. So the "read timeout" was only a one time failure. See the difference between "number of measurements" and "data send".

[ricki-z]

Regarding the problem with HTTPS: Please check your server at https://www.ssllabs.com/ssltest/analyze.html . As you can see the ciphers needed aren't supported yet.

[manne01]

Hallo, es hat mich grade leider einige Zeit gekostet rauszufinden, was mit meinem Feinstaubsensor los war. Auch bei mir lags am "zu modernen SSL Profil" am empfangenden Webserver meiner Custom API welches serverseitig zwischendurch mal aktiviert wurde.

Leider hat das Logging im Feinstaub Sensor garnicht geholfen. Es stand (max Info) immer nur dran: Sending Custom API (nach den öffentlichen APIs natürlich;-) )
## Sending to custom API
time for sending: 6ms

Im "Fehler" modus des log dasselbe. Allerdings auch kein "success" oder so.

Es wäre klasse, wenn das Logging für solche Verbindungsprobleme zB den Return Code oder sowas ins Log schreiben würde.

Bei mir Firmware NRZ-2020-133/DE vom Nov 29 2020 hab ich es nicht geschafft die Verbindung zur Custom API damit zu debuggen.

Erst diese Diskussion hier brachte mich auf den richtigen Weg. Besten Dank dafür!

[umnibot]

Hi Rajko. It seems like with firmware update NRZ-2020-133 the sensor stopped sending info to the custom server. Now looking at my server log it seems that the last was data sent on 2020-12-14 16:13:43

Nothing on the server was changed nor on the sensor (using http not https).
In the sensor debug dont see anything to tell me what is going on.

ws: root ...
ws: status ...
ws: values ...
PM10: 13.93
PM2.5: 8.32

Sending to sensor.community - SDS011

Succeeded - api.sensor.community
Temperature (°C): 27.02
Pressure (hPa): 94872.22
Humidity (%): 23.94

Sending to sensor.community - BME280

Succeeded - api.sensor.community

Sending to madavi.de:

Succeeded - api-rrd.madavi.de

Sending to custom api:

Time for Sending (ms): 5873
PM10: 8.20
PM2.5: 5.55

The sensor is not even trying to access the custom API. If I try to access the API over a browser, there are no restrictions and I see the attempt in the server log (but not see attempt from the sensor).
Was there anything changed or sould I change somethng?

Thanksd

[ricki-z]

Please check that the SSL certificate is max. 2048 bit. And that the supported ciphers contain one of the following:
TLS_RSA_WITH_AES_128_CBC_SHA256,
TLS_RSA_WITH_AES_256_CBC_SHA256,
TLS_RSA_WITH_AES_128_CBC_SHA,
TLS_RSA_WITH_AES_256_CBC_SHA

[umnibot]

Do I need SSL for HTTP only ?

[umnibot]

Also just checked that we have TLS_RSA_WITH_AES_128_CBC_SHA256

[umnibot]

[ricki-z]

SSL isn't needed. But in most cases the SSL cert and the missing ciphers are the cause for the not working self hosted API.
You could check if it's working without SSL.
Do you have access to the error logs of the web server? Maybe there is a hint why the data isn't saved.

[umnibot]

It doesnt working with or without SSL. and in the server logs there is no evidence of an attempt for trying to connect.
How does an attempt for sending data from the sensor should look like?

[umnibot]

The problem is sending the data to a custom endpoint that was working before and not working now. So no idea how Madavi and SC server values can help me here. I already know what to expect and can see my values https://data.sensor.community/airrohr/v1/sensor/40205/

The sensor just not sending the data to my endpoint. I wrote a simple PHP script that listens to POST requests and get its data

$post = file_get_contents('php://input');
$post = json_decode($post, true);
$post['timestamp'] = date("Y-m-d H:i:s",time());

$post = json_encode($post);

file_put_contents('values.php', $post);

Even if the request is empty it should log the current timestamp to the values.php file which is not the case.

[ricki-z]

There wasn't any change to the release firmware in the last months. So this shouldn't be the cause of the problem.
Maybe there were any updates for PHP on your hosting platform. Or there is any "firewall" or "antispam" tool on the way from the sensor to the server that is blocking the request.

[umnibot]

Hi Rajko,
Finally, I was able to resolve the issue. The problem was not with the new firmware. Somehow the subdomain I was using was blocking the receiving of data. The weird thing was that there was no log of the sensor trying to connect the server, nor information that it was blocked at least (not sure about the blocking statement). After exploring all the options I have decided to delete the subdomain and create it again... Surprisingly that fixed the problem and now everything is going as it should.
One again,
Thanks for your help.