Change all uv fields to uvi, BREAKING change to UV sensors

[zuckschwerdt]

BREAKING change to Acurite-Atlas, Bresser-6in1, Bresser-7in1, Cotech-367959, Emax-W6, AOK-5056, Oregon-UVR128, Oregon-UV800, Vevor-7in1.

This changes all "uv" named UV Index keys to "uvi" for a proper distinction between a raw UV reading and UV Index values.

Part of #3103

[DigiH]

I wonder what the two Bresser, the Cotech and FineOffset WS80 one decimal doubles are

DATA_FORMAT, "%.1f", DATA_DOUBLE
Either some very obscure way of presenting a usually integer UV Index scale with additional in between decimals, or also some other raw data not indicating the UV Index at all?!? With the latter also requiring to rename their "UV Index" label.

The initial implementations of these decoders might have moire insight into this, or if someone has one of these stations.

[zuckschwerdt]

Code and notes on all changed fields indicated UV Index values. Some present decimals, yes.

[DigiH]

Ok then, not that I have ever seen or known about decimal UV Index values, but if that is the case the changes look fine :)

Thanks for the disambiguation 👍

[gdt]

I guess this is changing a field called 'uv' to be called value vs index, to unconfuse it with the index.

But what is this field? Is it "some random thing in a decoder we think is related to uv"? Or does it have more semantics? I can see some sort of radiometric measurement that means something, before converting to index, but I really doubt that's what is going on.

I wonder then if we should rename it something that indicates that it is unclear, or how many drivers generate it. But I get it that unconfusing is a positive step, even if confusion remains, so this is not an objection.

[DigiH]

As I have this on my station I have to say I really don't know what kind of UV related value this might be, but with consumer stations generally having UVA only photodiodes I assume it is the raw diode value from which the eventual UV Index value is being calculated!?!

[gdt]

That could be. What I don't like is reporting a bunch of such values from different kinds of manufacturers/hardware using the same name, implying they are the same thing. Perhaps if we rename it UV Raw, and adopt a convention that "Foo Raw" means e.g. some unprocessed ADC value that we believe is related to Foo, and that while people might want to e.g. transmit it to Home Assistant and log/graph it, there's no reason to expect a Foo Raw from device A to mean the same thing as Foo Raw from device B. This is different from things like temperature, illuminance, station pressure, UV index.

I worked on the Ford TPMS decoder a while ago, and it had values that were in odd units (0.25 psi), and we didn't and don't put those values in the output. Generally, once mapped to real units, the upstream data isn't useful or interesting.

In this case, though, it sounds like in the over-the-air format there is both a "UV Raw" and a "UV Index"? If so, I guess that's ok. But if rtl_433 is calculating UV Index, then I would lean to suppressing the Raw in the output.

[zuckschwerdt]

As you suspect it's a mapping from a roughly linear raw(?) value.

rtl_433/src/devices/fineoffset.c

Lines 268 to 285 in bf7f9b7

	// UV value UVI
	// 0-432 0
	// 433-851 1
	// 852-1210 2
	// 1211-1570 3
	// 1571-2017 4
	// 2018-2450 5
	// 2451-2761 6
	// 2762-3100 7
	// 3101-3512 8
	// 3513-3918 9
	// 3919-4277 10
	// 4278-4650 11
	// 4651-5029 12
	// >=5230 13
	int uvi_upper[] = {432, 851, 1210, 1570, 2017, 2450, 2761, 3100, 3512, 3918, 4277, 4650, 5029};
	int uv_index = 0;
	while (uv_index < 13 && uvi_upper[uv_index] < uv_raw) ++uv_index;

added with #809

This raw value has a resolution of about 400 units per Index step. Perhaps a case for a 0.01 UV Index here :)

We could drop this, possibly raw, value if we represent all useful information from it in a standard way.

We could then also debate to use uv as the UV Index key and thus hit less devices with a breaking change. I have a feeling that "UVI" is more of a known term then "UV" alone.

[gdt]

I feel that UVI is better than UV, but if we (rtl_433) decide that a field named UV means "UV index", that's fine. There would then be a different change, to rename all fields that are UV that are not UV index, to something else.

Given that fineoffset code, I'd say just output the index, and not the raw. And make uv index have 2 places, if not 3. The point is that with 3, you can back out the raw if you want, or you can see the structure in how it changes, and that removes the last "but I want the info in the raw, even if just to see if it makes sense".

[gdt]

I am 99% sure the UV index is not constrained to integers. Precision greater than 0.1 does not really seem useful, so .1 seems good to me.

[DigiH]

I am 99% sure the UV index is not constrained to integers. Precision greater than 0.1 does not really seem useful, so .1 seems good to me.

Yes, it is not, but due to the limitations of current UV sensor technology, especially in consumer products, and usually being limited to only one UV spectrum, most stations do only present an integer value, at least the ones I have seen and used so far. With all these limitations I think that with any station showing .1 precision it is more of a marketing decision than a precision taken too seriously.

[gdt]

I am 99% sure the UV index is not constrained to integers. Precision greater than 0.1 does not really seem useful, so .1 seems good to me.

Yes, it is not, but due to the limitations of current UV sensor technology, especially in consumer products, and usually being limited to only one UV spectrum, most stations do only present an integer value, at least the ones I have seen and used so far. With all these limitations I think that with any station showing .1 precision it is more of a marketing decision than a precision taken too seriously.

Your comment is lacking a footnote to test results :-) There's precision and there's accuracy, and I've found that things that are repeatable even if off are still very useful to see relative changes - like 0.01C resolution in temperature. I really don't know what is justifiable (and without a proper evaluation nobody else does either). But, rtl_433's approach is, generally, to represent what's in the packet, at least if it isn't goofy. An index that more or less goes from 0 to 12 using 0.1 does not seem silly. I would agree that 0.01 is unlikely to make sense.

[gdt]

An index that more or less goes from 0 to 12 using 0.1 does not seem silly. I would agree that 0.01 is unlikely to make sense.

I just looked at data from a Davis Vantage Pro2, and it outputs uv index with resolution 0.1. Looking at last summer, at times that appear blue sky, the index changes smoothly at the 0.1 precision, and I don't see the noise that would be present if that precision were goofy. The curve looks reasonable and tracks solar radiation, and tracks theoretical solar radiation as well as measured solar radiation does. Arguably the data precision should be higher, as you don't want to lose information due to representation. So that's at least an existence proof that one device received by rtl_433 is sensibly reporting UV index to 0.1. I can certainly believe that there are some sensors for which this isn't true :-)

[DigiH]

The smoothness of the data was never really any concern for me, and I'm sure all other stations with an integer UV Index also have smooth raw data from which they round.

The only thing I was wondering about was that any station presenting a one decimal UV value to the user maybe should also have their UV Index renamed to UVA index or similar, as with any such precision a more precise label might also be suitable, while with any rounded values for the other stations a generic UV is sufficient.

If and when I might get such a station to replace my current one, I'm not too bothered about what current users assume they might be getting with that precision. But when I do get one I'd be very interested in which kind of UV sensor they have built in and what its precise nanometer range and responses are :)

[DigiH]

Or more bluntly, like for the FinhrOffset WS90, which seems at least to be honest about its UV accuracy, as most others usually only are with the temperature and humidity.

From the SW90 manual specification section

So what good is a single decimal presentation of a UV value which is only accurate to ±2 - when an integer would also definitely suffice instead, without giving the expectation of a more fine grained more accurate value.

And even your Davis Vantage Pro2, which admittedly is an even more professional unit, states its accuracy only as ±0.5. And it seems to be even more UVB weighted, so my UVA assumption above was wrong, for the Davis at least …

So the integer rounding of most other stations seems sensible to me, as it at least doesn't give the impression of a false accuracy, which isn't really there with even pro-user station UV sensors.

[gdt]

So the integer rounding of most other stations seems sensible to me, as it at least doesn't give the impression of a false accuracy, which isn't really there with even pro-user station UV sensors.

The point of rtl_433 reducing precision is not to convey errors to users. It is to use the smallest precision such that it is pretty clear there is no loss of possibly useful information. Error estimates should be conveyed separately.

And, as I said before, just because there is an accuracy spec, that does not mean that granularity that is finer than the accuracy spec is not useful or inappropriate. It's quite clear that my readings within an hour or two are stable to better than 0.1 UV index units, even if the absolute calibration is unclear.

Most importantly, people use rtl_433 as a way to log data and later analyze it. For that to work, you have to have enough resolution so that you are not losing information in the data transmission pipeline. So unless you are starting to see measurement noise in your data, you don't have enough resolution.

If you want to round to integer for storage or display, by all means do it. But you are arguing that because that's what you want to do, nobody should have access to the higher-resolution data provided by some measurement devices.

A comparable thing is temperature. I have multiple sensors, and some are 0.1 degree and some are 0.01. Yes, it's clearly ridiculous to expect absolute accuracy anywhere near 0.01C. But, when I look at the graphs, I see signal. I can see the heating system cycling with a 0.05F change in temperature (this isn't near a heat source). This isn't an rtl_433 sensor (si7021/esphome), but it would be bad to clamp resolution to even 0.1F for it, as it would harm the data. I say this even if I would not expect two of them to agree to 1F.

[DigiH]

No absolutely no capping for anyone and best to pass on the values as being broadcast and received by the stations, but with the very small scale of the UV index, from 0-16, but realistically for most people around the world from 0-10, an accuracy of ±2 within that small scale and at the same time presenting values with a 0.1 precision I find quite quite embarrassing - not on rtl_433's behalf, but from the manufacturers of these stations. And it definitely makes me stick to my 'purely marketing' decision opinion, as most users will take these values at face value, without ever questioning the science behind it.

[merbanan]

My opinion is that rtl_433 should be usable out of the box and relay as raw data as possible. Most often what the station receiver displays is what rtl_433 also should output. This does not work always but should be the guiding principle.

If we now look at what Wikipedia writes about UV Index we get this:
The UV index is designed as an open-ended, directly proportional to the intensity of UV radiation, and adjusting for wavelength based on what causes [human skin](https://en.wikipedia.org/wiki/Human_skin) to sunburn.

So by the name index is confusing to me and made me believe that it should have been an integer but is looks like it should be a float and then rtl_433 also should output a float value, even if sensors might output dubious readings.

Patch LGTM.

[zuckschwerdt]

I've asked the original author of the value to UVI mapping for some input with #764 (comment)

@DigiH you have the WH24, right? Can you confirm or improve on the derived linear formula to replace the indexed mapping?

[DigiH]

I really didn't want to necessarily propose any changes, but by now you should all know I am a stickler when it comes to accuracy and consistency 😜 otherwise I wouldn't have made three tries to get the battery ambiguity and now the UV index on the plate - I know there was at least another inconsistency, but for the life of me I cannot remember what it was, so it'll have to wait until I stumble across it again.

It's just that from my past job and projects I found it best to voice anything which I find peculiar, strange or just a bit unusual. Not so much for any immediate necessary action in all cases, but for everyone to throw in their ideas as well, and especially for the awareness for any future similar implementations and/or occurrences.

you have the WH24, right?

I have the WH65B , or more precisely a Froggit rebrand which gets recognised and decoded as a FineOffset WH65B. So yes, I suppose it is the same implementation as the WH24.

But still, any UV sensor array being used for proper scientific observations costs more than a few of our weather stations put together, and the only time, if at all, a weather station sensor will report a true to actual value is only during the short time of the day when it is, if at all, totally perpendicular to the sun's altitude and azimuth. Much the same as you can calculate the maximum theoretical solar power throughout the day, but your solar panels can only receive and use that maximum during that time of day when they are also directly perpendicular to the sun - unless you constantly tilt and rotate your weather station/solar panels to follow the sun.

I read you comment and yes, the 5029 to 5230 jump looks like a typo, but I'd also not spend too much time in trying to fine tune an already not totally accurate base reading, due to sensor restrictions. And so far the UV Index for my WH65B has been looking fine - not that I monitor it on a daily basis - and luckily there's never been any occurrence of 13 or even 12 in my parts of the world. I'll keep a closer eye on it however.

[DigiH]

Can you confirm or improve on the derived linear formula to replace the indexed mapping?

The only other thing I'm getting more and more confused with today is the actual UV Index scale range, which for the WH24/WH65B is 0-13, for the FineOffset WS90 above it's 1-15, the Davis Vantage Pro2 0-16 and the Wikipedia entry stating 0-11+ - open-ended yes on theory, but some starting at 1 instead of 0, and all stations defining a finite index range , at least in their specs.

So it will definitely be interesting to find out how the original 0-13 scale came to be, especially in the light of other FineOffset UV sensors.

[gdt]

Can you confirm or improve on the derived linear formula to replace the indexed mapping?

The only other thing I'm getting more and more confused with today is the actual UV Index scale range, which for the WH24/WH65B is 0-13, for the FineOffset WS90 above it's 1-15, the Davis Vantage Pro2 0-16 and the Wikipedia entry stating 0-11+ - open-ended yes on theory, but some starting at 1 instead of 0, and all stations defining a finite index range , at least in their specs.

So it will definitely be interesting to find out how the original 0-13 scale came to be, especially in the light of other FineOffset UV sensors.

I'm pretty sure there is no upper limit to the index, and to Benjamin's point calling it 'index' when it's really just a scale is confusing. Depending on where you are on the earth, season, etc. the value will vary. Yes, there's probably a "max observed so far" kind of number, and I bet that is what the top end is. Sort like solar radiation is 0 to 1200 W/m^2. In AU/NZ, the values are higher than in the US due to reduce ozone thickness or something like that.

I've definitely see values less than 1 out of Davis, and 0 at night. So I'd say anybody saying it has a lower limit of 1 is just off.

It's remarkably hard to find anything quantitative in a primary source from WHO, US EPA etc. But US National Weather Service says UVI 1 is 25 mW/m^2 of UV, without giving the wavelength range, and mentions "mid teens", which is all consistent with index has no upper bound in theory but has highest observed values.
https://www.cpc.ncep.noaa.gov/products/stratosphere/uv_index/uv_compute.shtml

[DigiH]

I've also been trying to find more information, and yes, it needs to be open-ended for any space probes, especially solar ones observing and getting closer to the sun.

I also wonder at what index value we couldn't survive at all any longer, knowing that high index unprotected exposure is already very harmful. And with so many sterilisers for phones and gear and hoovers these days implementing high UV radiation to kill germs and bugs, it is quite an interesting subject, but also very elusive.

In that respect I also wonder that, once the 0-13 scale has been confirmed and verified for the WH24 decoder, if the raw uv property would still really be useful to publish. I've always had it present for my WH65B, but always wondered what it actually was, without any identifiable unit, not more really than the encoded index scale.