Anyone using the allsky_light module?

[ranous]

I decided to add a TSL2591 sensor to my allsky cam and calculate the SQM value. The value calculated by the allsky_light module is fairly far off from the readings I get from my Unihedron SQM meter. Anyone have better luck with that module, or is it still considered experimental?

Digging into the code, I think the problem is the formula calculating the SQM from the lux reading returned by the sensor. It looks like the formula was taken from some other open source project. That formula has issues. Looks like they baked in a correction factor, most likely to compensate for issues in their system. I tracked down the correct formula for calculating the SQM from lux, and if I use it, I get me pretty close my SQM meter. I added a correction factor to the modules configuration so the user can supply their own scaling correction factor.

The main reason for needing a correction factor is to compensate for light loss due to any lens or filter you might be using. BTW, I've seen a IR cut filter used in some homebrew SQM meter build that use the TSL2591. This isn't necessary with that sensor as it has two sensors, one of which measures the IR component, and it's removed from the lux calculation, so the filter only serves to provide additional light loss to the system.

The other factor that can effect the lux reading is the temperature. According to the TSL2591 datasheet, the value can be effected by the temperature, and you should have a scaling factor based on the temperature delta from some base temperature. I'm uncertain at this point if that is a significant factor or not. There are SQM implementations that take it into account, as does the Unihedron meter. I'll check my results over time as the temperature changes and see if it drifts off from the Unihedron meter's results. If it looks like it's useful, I can add code to optionally read an environment variable from the allsky_temp or other modules that write out the ambient temperature and factor that in.

Another best practice I've seen in SQM implementation is they average several readings together, smoothing out the variances from individual readings.

[EricClaeys]

@Alex-developer, thoughts?

[Alex-developer]

Ranous

I have moved this discussion to the modules repo

It probably should still be experimental. I did compare it against my Primaluce Eagle 4 and 5 and got different results but have never gone back and looked at how to correct the code. I don't have a Unihedron meter :-(. I don't recall where the formula came from but I will check as it should have some attribution.

Totally agree that it should average readings over a period of time, I am adding this to a few of the modules at the moment to help with transitions between states.

If you are happy to share the formula you found that would be great or feel free to do a PR for the module

Thanks

Alex

[ranous]

Let me do some more experiments. I saw more variance over last night from the "real" SQM meter. Perhaps that was due to temperature variance. I also implemented some simple averaging. With my initial implementation, I noticed that all the samples were the same. Looks like the sensor is alway repeatedly measuring based on the last gain/exposure and when you read it, it returns the current result. As I had my exposure time set up to 600ms, all my samples were the same except for a couple times when my set spanned a new measurement. It works much better now that I wait for the last set integration time between readings. This means that you probably don't want too many samples with the longer integration time as it slows down the module. I've been using 5 samples which adds three seconds to the runtime.

Another issue seems to be the way the adafruit library calculates lux. If you compare it with Adafruit's C version, they use different algorithms. The C version had changed several times over the years as different issues have come up with the algorithm. Clearly Adafruit doesn't keep them in sync. Also, there are several issues on the c version github where there's lots of arguing over what the correct algorithm is. Some folks argue that it's not possible to get an accurate lux reading with the TSL2591 w/out calibrating it to a known reference light source. All I know is if I can come up with a scaling function that matches the results with my SQM meter over a range of different temps, I'm happy.

[EricClaeys]

@Alex-developer, if @ranous were to make improvements to the module (hint, hint) I assume we would incorporate them into the release and credit him?

[Alex-developer]

@Alex-developer, if @ranous were to make improvements to the module (hint, hint) I assume we would incorporate them into the release and credit him?

Yes and Yes :-)

[ranous]

Ok, I'm back from traveling and resuming this project again. I'm happy to contribute my changes to the module back. I'm still trying to tune the numbers. The thing I'm working on now is temperature compensation. It's clear from my measurements that the sensor loses sensitivity as the temperature decreases, leading to an extra increase to the SQM calculation. I'm still trying to nail down the coefficient to use. I need a clear night, as my data is pretty noisy due to hazy clouds moving through the field when taking measurements. My sensor and my SQM meter have different field of views, so I need to make sure they're measuring the same thing, and the variations of brightness due to the clouds moving though is throwing everything off. I might have a clear night tonight to try again.

In order for me to implement temp compensation, allsky_light needs the current temp. I was hoping to access the variables from other modules (allsky_temp in this case), but that doesn't seem available from allsky_shared. For now I'm stealing the read_extra_data function from allksy_publishdata, which extracts all the variables from the module datafiles. I'd suggest adding this to the shared module as I think this is not an uncommon occurance, where one module can use the data from another. The allsky_dewheater module is a good example. No need for the code to read the sensor which is already in allsky_temp and can automatically take advantage of any additional sensors added to allsky_temp over time.

[Alex-developer]

Ok, I'm back from traveling and resuming this project again. I'm happy to contribute my changes to the module back. I'm still trying to tune the numbers. The thing I'm working on now is temperature compensation. It's clear from my measurements that the sensor loses sensitivity as the temperature decreases, leading to an extra increase to the SQM calculation. I'm still trying to nail down the coefficient to use. I need a clear night, as my data is pretty noisy due to hazy clouds moving through the field when taking measurements. My sensor and my SQM meter have different field of views, so I need to make sure they're measuring the same thing, and the variations of brightness due to the clouds moving though is throwing everything off. I might have a clear night tonight to try again.

In order for me to implement temp compensation, allsky_light needs the current temp. I was hoping to access the variables from other modules (allsky_temp in this case), but that doesn't seem available from allsky_shared. For now I'm stealing the read_extra_data function from allksy_publishdata, which extracts all the variables from the module datafiles. I'd suggest adding this to the shared module as I think this is not an uncommon occurance, where one module can use the data from another. The allsky_dewheater module is a good example. No need for the code to read the sensor which is already in allsky_temp and can automatically take advantage of any additional sensors added to allsky_temp over time.

Having other module data available is a great idea :-)

I have created an issue to implement this #191

There is a function in the shared module 'setupForCommandLine' that makes all of the 'core' Allsky variables available to modules in the Periodic flow. It will be pretty easy to extend this to also add in the data from extra files.

Thanks

Alex

[Alex-developer]

I have added a function to the shared module that will allow you to read a variable See this PR AllskyTeam/allsky#4122

You could just take a copy of the allsky_shared module from the branch and try the code. If you do take the code you will need to ensure that you are on the latest release first.

The function is pretty simple

result = s.get_allsky_variable("AS_TEMPAMBIENT1")

will get the variable from the temp module, my sample json data from the extra file is below

{
    "AS_GPIOSTATE1": "Off",
    "AS_TEMPSENSOR1": "BME280-I2C",
    "AS_TEMPSENSORNAME1": "ENCLOSURE",
    "AS_TEMPAMBIENT1": "19.96",
    "AS_TEMPDEW1": "3.96",
    "AS_TEMPHUMIDITY1": "34.78",
    "AS_TEMPPRESSURE1": 1035.209603791197,
    "AS_TEMPRELHUMIDITY1": 34.80467138412865,
    "AS_TEMPALTITUDE1": -181.18362536100426
}

[Aripaule]

Hello,

what is the variable in the overlay module to activate the TSL2591 measurement?

[ranous]

The allsky_light module produces three outputs. AS_LIGHTSQM is the SQM measurement, AS_LIGHTLUX is the raw lux measurement, and AS_LIGHTNELM is the naked eye limiting magnitude. In overlays, you'd delete the AS_ bit.

[Aripaule]

The allsky_light module produces three outputs. AS_LIGHTSQM is the SQM measurement, AS_LIGHTLUX is the raw lux measurement, and AS_LIGHTNELM is the naked eye limiting magnitude. In overlays, you'd delete the AS_ bit.

Thank you, for some reason my sensor is not being recognized. I think that's why the variable doesn't appear in the selection menu

[chvvkumar]

I ported over this ESP32 based TSL2591 SQM meter to support Raspberry Pi:

https://github.com/chvvkumar/PiSQM/tree/main

Unfortunately, I don't have a SQM meter to compare the results but eyeballing results from the past week, the values I am getting seem to correlate my sky conditions (i.e. about 16-18 mag/arcsec^2 on overcast nights leaning towards the lower end on snow covered nights where more light reflects back on to the clouds).

If you have an SQM meter, could you see how far off this is from an actual meter?