Lights brightness goes low and than back without HA even seeing it

[RensRutten]

Describe the bug and how to reproduce it:
The adaptive light works, it changes to the right colours and brightness. I am using Tadfri lights with Zigbee. But after like 1-2 minutes, the brightness of the light goes down, like I assume 10% maybe, and then slightly it goes back to the appropriate brightness. In HA and Zigbee the brightness isn't changed, so i can't see to which brightness it goes. Nothing shows that the brightness changes, only the light does.
In my HA I am not sending any brightness if the light starts or something, I have tried many configurations for Adaptive light. Even uninstalled and installed again, but nothing seems to work, and I can't seem to figure out what is triggering it.

It did this on all Bulbs and spots, so it was not just 1 place..

All help is appreciated.

Steps to reproduce the behavior:
If the light is turned on, and I wait it goes low after some time, so i can check that every time, i can't pinpoint and exact time.

(If using Zigbee2MQTT), provide your configuration files (remove all personal information before posting):
Iam using Zigbee, So i have unistalled Adaptive light and now only have it on 1 light, but it did it on all lights, i will post down here the configuration of the 1 bulb iam trying to fix now, if i get that working i think it would work on others. It also chagned t
"devices": {

        "0x2c1165fffe1120d7": {
            "friendly_name": "Bulb Toilet"
        },

"groups": {
"
"5": {
"devices": [
"0x2c1165fffe1120d7/1"
],
"friendly_name": "Group_Toilet_light"
},
"mqtt": {
"base_topic": "zigbee2mqtt",
"force_disable_retain": false,
"include_device_information": false,
},
"ota": {
"disable_automatic_update_check": false,
"update_check_interval": 1440
},
"passlist": [],
"permit_join": false,
"serial": {
"adapter": "ezsp",
"disable_led": false,
}
Conf of the light
{
"brightness": 188,
"color": {
"x": 0.5056,
"y": 0.4152
},
"color_mode": "color_temp",
"color_options": {
"execute_if_off": true
},
"color_temp": 454,
"color_temp_startup": 65535,
"linkquality": 196,
"power_on_behavior": "off",
"state": "OFF",
"update": {
"installed_version": 268572245,
"latest_version": 268572245,
"state": "idle"
},
"update_available": null
}

It's the TRADFRI bulb E27 CWS 806lm

If I am missing something, please let me know.