Patchman server 3.0.13 : Today's client reports are not processed

[Aleksey-Maksimov]

Hello,

After updating the Patchman server to the latest version 3.0.13, we noticed one strange thing. All of today's client reports (clients send a report to the server every morning from 6 to 7 am) are not processed by the server at 7:15.

Only old reports from yesterday are displayed as processed

In the server log, we only see information about old reports that were already processed yesterday.

...
Dec 25 07:15:01 PM-SRV patchman[387757]: Processing Reports for all Hosts
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1270 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1271 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1272 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1273 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1274 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1275 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1276 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1277 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1278 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1279 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1280 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1281 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1282 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1283 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1284 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1285 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1286 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1287 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1288 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1289 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1290 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1291 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1292 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1293 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1294 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1295 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1296 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1297 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1298 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1299 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1300 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1301 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1302 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1303 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1304 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1305 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1306 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1307 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1308 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1309 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1310 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1311 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1312 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1313 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1314 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1315 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1316 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1317 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1318 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1319 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1320 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1321 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1322 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1323 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1324 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1325 has already been processed
Dec 25 07:15:01 PM-SRV patchman[387757]: Report 1326 has already been processed
Dec 25 07:15:02 PM-SRV patchman[387757]: No orphaned Packages found.
Dec 25 07:15:02 PM-SRV patchman[387757]: No orphaned Package names found.
Dec 25 07:15:02 PM-SRV patchman[387757]: No orphaned Package Architectures found.
Dec 25 07:15:02 PM-SRV patchman[387757]: No orphaned Machine Architectures found.
Dec 25 07:15:02 PM-SRV patchman[387757]: No Repositories with zero Mirrors found.
Dec 25 07:15:02 PM-SRV patchman[387757]: No orphaned Modules found.
Dec 25 07:15:02 PM-SRV patchman[387757]: Refreshing metadata for all Repos
...

[furlongm]

What OS is patchman running on?

Is it running celery?

Do the reports get processed eventually?

[Aleksey-Maksimov]

What OS is patchman running on?

Debian 12

Is it running celery?

No, we did not install Celery because the documentation states that it is an optional component.

Do the reports get processed eventually?

Initially we deployed the server version v3.0.10.
Our configuration was described here:
https://blog.it-kb.ru/2024/11/19/deploying-patchman-linux-packages-updates-status-monitoring-system-on-debian-12/
On this version, today's client reports were processed successfully.
Later we updated from the repositories to the latest version 3.0.13 and after that we noticed that today's reports are not processed. The next day these (already the day before yesterday) reports will be processed, but reports for today are not processed today. This problem did not exist previously on version 3.0.10.

[furlongm]

The changes between 3.0.10 and 3.0.13 are minimal and should have no impact:

v3.0.10...v3.0.13

What is likely happening is that you are seeing outdated information in the web interface due to memcached.

You can reduce the cache timeout or you can just disable memcached completely.

See #575 (comment) for details.

[Aleksey-Maksimov]

My /etc/patchman/local_settings.py config file already contains "CACHE_MIDDLEWARE_SECONDS = 30". But this does not change the situation. All today's reports are shown as unprocessed.

[furlongm]

Are they actually unprocessed though? Or just showing as unprocessed in the web interface?

If you disable caching completely, does that fix the issue? NB you'll probably need to restart the webserver for the settings to take effect.

[Aleksey-Maksimov]

Are they actually unprocessed though? Or just showing as unprocessed in the web interface?

How can I check this?

If you disable caching completely, does that fix the issue?

How to disable caching completely?

NB you'll probably need to restart the webserver for the settings to take effect.

I understand that.

[furlongm]

Are they actually unprocessed though? Or just showing as unprocessed in the web interface?

How can I check this?

patchman -p

If you disable caching completely, does that fix the issue?

How to disable caching completely?

systemctl stop memcached is one way.

or via local_settings.py:

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.dummy.DummyCache',
    }
}

CACHES = None might also work

NB you'll probably need to restart the webserver for the settings to take effect.

I understand that.

[Aleksey-Maksimov]

# patchman -d
No orphaned Packages found.
No orphaned Package names found.
No orphaned Package Architectures found.
No orphaned Machine Architectures found.
No Repositories with zero Mirrors found.
No orphaned Modules found.

[Aleksey-Maksimov]

The "CACHES = None" option does not work. This results in a 500 web server error.

But after I disabled caching like this:

CACHES = {
'default': {
'BACKEND': 'django.core.cache.backends.dummy.DummyCache',
}
}

All client reports were processed successfully.
So the problem is that the patchman is somehow not working correctly with caching?

[furlongm]

The web UI is caching the DB results, so when you see unprocessed reports in the UI you are getting an (out-of-date) cached result. The reports are actually processed in the DB (you could also check there to see if they are really processed or not).

[Aleksey-Maksimov]

If caching is enabled in the default configuration, then running the "patchman -a" command processes all reports except today's ones (while the "patchman -p" command indicates that there are no unprocessed reports).

If caching is disabled, then all reports (including today's) are processed when executing the "patchman -a" command

It turns out that with caching enabled, today's reports are simply not processed.

[furlongm]

patchman -p would be a better indicator of unprocessed reports.

[Aleksey-Maksimov]

Sorry, I wrote it wrong. I meant "patchman -p". The problem persists.