Issue updating secondary pi-hole gravity database - pihole-FTL service fails

[tchubaba]

Hi.

So today I installed orbital-sync in docker, but ran into an issue when trying to run the service. Well, the first time it ran, it worked fine, but subsequent runs always fail. It fails at updating gravity on the secondary pi-hole instance. I enabled verbose mode, and here's what I get:

1/16/2023, 5:31:00 PM: ✔️ Successfully signed in to http://10.25.50.10/admin!
1/16/2023, 5:31:00 PM: ➡️ Downloading backup from http://10.25.50.10/admin...
1/16/2023, 5:31:00 PM: ✔️ Backup from http://10.25.50.10/admin completed!
1/16/2023, 5:31:00 PM: ➡️ Signing in to http://10.25.50.11/admin...
1/16/2023, 5:31:00 PM: ✔️ Successfully signed in to http://10.25.50.11/admin!
1/16/2023, 5:31:00 PM: ➡️ Uploading backup to http://10.25.50.11/admin...
1/16/2023, 5:31:04 PM: ✔️ Backup uploaded to http://10.25.50.11/admin!
1/16/2023, 5:31:04 PM: Result:
Processed adlist (43 entries)<br>
Processed adlist group assignments (43 entries)<br>
Processed blacklist (exact) (11 entries)<br>
Processed blacklist (regex) (1 entry)<br>
Processed client (1 entry)<br>
Processed client group assignments (1 entry)<br>
Processed local DNS records (6 entries)<br>
Processed black-/whitelist group assignments (34 entries)<br>
Processed group (2 entries)<br>
Processed whitelist (exact) (9 entries)<br>
Processed whitelist (regex) (13 entries)<br>
OK
1/16/2023, 5:31:04 PM: ➡️ Updating gravity on http://10.25.50.11/admin...
1/16/2023, 5:31:38 PM: ⚠ Error: Failed updating gravity on "http://10.25.50.11/admin".
1/16/2023, 5:31:38 PM: {"host":"http://10.25.50.11","path":"/admin","status":200,"eventStream":"data:   [i] Neutrino emissions detected... \n <------  [✓] Pulling blocklist source list into range \n  \n   [i] Preparing new gravity database...\n <------  [✓] Preparing new gravity database \n   [i] Using libz compression  \n   [i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts \n   [i] Status: Pending...\n <------  [✓] Status: Retrieval successful \n   [i] Analyzed 168834 domains, 15 domains invalid!       Sample of invalid domains:       - 0.0.0.0 \n       - broadcasthost       - fe       - ff \n   [i] List stayed unchanged \n  \n   [i] Target: http://sysctl.org/cameleon/hosts \n   [i] Status: Pending...\n <------  [✓] Status: No changes detected \n   [i] Analyzed 20562 domains, 1 domains invalid!       Sample of invalid domains: \n       - localhost \n  \n   [i] Target: https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt \n   [i] Status: Pending...\n <------  [✓] Status: No changes detected \n   [i] Analyzed 34 domains \n  \n   [i] Target: https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt \n   [i] Status: Pending...\n <------  [✓] Status: No changes detected \n   [i] Analyzed 2701 domains \n  \n   [i] Target: https://raw.githubusercontent.com/kboghdady/youTube_ads_4_pi-hole/master/black.list \n   [i] Status: Pending...\n <------  [✓] Status: Retrieval successful \n   [i] Analyzed 16848 domains \n   [i] List stayed unchanged \n  \n   [i] Target: https://github.com/Perflyst/PiHoleBlocklist/blob/master/SmartTV.txt \n   [i] Status: Pending...\n <------  [✓] Status: Retrieval successful \n   [i] Analyzed 1030 domains, 1030 domains invalid!       Sample of invalid domains: \n       - \"       - \">       - +14       - 0;\"> \n       - 175px\"> \n   [i] List has been updated \n  \n   [i] Target: https://github.com/Perflyst/PiHoleBlocklist/blob/master/AmazonFireTV.txt \n   [i] Status: Pending...\n <------  [✓] Status: Retrieval successful \n   [i] Analyzed 665 domains, 665 domains invalid! \n       Sample of invalid domains:       - \"       - \"> \n       - 0;\"> \n       - 175px\"> \n       - 1rem;\"> \n   [i] List has been updated \n  \n   [i] Target: https://github.com/Perflyst/PiHoleBlocklist/blob/master/android-tracking.txt \n   [i] Status: Pending...\n <------  [✓] Status: Retrieval successful \n   [i] Analyzed 687 domains, 686 domains invalid!       Sample of invalid domains: \n       - \"       - \">       - 0;\">       - 175px\"> \n       - 1rem;\"> \n   [i] List stayed unchanged \n  \n   [i] Target: https://github.com/Perflyst/PiHoleBlocklist/blob/master/SessionReplay.txt \n   [i] Status: Pending...\n <------  [✓] Status: Retrieval successful \n   [i] Analyzed 539 domains, 539 domains invalid! \n       Sample of invalid domains: \n       - \" \n       - \"> \n       - 0;\">       - 175px\">       - 1rem;\"> \n   [i] List has been updated \n  \n   [i] Target: https://adaway.org/hosts.txt \n   [i] Status: Pending...\n <------  [✓] Status: No changes detected \n   [i] Analyzed 7357 domains, 2 domains invalid!       Sample of invalid domains: \n       - localhost \n  \n   [i] Target: https://v.firebog.net/hosts/AdguardDNS.txt \n   [i] Status: Pending...\n <------  [✓] Status: No changes detected \n   [i] Analyzed 47533 domains \n  \n   [i] Target: https://raw.githubusercontent.com/anudeepND/blacklist/master/adservers.txt \n   [i] Status: Pending...\n <------  [✓] Status: Retrieval successful \n   [i] Analyzed 42536 domains \n   [i] List stayed unchanged \n  \n   [i] Target: https://v.firebog.net/hosts/Easylist.txt \n   [i] Status: Pending...\n <------  [✓] Status: No changes detected \n   [i] Analyzed 20164 domains, 3 domains invalid!       Sample of invalid domains: \n       - clickbicom       - snortedbincom       - webytom \n  \n   [i] Target: https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&amp;amp;amp;showintro=0&amp;amp;amp;mimetype=plaintext \n   [i] Status: Pending...\n <------  [✓] Status: No changes detected \n   [i] Analyzed 3666 domains \n  \n   [i] Target: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/UncheckyAds/hosts \n   [i] Status: Pending...\n <------  [✓] Status: Retrieval successful \n   [i] Analyzed 9 domains \n   [i] List stayed unchanged \n  \n   [i] Target: https://raw.githubusercontent.com/bigdargon/hostsVN/master/hosts \n   [i] Status: Pending...\n <------  [✓] Status: Retrieval successful \n   [i] Analyzed 17185 domains \n   [i] List stayed unchanged \n  \n   [i] Target: https://raw.githubusercontent.com/jdlingyu/ad-wars/master/hosts \n   [i] Status: Pending...\n <------  [✓] Status: Retrieval successful \n   [i] Analyzed 1678 domains, 2 domains invalid!       Sample of invalid domains: \n       - localhost \n   [i] List stayed unchanged \n  \n   [i] Target: https://v.firebog.net/hosts/Easyprivacy.txt \n   [i] Status: Pending...\n <------  [✓] Status: No changes detected \n   [i] Analyzed 15228 domains \n  \n   [i] Target: https://v.firebog.net/hosts/Prigent-Ads.txt \n   [i] Status: Pending...\n <------  [✓] Status: No changes detected \n   [i] Analyzed 3749 domains \n  \n   [i] Target: https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-blocklist.txt \n   [i] Status: Pending...\n <------  [✓] Status: Retrieval successful \n   [i] Analyzed 16482 domains \n   [i] List stayed unchanged \n  \n   [i] Target: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.2o7Net/hosts \n   [i] Status: Pending...\n <------  [✓] Status: Retrieval successful \n   [i] Analyzed 2030 domains \n   [i] List stayed unchanged \n  \n   [i] Target: https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt \n   [i] Status: Pending...\n <------  [✓] Status: Retrieval successful \n   [i] Analyzed 347 domains \n   [i] List stayed unchanged \n  \n   [i] Target: https://www.github.developerdan.com/hosts/lists/ads-and-tracking-extended.txt \n   [i] Status: Pending...\n <------  [✓] Status: No changes detected \n   [i] Analyzed 429101 domains \n  \n   [i] Target: https://hostfiles.frogeye.fr/firstparty-trackers-hosts.txt \n   [i] Status: Pending...\n <------  [✓] Status: No changes detected \n   [i] Analyzed 14071 domains \n  \n   [i] Target: https://hostfiles.frogeye.fr/multiparty-trackers-hosts.txt \n   [i] Status: Pending...\n <------  [✓] Status: No changes detected \n   [i] Analyzed 15513 domains \n  \n   [i] Target: https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/android-tracking.txt \n   [i] Status: Pending...\n <------  [✓] Status: Retrieval successful \n   [i] Analyzed 80 domains \n   [i] List stayed unchanged \n  \n   [i] Target: https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/SmartTV.txt \n   [i] Status: Pending...\n <------  [✓] Status: Retrieval successful \n   [i] Analyzed 225 domains \n   [i] List stayed unchanged \n  \n   [i] Target: https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/AmazonFireTV.txt \n   [i] Status: Pending...\n <------  [✓] Status: Retrieval successful \n   [i] Analyzed 17 domains \n   [i] List stayed unchanged \n  \n   [i] Target: https://raw.githubusercontent.com/DandelionSprout/adfilt/master/Alternate%20versions%20Anti-Malware%20List/AntiMalwareHosts.txt \n   [i] Status: Pending...\n <------  [✓] Status: Retrieval successful \n   [i] Analyzed 12696 domains, 2 domains invalid! \n       Sample of invalid domains:       - 2600:1f18:43d1:2a04:ee05:261e:d408:7ba1 \n       - 2a02:1668:1034::2 \n   [i] List stayed unchanged \n  \n   [i] Target: https://osint.digitalside.it/Threat-Intel/lists/latestdomains.txt \n   [i] Status: Pending...\n <------  [✓] Status: No changes detected \n   [i] Analyzed 63 domains \n  \n   [i] Target: https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt \n   [i] Status: Pending...\n <------  [✓] Status: No changes detected \n   [i] Analyzed 2736 domains, 1 domains invalid! \n       Sample of invalid domains:       - disconnect \n  \n   [i] Target: https://v.firebog.net/hosts/Prigent-Malware.txt \n   [i] Status: Pending...\n <------  [✓] Status: No changes detected \n   [i] Analyzed 42043 domains \n  \n   [i] Target: https://www.malwaredomainlist.com/hostslist/hosts.txt \n   [i] Status: Pending...\n <------  [✓] Status: No changes detected \n   [i] Analyzed 1 domains, 1 domains invalid!       Sample of invalid domains: \n       - localhost \n  \n   [i] Target: https://bitbucket.org/ethanr/dns-blacklists/raw/8575c9f96e5b4a1308f2f12394abd86d0927a4a0/bad_lists/Mandiant_APT1_Report_Appendix_D.txt \n   [i] Status: Pending...\n <------  [✓] Status: No changes detected \n   [i] Analyzed 2046 domains \n  \n   [i] Target: https://phishing.army/download/phishing_army_blocklist_extended.txt \n   [i] Status: Pending...\n <------  [✓] Status: No changes detected \n   [i] Analyzed 96470 domains \n  \n   [i] Target: https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-malware.txt \n   [i] Status: Pending...\n <------  [✓] Status: Retrieval successful \n   [i] Analyzed 909 domains \n   [i] List stayed unchanged \n  \n   [i] Target: https://raw.githubusercontent.com/Spam404/lists/master/main-blacklist.txt \n   [i] Status: Pending...\n <------  [✓] Status: Retrieval successful \n   [i] Analyzed 8146 domains \n   [i] List stayed unchanged \n  \n   [i] Target: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Risk/hosts \n   [i] Status: Pending...\n <------  [✓] Status: Retrieval successful \n   [i] Analyzed 2189 domains \n   [i] List stayed unchanged \n  \n   [i] Target: https://urlhaus.abuse.ch/downloads/hostfile/ \n   [i] Status: Pending...\n <------  [✓] Status: No changes detected \n   [i] Analyzed 566 domains \n  \n   [i] Target: https://raw.githubusercontent.com/HorusTeknoloji/TR-PhishingList/master/url-lists.txt \n   [i] Status: Pending...\n <------  [✓] Status: Retrieval successful \n   [i] Analyzed 826176 domains, 38 domains invalid!       Sample of invalid domains: \n       - 123adsfsa14       - 974gfbjhb23hbfkyfaby       - alsgus132 \n       - bestbuycom \n   [i] List stayed unchanged \n  \n   [i] Target: https://codeberg.org/spootle/blocklist/raw/branch/master/blocklist.txt \n   [i] Status: Pending...\n <------  [✓] Status: No changes detected \n   [i] Analyzed 117240 domains, 11 domains invalid!       Sample of invalid domains: \n       - ::1ip6-localhostip6-loopback       - ::1localhost       - fe00::0ip6-localnet \n       - ff00::0ip6-mcastprefix \n       - ff02::1ip6-allnodes \n  \n   [i] Target: https://raw.githubusercontent.com/0Zinc/easylists-for-pihole/master/easylist.txt \n   [i] Status: Pending...\n <------  [✓] Status: Retrieval successful \n   [i] Analyzed 20211 domains \n   [i] List stayed unchanged \n  \n   [i] Target: https://raw.githubusercontent.com/0Zinc/easylists-for-pihole/master/easyprivacy.txt \n   [i] Status: Pending...\n <------  [✓] Status: Retrieval successful \n   [i] Analyzed 16340 domains \n   [i] List stayed unchanged \n  \n   [i] Creating new gravity databases...\n <------  [✓] Creating new gravity databases \n   [i] Storing downloaded domains in new gravity database...\n <------  [✓] Storing downloaded domains in new gravity database \n   [i] Building tree...\n <------  [✓] Building tree   [i] Swapping databases...\n <------  [✓] Swapping databases \n   [✓] The old database remains available. \n   [i] Number of gravity domains: 1993707 (1639554 unique domains) \n   [i] Number of exact blacklisted domains: 2 \n   [i] Number of regex blacklist filters: 1 \n   [i] Number of exact whitelisted domains: 8 \n   [i] Number of regex whitelist filters: 13 \n   [i] Cleaning up stray matter...\n <------  [✓] Cleaning up stray matter \n   [✗] Job for pihole-FTL.service failed. See \"systemctl status pihole-FTL.service\" and \"journalctl -xe\" for details. \n  \n   [✗] DNS service is NOT running"}
1/16/2023, 5:31:38 PM: ⚠ Failure: 0/1 hosts synced.
1/16/2023, 5:31:38 PM: Waiting 30 minutes...

It does this every time it runs now. When this happens it makes the pihole-FTL.service in the secondary pihole fail with these messages:

pihole-FTL.service - Pi-hole FTL
     Loaded: loaded (/etc/systemd/system/pihole-FTL.service; enabled; vendor preset: enabled)
     Active: failed (Result: start-limit-hit) since Mon 2023-01-16 11:31:04 CST; 4min 27s ago
    Process: 1482 ExecStartPre=/opt/pihole/pihole-FTL-prestart.sh (code=exited, status=0/SUCCESS)
    Process: 1495 ExecStart=/usr/bin/pihole-FTL -f (code=exited, status=0/SUCCESS)
    Process: 1540 ExecStopPost=/opt/pihole/pihole-FTL-poststop.sh (code=exited, status=0/SUCCESS)
   Main PID: 1495 (code=exited, status=0/SUCCESS)
        CPU: 96ms

Jan 16 11:31:04 DietPi systemd[1]: Stopped Pi-hole FTL.
Jan 16 11:31:04 DietPi systemd[1]: pihole-FTL.service: Start request repeated too quickly.
Jan 16 11:31:04 DietPi systemd[1]: pihole-FTL.service: Failed with result 'start-limit-hit'.
Jan 16 11:31:04 DietPi systemd[1]: Failed to start Pi-hole FTL.
Jan 16 11:31:04 DietPi systemd[1]: pihole-FTL.service: Start request repeated too quickly.
Jan 16 11:31:04 DietPi systemd[1]: pihole-FTL.service: Failed with result 'start-limit-hit'.
Jan 16 11:31:04 DietPi systemd[1]: Failed to start Pi-hole FTL.
Jan 16 11:31:38 DietPi systemd[1]: pihole-FTL.service: Start request repeated too quickly.
Jan 16 11:31:38 DietPi systemd[1]: pihole-FTL.service: Failed with result 'start-limit-hit'.
Jan 16 11:31:38 DietPi systemd[1]: Failed to start Pi-hole FTL.

After this happens, I have to manually restart the pihole-FTL service for it to work again. Unfortunately I can't make use of orbital-sync with this issue. Any ideas what could be causing this?

[mattwebbio]

Sorry, wanted to at least respond to show I looked at this but I did a little looking around and was unable to figure out what may be causing this. It seems to be a Pi-hole issue, but I have no idea why using the teleporter would trigger it. I would try replicating it yourself by using the teleporter (perform backup in the admin UI on the primary and restore in the admin UI on the secondary) and if you experience the same issue, try creating an issue with Pi-hole.

[tchubaba]

Yeah, I think you may have been right. I tried what you suggested (restoring backup from the primary on the secondary via teleporter) and I got this error:

 Error
There was a problem applying your settings.
Debugging information:
PHP error (2): fsockopen(): unable to connect to 127.0.0.1:4711 (Connection refused) in /var/www/html/admin/scripts/pi-hole/php/FTL.php:47

Which also caused the pihole-FTL.service to fail with the same error message. I will reach out to the folks at Pi-hole directly then to see what may be causing this issue. Thanks!

[ch33f]

simple set SYNC_LOCALDNSRECORDS and SYNC_LOCALCNAMERECORDS to false.
It seems that teleporter has Problems with importing Local DNS Records