Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

news:updater:before-update do not work #22

Open
frogueat opened this issue Oct 1, 2020 · 12 comments
Open

news:updater:before-update do not work #22

frogueat opened this issue Oct 1, 2020 · 12 comments

Comments

@frogueat
Copy link

frogueat commented Oct 1, 2020 •
edited
Loading

Hi!

I have a problem with the new updater. Calling
sudo -u www-data nextcloud-news-updater /var/www/nextcloud --loglevel before-update
I get this:
2020-10-01 09:21:43,998 - Nextcloud News Updater - INFO - Running update in an interval of 900 seconds using 10 threads
2020-10-01 09:21:43,998 - Nextcloud News Updater - INFO - Running before update command: php -f /var/www/nextcloud/occ news:updater:before-update
2020-10-01 09:21:44,296 - Nextcloud News Updater - ERROR - Command '['php', '-f', '/var/www/nextcloud/occ', 'news:updater:before-update']' returned non-zero exit status 1.: Trying again in 30 seconds
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/nextcloud_news_updater/api/updater.py", line 69, in run
self.before_update()
File "/usr/local/lib/python3.6/dist-packages/nextcloud_news_updater/api/cli.py", line 73, in before_update
self.cli.run(self.api.before_cleanup_command)
File "/usr/local/lib/python3.6/dist-packages/nextcloud_news_updater/api/cli.py", line 12, in run
return check_output(commands)
File "/usr/lib/python3.6/subprocess.py", line 356, in check_output
**kwargs).stdout
File "/usr/lib/python3.6/subprocess.py", line 438, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['php', '-f', '/var/www/nextcloud/occ', 'news:updater:before-update']' returned non-zero exit status 1.

To tackle it down, I did call
sudo -u www-data php -f /var/www/nextcloud/occ news:updater:before-update
I get this:
An unhandled exception has been thrown:
Error: Call to a member function execute() on int in /var/www/nextcloud/apps/news/lib/Db/NewsMapperV2.php:77
Stack trace:
#0 /var/www/nextcloud/apps/news/lib/Service/FolderServiceV2.php(101): OCA\News\Db\NewsMapperV2->purgeDeleted()
#1 /var/www/nextcloud/apps/news/lib/Service/UpdaterService.php(48): OCA\News\Service\FolderServiceV2->purgeDeleted()
#2 /var/www/nextcloud/apps/news/lib/Command/Updater/BeforeUpdate.php(44): OCA\News\Service\UpdaterService->beforeUpdate()
#3 /var/www/nextcloud/apps/mail/vendor/symfony/console/Command/Command.php(255): OCA\News\Command\Updater\BeforeUpdate->execute()
#4 /var/www/nextcloud/apps/mail/vendor/symfony/console/Application.php(1005): Symfony\Component\Console\Command\Command->run()
#5 /var/www/nextcloud/apps/mail/vendor/symfony/console/Application.php(271): Symfony\Component\Console\Application->doRunCommand()
#6 /var/www/nextcloud/apps/mail/vendor/symfony/console/Application.php(147): Symfony\Component\Console\Application->doRun()
#7 /var/www/nextcloud/lib/private/Console/Application.php(215): Symfony\Component\Console\Application->run()
#8 /var/www/nextcloud/console.php(100): OC\Console\Application->run()
#9 /var/www/nextcloud/occ(11): require_once('/var/www/nextcl...')

I can't find any logs. Im running nc 20 rc2 on an ubuntu 18.04 box (Linux 4.15.0-118-generic x86_64), php 7.4, pip3 and postgresql. Please help

Kind regards, Günther
@john-2000
Copy link

Hi,
have you been able to fix this? Have you been able to get news to update again?

I am seeing a lot non-zero exit status 1 coming up

2020-10-05 11:58:30,560 - Nextcloud News Updater - ERROR - Command '['php', '-f', '/var/www/nextcloud/occ', 'news:updater:after-update']' returned non-zero exit status 1.: Trying again in 30 seconds Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/nextcloud_news_updater/api/updater.py", line 80, in run self.after_update() File "/usr/local/lib/python3.8/dist-packages/nextcloud_news_updater/api/cli.py", line 89, in after_update self.cli.run(self.api.after_cleanup_command) File "/usr/local/lib/python3.8/dist-packages/nextcloud_news_updater/api/cli.py", line 12, in run return check_output(commands) File "/usr/lib/python3.8/subprocess.py", line 411, in check_output return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, File "/usr/lib/python3.8/subprocess.py", line 512, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['php', '-f', '/var/www/nextcloud/occ', 'news:updater:after-update']' returned non-zero exit status 1.

thanks
J

@frogueat
Copy link
Author

frogueat commented Oct 8, 2020

Hi!
The error with before-update is fixed with the update of the news-app. But updating the feeds is for me not working properly and reliable (many outputs). So i'm switching to feedly till the feature has settled ;-)

thanks, G.

Many outputs in one run like this:
2020-10-08 06:58:30,111 - Nextcloud News Updater - ERROR - Command '['php', '-f', '/var/www/nextcloud/occ', 'news:updater:after-update']' returned non-zero exit status 1.: Trying again in 30 seconds
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/nextcloud_news_updater/api/updater.py", line 80, in run
self.after_update()
File "/usr/local/lib/python3.6/dist-packages/nextcloud_news_updater/api/cli.py", line 89, in after_update
self.cli.run(self.api.after_cleanup_command)
File "/usr/local/lib/python3.6/dist-packages/nextcloud_news_updater/api/cli.py", line 12, in run
return check_output(commands)
File "/usr/lib/python3.6/subprocess.py", line 356, in check_output
**kwargs).stdout
File "/usr/lib/python3.6/subprocess.py", line 438, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['php', '-f', '/var/www/nextcloud/occ', 'news:updater:after-update']' returned non-zero exit status 1.

@frogueat
Copy link
Author

frogueat commented Oct 8, 2020 •
edited
Loading

If one wants to test, this is my opml:

<?xml version="1.0" encoding="UTF-8"?>
<opml version="2.0">
  <head>
    <title>Subscriptions</title>
  </head>
  <body>
    <outline title="Nachrichten" text="Nachrichten"/>
    <outline title="Apple" text="Apple"/>
    <outline title="Linux" text="Linux">
      <outline title="Linux-Magazin" text="Linux-Magazin" type="rss" xmlUrl="http://www.linux-magazin.de/" htmlUrl="http://www.linux-magazin.de/"/>
      <outline title="Pro-Linux News" text="Pro-Linux News" type="rss" xmlUrl="http://www.pro-linux.de/rss/index1.xml" htmlUrl="https://www.pro-linux.de/"/>
      <outline title="OMG! Ubuntu!" text="OMG! Ubuntu!" type="rss" xmlUrl="http://www.omgubuntu.co.uk/" htmlUrl="https://www.omgubuntu.co.uk/"/>
      <outline title="Linux und Open Source" text="Linux und Open Source" type="rss" xmlUrl="https://www.heise.de/thema/Linux-und-Open-Source?view=atom" htmlUrl="https://www.heise.de"/>
    </outline>
    <outline title="Linux" text="Linux"/>
    <outline title="Nachrichten" text="Nachrichten">
      <outline title="nachrichten.at - Linz" text="nachrichten.at - Linz" type="rss" xmlUrl="http://www.nachrichten.at/oberoesterreich/linz/" htmlUrl="https://www.nachrichten.at/"/>
      <outline title="derStandard.at › Inland" text="derStandard.at › Inland" type="rss" xmlUrl="https://derstandard.at/?page=rss&amp;ressort=inland" htmlUrl="https://derstandard.at/Inland"/>
    </outline>
    <outline title="Podcasts" text="Podcasts"/>
    <outline title="Tech-Podcasts" text="Tech-Podcasts"/>
    <outline title="IT-News allgemein" text="IT-News allgemein"/>
    <outline title="Tech-Podcasts" text="Tech-Podcasts">
      <outline title="The Morning After" text="The Morning After" type="rss" xmlUrl="http://api.spokenlayer.com/feed/channel/the-morning-after-multi-ext/3c9929b72538c12bd92ac6762f8d798b9d4e8cdca7692ea74f466061d01816cb" htmlUrl="https://www.engadget.com"/>
      <outline title="Late Night Linux" text="Late Night Linux" type="rss" xmlUrl="https://latenightlinux.com/feed/" htmlUrl="https://latenightlinux.com/"/>
      <outline title="Mac-TV.de HD Die TV-Sendung für Apple-Anwender" text="Mac-TV.de HD Die TV-Sendung für Apple-Anwender" type="rss" xmlUrl="http://www.mac-tv.de/RSS_Podcast_HD.lasso" htmlUrl="https://www.mac-tv.de/"/>
      <outline title="Donau Tech Radio - DTR" text="Donau Tech Radio - DTR" type="rss" xmlUrl="https://dtr.fm/feed/mp3/" htmlUrl="https://dtr.fm/"/>
      <outline title="[a:web]" text="[a:web]" type="rss" xmlUrl="https://frogueat:start%[email protected]/aweb/podcast/" htmlUrl="http://hearthis.at/aweb/"/>
      <outline title="Apfeltalk® LIVE! Videopodcast (HD)" text="Apfeltalk® LIVE! Videopodcast (HD)" type="rss" xmlUrl="http://feeds.apfeltalk.de/ATLN-Video" htmlUrl="http://www.apfeltalk.de/"/>
      <outline title="Show-mp3 – Sunday Morning Linux Review" text="Show-mp3 – Sunday Morning Linux Review" type="rss" xmlUrl="http://smlr.us/?feed=rss2&amp;cat=4" htmlUrl="https://smlr.us/"/>
      <outline title="Ubuntu Podcast" text="Ubuntu Podcast" type="rss" xmlUrl="http://ubuntupodcast.org/feed/podcast" htmlUrl="http://ubuntupodcast.org/"/>
      <outline title="c't uplink" text="c't uplink" type="rss" xmlUrl="https://www.heise.de/ct/uplink/ctuplinkvideohd.rss" htmlUrl="https://www.heise.de/ct/"/>
    </outline>
    <outline title="Apple" text="Apple">
      <outline title="MacRumors: Mac News and Rumors - All Stories" text="MacRumors: Mac News and Rumors - All Stories" type="rss" xmlUrl="https://www.macrumors.com/" htmlUrl="https://www.macrumors.com/"/>
      <outline title="Apple Newsroom" text="Apple Newsroom" type="rss" xmlUrl="https://www.apple.com/de/newsroom/" htmlUrl="https://www.apple.com"/>
      <outline title="Apple Keynotes (HD)" text="Apple Keynotes (HD)" type="rss" xmlUrl="http://podcasts.apple.com/apple_keynotes_hd/apple_keynotes_hd.xml" htmlUrl="http://www.apple.com/"/>
    </outline>
    <outline title="Podcasts" text="Podcasts">
      <outline title="Ö1 Contra" text="Ö1 Contra" type="rss" xmlUrl="https://files.orf.at/podcast/oe1/oe1_contra.xml" htmlUrl="https://oe1.orf.at/podcast/"/>
      <outline title="OMR Daily" text="OMR Daily" type="rss" xmlUrl="https://omr.com/de/feed/" htmlUrl="https://omr.com/"/>
      <outline title="Mac &amp; i" text="Mac &amp; i" type="rss" xmlUrl="https://www.heise.de/mac-and-i/mac-and-i-audio-podcast.rss" htmlUrl="https://www.heise.de/mac-and-i/"/>
      <outline title="Das Coronavirus-Update mit Christian Drosten" text="Das Coronavirus-Update mit Christian Drosten" type="rss" xmlUrl="https://www.ndr.de/nachrichten/info/podcast4684.xml" htmlUrl="https://www.ndr.de/nachrichten/info/podcast4684.html"/>
      <outline title="Ö1 Gedanken für den Tag" text="Ö1 Gedanken für den Tag" type="rss" xmlUrl="https://files.orf.at/podcast/oe1/oe1_gedanken.xml" htmlUrl="https://oe1.orf.at/podcast/"/>
      <outline title="Ö1 Digital Leben" text="Ö1 Digital Leben" type="rss" xmlUrl="http://files.orf.at/podcast/oe1/oe1_digitalleben.xml" htmlUrl="https://oe1.orf.at/podcast/"/>
      <outline title="Ö1 Europa-Journal" text="Ö1 Europa-Journal" type="rss" xmlUrl="https://files.orf.at/podcast/oe1/oe1_europajournal.xml" htmlUrl="https://oe1.orf.at/podcast/"/>
      <outline title="Ö1 matrix" text="Ö1 matrix" type="rss" xmlUrl="http://files.orf.at/podcast/oe1/oe1_matrix.xml" htmlUrl="https://oe1.orf.at/podcast/"/>
      <outline title="extra 3" text="extra 3" type="rss" xmlUrl="http://www.ndr.de/podcast/extradrei196.xml" htmlUrl="https://www.ndr.de/fernsehen/sendungen/extra_3/video-podcast/index.html"/>
      <outline title="Ö1 Wissen aktuell" text="Ö1 Wissen aktuell" type="rss" xmlUrl="http://static.orf.at/podcast/oe1/oe1_wissen.xml" htmlUrl="https://oe1.orf.at/podcast/"/>
      <outline title="Ö1 #doublecheck" text="Ö1 #doublecheck" type="rss" xmlUrl="http://static.orf.at/podcast/oe1/oe1_doublecheck.xml" htmlUrl="https://oe1.orf.at/podcast/"/>
      <outline title="Ö1 Klartext" text="Ö1 Klartext" type="rss" xmlUrl="https://files.orf.at/podcast/oe1/oe1_klartext.xml" htmlUrl="https://oe1.orf.at/podcast/"/>
    </outline>
    <outline title="IT-News allgemein" text="IT-News allgemein">
      <outline title="The Nextcloud Podcast" text="The Nextcloud Podcast" type="rss" xmlUrl="https://nextcloud.com/podcast-feed.rss" htmlUrl="https://nextcloud.com/podcast/"/>
      <outline title="netzpolitik.org" text="netzpolitik.org" type="rss" xmlUrl="https://netzpolitik.org/feed" htmlUrl="https://netzpolitik.org"/>
      <outline title="Golem.de" text="Golem.de" type="rss" xmlUrl="http://rss.golem.de/rss.php?feed=ATOM1.0" htmlUrl="https://www.golem.de/"/>
      <outline title="The latest updates – Nextcloud" text="The latest updates – Nextcloud" type="rss" xmlUrl="https://nextcloud.com/feed/" htmlUrl="https://nextcloud.com/"/>
      <outline title="Engadget RSS Feed" text="Engadget RSS Feed" type="rss" xmlUrl="http://de.engadget.com/rss.xml" htmlUrl="https://www.engadget.com/rss.xml"/>
      <outline title="heise online News" text="heise online News" type="rss" xmlUrl="https://www.heise.de/newsticker/" htmlUrl="https://www.heise.de/newsticker/"/>
      <outline title="Mozilla UX" text="Mozilla UX" type="rss" xmlUrl="https://blog.mozilla.org/ux/feed/" htmlUrl="https://blog.mozilla.org/ux"/>
      <outline title="Release notes from news" text="Release notes from news" type="rss" xmlUrl="https://github.com/nextcloud/news/releases.atom" htmlUrl="https://github.com/nextcloud/news/releases"/>
    </outline>
    <outline title="Artikel ohne Feed" text="Artikel ohne Feed" type="rss" xmlUrl="http://nextcloud/nofeed" htmlUrl="http://nextcloud/nofeed"/>
  </body>
</opml>

@dvzrv
Copy link
Contributor

dvzrv commented Oct 23, 2020

@frogueat I ran into similar issues with version 11.0.0, but it was related to my systemd service file, which didn't allow access to local sockets (AF_UNIX address family).

Starting it with the following permissions/settings works (paths are probably Arch Linux specific):

[Unit]
Description=Update nextcloud news feeds
After=network.target network-online.target

[Service]
CapabilityBoundingSet=
DeviceAllow=
DevicePolicy=closed
Environment=NEXTCLOUD_CONFIG_DIR=/etc/webapps/nextcloud/config
ExecStart=/usr/bin/nextcloud-news-updater -c /etc/webapps/nextcloud/news/nextcloud-news-updater.ini
Group=http
LockPersonality=true
NoNewPrivileges=true
PrivateTmp=true
PrivateDevices=true
PrivateUsers=true
ProtectClock=true
ProtectControlGroups=true
ProtectHome=true
ProtectHostname=true
ProtectKernelLogs=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectSystem=strict
ReadWritePaths=/etc/webapps/nextcloud
RemoveIPC=true
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
RestrictNamespaces=true
RestrictRealtime=true
RestrictSUIDSGID=true
SystemCallArchitectures=native
SystemCallFilter=@system-service
UMask=007
User=http

[Install]
WantedBy=multi-user.target

Only ReadWritePaths needs to be overridden to add the instance's data directory.

@BruderJo
Copy link

Hello,

I found the same issue with the standalone news-updater script after updating to nextcloud 20 and news app 15.0.6.
The issue seems related to the command syntax.

occ news:updater:update-feed <user-id> <feed-id>

After updating lib/python3.6/dist-packages/nextcloud_news_updater/api/cli.py it was running fine again.
You need to switch user_id & feed_id.

*** cli.py.orig	2018-01-08 18:08:15.000000000 +0000
--- cli.py	2020-11-27 13:36:17.654095505 +0000
***************
*** 54,61 ****
          self.api = api
  
      def update_feed(self, feed: Feed) -> None:
!         command = self.api.update_feed_command + [str(feed.feed_id),
!                                                   feed.user_id]
          self.logger.info('Running update command: %s' % ' '.join(command))
          self.cli.run(command)
  
--- 54,60 ----
          self.api = api
  
      def update_feed(self, feed: Feed) -> None:
!         command = self.api.update_feed_command + [feed.user_id, str(feed.feed_id)]
          self.logger.info('Running update command: %s' % ' '.join(command))
          self.cli.run(command)

@BernhardPosselt
Copy link
Member

You need to set the api level, check the README

@bcutter
Copy link

bcutter commented Dec 1, 2020

Hello,

I found the same issue with the standalone news-updater script after updating to nextcloud 20 and news app 15.0.6.
The issue seems related to the command syntax.

occ news:updater:update-feed <user-id> <feed-id>

After updating lib/python3.6/dist-packages/nextcloud_news_updater/api/cli.py it was running fine again.
You need to switch user_id & feed_id.

*** cli.py.orig	2018-01-08 18:08:15.000000000 +0000
--- cli.py	2020-11-27 13:36:17.654095505 +0000
***************
*** 54,61 ****
          self.api = api
  
      def update_feed(self, feed: Feed) -> None:
!         command = self.api.update_feed_command + [str(feed.feed_id),
!                                                   feed.user_id]
          self.logger.info('Running update command: %s' % ' '.join(command))
          self.cli.run(command)
  
--- 54,60 ----
          self.api = api
  
      def update_feed(self, feed: Feed) -> None:
!         command = self.api.update_feed_command + [feed.user_id, str(feed.feed_id)]
          self.logger.info('Running update command: %s' % ' '.join(command))
          self.cli.run(command)

@BruderJo for president. Thanks a lot! I was struggling... #21 (comment) until I found your hotfix.

@bcutter bcutter mentioned this issue Dec 1, 2020
Open
@BernhardPosselt
Copy link
Member

@Grotax  @DriverXX any ideas?

@Grotax
Copy link
Member

Grotax commented Dec 15, 2020

Sooner or later a complete rewrite will be required, as we will deprecate the user api.
I don't know if an external updater is still required.

If someone thinks so please open a discussion in the news repo else I would suggest to archive this project.

@BernhardPosselt
Copy link
Member

@Grotax unless you build in a threaded and threadsafe update mechanism into Nextcloud this thing will still be a hard requirement for medium - large installations. I know of some universities that use it because their Nextcloud cron mechanism is stuck in limbo otherwise.

@Grotax
Copy link
Member

Grotax commented Dec 15, 2020

I see, then someone will need to reimplement something like this.
Maybe the universities are interested to work on this as a project or bachelor thesis.

@dehuszar
Copy link

Has anyone found a way around this? I have successfully updated every feed I manually attempt to update when running statements like: sudo -u www-data php -d memory_limit=-1 -f /var/www/html/occ news:updater:update-feed "sam" 24, but trying to run the updater overall does not work, giving the same before-update error that everyone else is reporting.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@BernhardPosselt @dvzrv @dehuszar @frogueat @Grotax @john-2000 @bcutter @BruderJo