[SOLVED] How hard would it be to implement something like a custom RSS retriever for Twitter?

[felipe19930]

I didn't want to open a new issue for this because it's mostly a question, but I wonder if RSS Guard provides an easy interface to create custom feeds for websites that don't provide a RSS interface at all, like Twitter.

Thanks!

[martinrotter]

Hello @felipe19930. Discussions are exactly for these kinds of talks, okay.

Generally, at this point we have several options and if they are feasible depends on how exactly you would want the feature to behave:

1. I could implement Script-based website to feed conversion #265 which would allow you to do this:
   1. Download some fixed URL contents either directly automatically via RSS Guard or via calling external arbitrary script (with parameter).
   2. Process downloaded data with custom script inside RSS Guard - the script could "convert" data to ATOM/RSS XML format.
   3. Process resulting XML with standard "plugin".
2. Or we could write custom service plugin specifically designed to work with Twitter, probably via Twitter API.

Difference is that 1. is more general and very flexible solution which will allow you to process all kinds of websites w/o feeds, convert them to feeds and then process feeds while option 2. is specifically designed to only work with Twitter (in a similar fashion like other already existing plugins like Inoreader or Nextcloud News plugin).

Option 1. is meant to be programmed by me and then configured by users, option 2. must be entirely programmed either by me or by some contributor - at least basic knowledge of C++/Qt is needed.

Note that if you would require some advanced Twitter-related functionality like adding +1 to posts or maybe commenting (?) on posts, only option 2. is viable. Also note that there is already existing well-defined RSS Guard plugin API for writing new "plugins" that every at least semi-competent C++ programmer could use.

[martinrotter]

Excerpt of Tweeper's readme:

  41 Tweeper can be used via web or as a command line program, for example as
  42 a filter in your feed reader, by passing the URL of the user's public timeline
  43 as the first argument.
  44 
  45 Example of use on the command line:
  46 
  47   $ php tweeper.php https://twitter.com/NSACareers

[martinrotter]

https://git.ao2.it/tweeper.git/blob/master:/README

[martinrotter]

Btw, in the meantime, I started to work on solution 1 - see #265 to track progress. I will add some examples with twitter and document the whole thing properly later in the documentation.

[felipe19930]

Awesome! Thanks for such a in-depth response too.

Note that if you would require some advanced Twitter-related functionality like adding +1 to posts or maybe commenting (?) on posts, only option 2. is viable.

My use-case is very simple actually: There are some Twitter accounts I want to "follow" (i.e. monitor their tweets), because they often post shopping coupons (or great deals in general), so I'm interested in getting those as quick as possibl,e before they expire. Right now, I'm using a nitter instance to get these tweets in the RSS format, and while it's been working great for a couple of months now, nitter appears to have a 10 minute delay or something before a new tweet appears, which is not that interesting for my use-case.

I like option 1 because, like you said, would also work for other sites, and I know a few that this feature would come in handy.

Tweeper sounds like a great alternative once #265 gets implemented, so I'll start getting to know it better!

Thanks for starting the implementation too, because if we were to depend on my C++ knowledge to do anything, then nothing would be done. 😅

[martinrotter]

@felipe19930 Yes, I am implementing the ticket #265 right now and it is quite serious stuff. I will commit some code soon and expect it to be done in a week. I have actually tried Tweeper now as part if my internal testing and I must say that it works nicely. It takes URL of feed as input and produces RSS feed XML file, although it was quite tricky to install it on Windows. I had to install XAMPP (to get "php.exe"), then I had to install Composer, then install Symfony via Composer, then install Tweeper.

Could you move your next messages directly to #265 as we in fact "solved" this discussion. :)

[martinrotter]

@felipe19930 OK, raw functionality is there. You can now download latest development build and test the feature.

https://github.com/martinrotter/rssguard/blob/master/resources/docs/Feed-formats.md#websites-scraping-and-related-advanced-features

Note that the feature was so far tested only under Windows. I will test with Linux on the evening within a week. If you have any questions, ask.

#265

[martinrotter]

@felipe19930 OK I pushed the changes and development build should arrive there in a 15 minutes. I updated documentation with newest examples which worked for me:

https://github.com/martinrotter/rssguard/blob/master/resources/docs/Feed-formats.md#websites-scraping-and-related-advanced-features

and tested on Windows. I believe that current implementation could also work on Linux and Mac OS but will test tonight.

[felipe19930]

That seems a little convoluted, in my humble opinion. Also, what if for some reason I needed to pass the literal # character as an argument? Would it work?

Anyway, I've looked up on how QuiteRSS does it (well, QuiteRSS doesn't have this feature). To be more specific, I've searched on how their "Open in external browser" feature works, and it appears to handle extra arguments just fine. I have this on my QuiteRSS' settings:

And using strace to monitor it, QuiteRSS appears to handle every argument pretty well:

And I assume this is the code that handles the "Open in external browser" action:

https://github.com/QuiteRSS/quiterss/blob/271c55756dcf19ca163a603a54d12f165548a602/src/newstabwidget.cpp#L2198

Since both projects share the same license, perhaps you could borrow QuiteRSS's code (please forgive me if I'm talking non-sense and this is not applicable for RSS Guard 😅)

[martinrotter]

EDIT: I checked their code. And it is non-usable for our use-case because in their use-case, they simply launch external command and do not care about it, but in our use case we need to grab command's standard output and also in same cases provided data to command's standard input. Moreover, they use windows-specific APIs on Windows.

As for now, I tested the last code on Linux:

It works, but I will check their code.

[felipe19930]

It's okay, if it works, it works!

Thanks a lot ❤️

[martinrotter]

@felipe19930 As for "#" character. Yes, sadly, there is probably no way of using it in your command string. You could probably use it if encoded, for example like this in Bash: echo -e '\u0023' -> "#".

We can certainly pick a different character as separator, just suggest one.

[CsatiZoltan]

What about creating an interface to RSS feed generators that can create a feed link to almost any webpage? Something like https://rss.app/ or a free alternative.

[martinrotter]

@CsatiZoltan At this point, features related to "scraping" are implemented and I tested them like hell. They work nicely.

As for Twitter, I believe there is like 10+ ways of getting Twitter RSS/ATOM/JSON feeds. You can use

https://git.ao2.it/tweeper.git/blob/master:/README

which is also written in docs, specifically this line php#tweeper.php#-v#0#https://twitter.com/NSACareers in RSS Guard feed's settings dialog in "source" attribute with "script" option will do everything needed.

Also, you can probably just scrape the twitter with generic Python-based script with some regexp magic in a similar fashin like these.

As for rss.app if I understand it correctly, it should already work with RSS Guard and pretty much every other feed reader at this point.

creating an interface to RSS feed generators

Could you specify exactly what and how should this interface look/work?

[CsatiZoltan]

@martinrotter

Could you specify exactly what and how should this interface look/work?

I don't have a firm idea, I was just thinking that you could add a push button in the "Add new feed" window that opens a dialogue box, where the user must give the link for which he/she wants the feed. Then RSS Guard would call rss.app or another scraper and fetch the link they generate. So it would be a convenience option that the user wouldn't have to do that manually on rss.app and paste the link, but RSS Guard would do that in the background. Similarly to what it does with tweeper (see the issue below).

For curiosity, I tried the line php#tweeper.php#-v#0#https://twitter.com/NSACareers. It gave me the following error:

time="   185.768" type="debug" -> core: Running custom script for guessing 'php#tweeper.php#-v#0#https://twitter.com/NSACareers' to obtain feed data.
time="   185.806" type="debug" -> core: XML of feed 'php#tweeper.php#-v#0#https://twitter.com/NSACareers' is not valid and cannot be loaded. Error: 'error occurred while parsing element' (line 1, column 1).

As for rss.app if I understand it correctly, it should already work with RSS Guard and pretty much every other feed reader at this point.

Yes, it works, but it only supports two links for a free plan.

[martinrotter]

@CsatiZoltan In order for php#tweeper.php#-v#0#https://twitter.com/NSACareers to work, you need to have PHP installed and configured, as well as Tweeper itself.

The error message you got was bit misleading, fixed that little nuance. Should report better error messages now.

[martinrotter]

@CsatiZoltan As for the idea itself, I believe that the intended feature is out-of-scope for RSS Guard. :) I do not want to add yet another weak dependency to maintain. :)

[CsatiZoltan]

@CsatiZoltan As for the idea itself, I believe that the intended feature is out-of-scope for RSS Guard. :) I do not want to add yet another weak dependency to maintain. :)

Completely understand, I also strive for minimal dependencies in my projects...