Skip to content

minhhungit/github-action-rss-crawler

Folders and files

NameName
Last commit message
Last commit date

Latest commit

38f1a5d · Sep 9, 2024
Aug 16, 2020
Sep 9, 2024
Sep 21, 2021
Aug 15, 2020
Aug 15, 2020
Feb 17, 2021
Sep 9, 2024
Sep 9, 2024
Aug 15, 2020

Repository files navigation

Rss auto crawling using Github Action

Github Action does all these steps automatically, it run rss crawler every 4 hours

Steps:

  • Github will pull repository, build and run crawler code (crawling code is C# (.net core), github will run it directly)
  • Read channel urls from LiteDB
  • Fetch rss feed items
  • Insert feed items into LiteDB after checking blacklist and existing
  • Generate all rss items to static page (index.html - https://minhhungit.github.io/github-action-rss-crawler/ )
  • Commit change (litedb database & index.html page) and push to this repo

Workflow

on:
  schedule:
    # Runs every 4h
    - cron: '0 */4 * * *'
  workflow_dispatch:
  
jobs:
  update-readme-with-blog:
    name: Crawl rss and generate static page
    runs-on: windows-2019
    steps:
      - uses: actions/checkout@main
        with:
          repository: minhhungit/github-action-rss-crawler
          token: ${{ secrets.GITHUB_TOKEN }}
      - uses: actions/setup-dotnet@v1
        with:
          dotnet-version: 3.1.x
      #- run: dotnet build DemoApp\DemoApp.sln      
      - run: dotnet run --project RssCrawler\RssCrawler.csproj
      - run: git config --local user.email "[email protected]"
      - run: git config --local user.name "Jin"
      - run: git add .
      - run: git commit -m "Add changes"
      - run: git push

Demo

https://minhhungit.github.io/github-action-rss-crawler/

Donate ^^

If you like my works and would like to support then you can buy me a coffee ☕️ anytime

Buy Me a Coffee at ko-fi.com

I would appreciate it!!!