External link checking - what am I missing?

[rmoff]

First, thanks for Lychee - I'm a long time fan!

I'm trying to use it with GH actions to check outbound links from my blog, rmoff.net.

If I run lychee https://rmoff.net it only scans 84 pages, complaining only about a couple of twitter links.

$ lychee https://rmoff.net
  84/84 ETA 0s ████████████████████ Finished extracting links
Issues found in 1 input. Find details below.

[https://rmoff.net/]:
✗ [403] https://twitter.com/rmoff/status/1587382202781913089 | Failed: Network error: Forbidden
✗ [403] https://twitter.com/rmoff/ | Failed: Network error: Forbidden

🔍 84 Total ✅ 81 OK 🚫 2 Errors (HTTP:1|Unknown:1)

HOWEVER, I know there are some broken links on https://rmoff.net/2023/10/02/learning-apache-flink-s01e02-what-is-flink/, which it picks up if I run lychee https://rmoff.net/2023/10/02/learning-apache-flink-s01e02-what-is-flink/

lychee https://rmoff.net/2023/10/02/learning-apache-flink-s01e02-what-is-flink/
  115/115 ETA 0s ████████████████████ Finished extracting links
Issues found in 1 input. Find details below.

[https://rmoff.net/2023/10/02/learning-apache-flink-s01e02-what-is-flink/]:
✗ [404] https://flink.apache.org/use-cases/#data-analytics-applicationsa-nameanalyticsa | Failed: Network error: Not Found
✗ [404] https://www.youtube.com/watch?v=izYsMQWeUbE/ | Failed: Network error: Not Found
✗ [404] https://www.youtube.com/watch?v=rJNH5WhWAj4/ | Failed: Network error: Not Found
✗ [404] https://www.youtube.com/watch?v=_yHds9SvMfE/ | Failed: Network error: Not Found
✗ [406] https://eng.uber.com/athenax/ | Failed: Network error: Not Acceptable
✗ [404] https://flink.apache.org/roadmap/#feature-radar | Failed: Network error: Not Found
✗ [404] https://flink.apache.org/use-cases/ | Failed: Network error: Not Found
✗ [404] https://flink.apache.org/powered-by/ | Failed: Network error: Not Found
✗ [404] https://www.youtube.com/watch?v=17tUR4TsvpM/ | Failed: Network error: Not Found
✗ [404] https://www.youtube.com/watch?v=0cJ565r2FVI/ | Failed: Network error: Not Found
✗ [403] https://twitter.com/rmoff/ | Failed: Network error: Forbidden
✗ [404] https://flink.apache.org/roadmap/ | Failed: Network error: Not Found
✗ [404] https://flink.apache.org/community/ | Failed: Network error: Not Found
✗ [406] https://www.uber.com/en-GB/blog/real-time-exactly-once-ad-event-processing/ | Failed: Network error: Not Acceptable
✗ [404] https://flink.apache.org/use-cases/#data-pipeline-applications-a-namepipelinesa | Failed: Network error: Not Found
✗ [404] https://www.youtube.com/watch?v=Do7C4UJyWCM/ | Failed: Network error: Not Found
✗ [404] https://flink.apache.org/use-cases/#event-driven-applications-a-nameeventdrivenappsa | Failed: Network error: Not Found
✗ [404] https://flink.apache.org/roadmap/#scenarios-we-focus-on | Failed: Network error: Not Found
✗ [404] https://flink.apache.org/roadmap/#developer-experience | Failed: Network error: Not Found

🔍 115 Total ✅ 93 OK 🚫 19 Errors (HTTP:16|Unknown:3)

What do I need to change in my invocation when I pass the root URL (https://rmoff.net) so that it will crawl all my site and report all broken outbound links?

thanks!

[mre]

Hey Robin!
Recursion is not supported yet. It is the most wanted feature on the list right now.
#78
We had several attempts already, but so far we haven't merged any pull requests due to a lack of architectural coherence. If you have a sitemap, you can use that as an input for now.

[rmoff]

ah, sorry 🤦 I think I remember already asking a similar question in the past!
I'll try with the sitemap. Thanks!