Welcome to crawly Discussions!

[oltarasenko]

👋 Welcome!

We’re using Discussions as a place to connect with other members of our community. We hope that you:

• Ask questions you’re wondering about.
• Share ideas.
• Engage with other community members.
• Welcome others and are open-minded. Remember that this is a community we
  build together 💪.

To get started, comment below with an introduction of yourself and tell us about what you do with this community.

[nuno84]

Hi,
First of all, thank you for your work. This seems a very interesting project and I think this would be usefull for my pet project (I am transitioning from Rails to Phoenix).
I want to load some events from facebook pages, since they blocked the API in 2021 to page owners only.
But the page gets redirected to "allow cookies". How can I pass it?
In this case I am using HTTPoison, which (I think) is how Crawly internally processes GET requests.
Example:

{:ok, %HTTPoison.Response{body: body}} = HTTPoison.get("https://m.facebook.com/TheatroCirco/events/", [], follow_redirect: true)
price = document |> Floki.find("#page") # Gets nothing!

But what I get is the popup for cookies and not the page itself which is loaded afterwards by some JS and gets below the cookies popup.
If you try to open in chrome incognito the content is below, but using the elixir command it is not since it was not loaded (yet).

Thank you very much for any idea/advice you may have to help me overcome this issue.

[Ziinc]

You might want to use the splash fetcher, as Facebook would not load content without js as you have mentioned. The HTTPoison fetcher would not be sufficient to circumvent Facebook's crawling protections.

[nuno84]

Hi Ziinc,
I will have a try on that approach, it seems to be the right answer.
Thank you so much

[nuno84]

It is working now.
By the way, is there any way to pass the option "splash.images_enabled = false" on crawly to splash to avoid loading images (I dont need to load them) .
Couldnt find it on crawly github repo.
Thank you for your help.