Prerender: Don't redirect search engines

[ChristopherChudzicki]

Description/Context

We use Prerender.io to with MIT Learn for the purpose of generating social media previews.

Our prerender fastly integration (based on https://docs.prerender.io/docs/fastly-integration) also redirects user agents googlebot, bingbot, and google.

We'd like to not mess with search engines—e.g., google runs Javascript (See [google docs](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics#:~:text=Once%20Google's%20resources%20allow%2C%20a,HTML%20to%20index%20the%20page; also https://vercel.com/blog/how-google-handles-javascript-throughout-the-indexing-process).

Plan/Design

Suggest we remove googlebot and google from the VCL redirects. Maybe bingbot too?

Overview of Google crawlers and fetchers (user agents)

[Ardiea]

Sure this is very straight forward. Just googlebot, bingbot, and google? Any others?

[ChristopherChudzicki]

@Ardiea Google clearly recommends using their JS rendering rather than prerendering (which I guess they call dynamic rendering...maybe because it's "dynamic" in the sense of serving different content to bots vs humans.) I wish I had a better citation for Bing, though https://webmasters.stackexchange.com/a/140252 from 2023 suggests Bing [and only Bing+Google] run JS.

So, yeah, I'd say only googlebot, bingbot, and google.