Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add support for google adsense user agents #170

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

nitishmittal17
Copy link

Added support for google adsense user agents. Fix for issue - #167

@thoop
Copy link
Contributor

thoop commented Mar 28, 2019

Since we remove <script> tags from the page and I know some of those check for the ad tracking script to approve ads, I'd want to be really careful here about adding those.

We haven't heard of any issues caused by not having those user agents being checked in the middleware. Did you experience any issues there?

@jirikrepl
Copy link
Contributor

jirikrepl commented Sep 11, 2020

That's a good point, if that's true that you are removing <scripts> just adding User-agent: Mediapartners-Google probably would not solve the crawling issue with AdSense.

As they wrote here: https://support.google.com/adsense/answer/99376?hl=en&ref_topic=1348129

The crawler will attempt to access URLs only where our ad tags are implemented.
Only pages displaying Google ads should be sending requests to our systems and being crawled.

It seems to me, that AdSense will not work properly - Adsense crawler will not be able to crawl a website rendered with client-side JavaScript framework (e.g. Angular, Meteor with Blaze).

Screenshot 2020-09-11 at 18 13 57

Do you guys have some ideas? Maybe create a special way how to handle AdSense crawler in prerender?

jirikrepl added a commit to jirikrepl/galaxy-seo-package that referenced this pull request Sep 13, 2020
jirikrepl added a commit to jirikrepl/prerender-node that referenced this pull request Jan 15, 2022
beware of:
prerender#170

test only on typewizz.com
@jirikrepl
Copy link
Contributor

jirikrepl commented Jan 16, 2022

Since we remove <script> tags from the page and I know some of those check for the ad tracking script to approve ads, I'd want to be really careful here about adding those.

We haven't heard of any issues caused by not having those user agents being checked in the middleware. Did you experience any issues there?

My site couldn't be approved by AdSense, because of the error no valuable content.
Adsense bot saw only a bit of initial HTML (body with one script tag, I am using Meteor).

Only after I add those user agents to prerender-node, and triggered the review process again my site was approved.

It's a good point with having those <script> tags removed.
Hard to say now when my site is approved if they check something.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants