Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error raised when there's a valid <base> tag in <head> #61

Open
svenaas opened this issue Nov 5, 2020 · 3 comments
Open

Error raised when there's a valid <base> tag in <head> #61

svenaas opened this issue Nov 5, 2020 · 3 comments

Comments

@svenaas
Copy link

svenaas commented Nov 5, 2020 •
edited
Loading

After several years of happy operation our Cobweb-dependent crawler ran into a page at https://sso.cas.org/ where the <head> contains this <base> tag:

<base href="https://sso.cas.org/"/>

Our log file was reporting

Error loading http://our.example.com/url: undefined method `present?' for "https://sso.cas.org/":String

and I believe I've traced the problem to a bug in Cobweb's lib/content_link_parser.rb. In the code

14    if @doc.at("base[href]")
15      @base_url = @doc.at("base[href]").attr("href").to_s if @doc.at("base[href]").attr("href").to_s.present?
16    end

I believe the second line is intended to be:

15      @base_url = @doc.at("base[href]").attr("href").to_s if @doc.at("base[href]").attr("href").present?

though I haven't been under the hood in Cobweb before and may be misunderstanding what you're trying to do.
@stewartmckee
Copy link
Owner

I'm sorry, i didn't pick this up for some reason.... the to_s is there to make sure the present? function is available. present? is a rails function though, so it may be you're not running within rails. I'm not sure why a rails function is in there as cobweb shouldn't have a dependency on rails. as a workaround, you could try to require the function from ActiveSupport...

require 'active_support/core_ext/string'

which will pull in the present? function I believe. I'll look to remove this dependency shortly.

@svenaas
Copy link
Author

svenaas commented Feb 5, 2021

Thanks, Stewart! Our Cobweb-related code isn't a Rails project, but I use Rails often enough that I didn't think of that explanation. If we need to work around this before you eliminate the dependency we can patch String by loading ActiveSupport's extensions as you suggest.

@stewartmckee
Copy link
Owner

stewartmckee commented Feb 5, 2021 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@svenaas @stewartmckee