Always read data/list.txt as UTF-8 to avoid "ArgumentError: invalid byte sequence in US-ASCII" when parsing it #118
Comments
|
Thankis @dentarg, I'll investigate. Are you able to tell me which line in the definition file is causing the issue? |
|
@weppos I hope this help (I'm in a hurry now, so I haven't checked this too closely) $ LANG= LANGUAGE= LC_ALL= LC_CTYPE= irb
irb(main):001:0> require 'public_suffix' ; list_data = File.read(PublicSuffix::List::DEFAULT_LIST_PATH) ; nil
=> nil
irb(main):002:0> list_data.class
=> String
irb(main):007:0> ctr = 0 ; outside_line = "" ; list_data.each_line { |line| ctr += 1 ; outside_line = line ; line.strip! } ; nil
ArgumentError: invalid byte sequence in US-ASCII
from (irb):7:in `strip!'
from (irb):7:in `block in irb_binding'
from (irb):7:in `each_line'
from (irb):7
from /Users/dentarg/.rubies/ruby-2.2.5/bin/irb:11:in `<main>'
irb(main):008:0> ctr
=> 610
irb(main):009:0> outside_line
=> "\xE5\x85\xAC\xE5\x8F\xB8.cn\n" |
|
This was with 2.0.3: irb(main):010:0> PublicSuffix::List::DEFAULT_LIST_PATH
=> "/Users/dentarg/.gem/ruby/2.2.5/gems/public_suffix-2.0.3/lib/public_suffix/../../data/list.txt" |
|
Hmm... maybe I was naive to believe that everything would be good by $ LANG= LANGUAGE= LC_ALL= LC_CTYPE= irb
irb(main):001:0> require 'public_suffix'
=> true
irb(main):002:0> File.read(PublicSuffix::List::DEFAULT_LIST_PATH, encoding: Encoding::UTF_8).each_line.to_a[610]
=> "\u7F51\u7EDC.cn\n"
irb(main):003:0> File.read(PublicSuffix::List::DEFAULT_LIST_PATH, encoding: Encoding::UTF_8).each_line.to_a[610].strip!
=> "\u7F51\u7EDC.cn"
irb(main):004:0> File.read(PublicSuffix::List::DEFAULT_LIST_PATH).each_line.to_a[610]
=> "\xE7\xBD\x91\xE7\xBB\x9C.cn\n"
irb(main):005:0> File.read(PublicSuffix::List::DEFAULT_LIST_PATH).each_line.to_a[610].strip!
ArgumentError: invalid byte sequence in US-ASCII
from (irb):5:in `strip!'
from (irb):5
from /Users/dentarg/.rubies/ruby-2.2.5/bin/irb:11:in `<main>'
irb(main):006:0> %w(LANG LANGUAGE LC_ALL LC_CTYPE).map { |v| ENV[v] }
=> ["", "", "", ""]$ irb
irb(main):001:0> require 'public_suffix'
=> true
irb(main):002:0> File.read(PublicSuffix::List::DEFAULT_LIST_PATH).each_line.to_a[610]
=> "网络.cn\n"
irb(main):003:0> File.read(PublicSuffix::List::DEFAULT_LIST_PATH).each_line.to_a[610].strip!
=> "网络.cn"
irb(main):004:0> %w(LANG LANGUAGE LC_ALL LC_CTYPE).map { |v| ENV[v] }
=> ["en_US.UTF-8", "en_US.UTF-8", "en_US.UTF-8", "en_US.UTF-8"] |
|
I'm having this problem with version |
|
Bump. Is this project dead? Does anyone have a fork or alternate project where this is working? @weppos |
It is not dead. If your operating environment is set with the correct UTF8 language value, the library will work perfectly. |
|
FWIW, it would seem correct if gem wouldn't depend/be agnostic to any environment setups for nominal operation. |
|
@SeanDunford @aleksandrs-ledovskis feel free to provide a patch and I will review it. So far, the only one that provided a practical help was @dentarg but even him admitted the problem may not be that easy to solve. Frankly, I am reluctant to put any effort into trying to make UTF-8 work because the real solution is to pre-process the list and have it stored in Punycode as this is how names should be managed and compared. It's just not a the top of my priorities right now. PRs are always welcome. |
|
This is still broken in A workaround is setting: |
Looks like
|
|
I'm encountering an error that is probably related to this: domain = PublicSuffix.domain(request.host)
Tenant.find_by!(domain: domain)Raises: Forcing UTF-8 works: domain = PublicSuffix.domain(host).to_s.force_encoding('UTF-8')Ruby: 3.0.0 |
|
Two workarounds below.
ruby -E utf-8 ./foo.rb
require 'public_suffix'
Encoding.default_external = 'utf-8'
puts PublicSuffix.parse('example.com').inspect |
If your environment fails to specify UTF-8, Ruby defaults to US-ASCII and when public_suffix try to parse the list data, it fails:
Passing
encoding: Encoding::UTF_8toFile.readmakes it work, even if the default encoding isn't UTF-8:Related to #94 (maybe the list data has changed since?)
The text was updated successfully, but these errors were encountered: