Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unicode characters breaks YARD generation #1520

Open
thomthom opened this issue Nov 17, 2023 · 1 comment
Open

Unicode characters breaks YARD generation #1520

thomthom opened this issue Nov 17, 2023 · 1 comment

Comments

@thomthom
Copy link
Contributor

Unicode characters in the Ruby or README files is breaking YARD with incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError

When I used the superscript 3 character (³) in my files, YARD will fail to generate correct output.

I have verified that I saved the files with UTF-8 encoding by inspecting the files with a hex-editor:
image

Steps to reproduce

.yardopts

--title "test³"
--markup markdown
*.rb
-
README.md

hello.rb

module Hello
    # hi³ there.
    def self.hi
        345
    end
end

README.md

# Hello World³

Actual Output

C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/templates/default/layout/html/layout.erb:21:in `_erb_cache_5': incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError)
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:289:in `erb'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/templates/default/layout/html/setup.rb:62:in `layout'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:369:in `render_section'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:261:in `block (2 levels) in run'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:258:in `each'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:258:in `block in run'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:400:in `add_options'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:257:in `run'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:136:in `run'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/templates/default/fulldoc/html/setup.rb:52:in `block in serialize_index'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/engine.rb:123:in `block in with_serializer'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/logging.rb:82:in `capture'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/engine.rb:121:in `with_serializer'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/templates/default/fulldoc/html/setup.rb:51:in `serialize_index'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/templates/default/fulldoc/html/setup.rb:68:in `serialize_file'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/templates/default/fulldoc/html/setup.rb:11:in `block in init'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/templates/default/fulldoc/html/setup.rb:10:in `each'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/templates/default/fulldoc/html/setup.rb:10:in `each_with_index'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/templates/default/fulldoc/html/setup.rb:10:in `init'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:195:in `initialize'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:131:in `new'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:136:in `run'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/engine.rb:105:in `generate'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/cli/yardoc.rb:357:in `run_generate'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/cli/yardoc.rb:267:in `run'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/cli/command.rb:14:in `run'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/cli/command_parser.rb:72:in `run'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/cli/command_parser.rb:54:in `run'
        from C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/bin/yard:13:in `<top (required)>'
        from C:/Ruby32-x64/bin/yard:32:in `load'
        from C:/Ruby32-x64/bin/yard:32:in `<main>'

If I remove the superscript 3 from the README the output becomes:

C:\Users\tthomas2\Desktop\yard-bug>yard doc --debug
[debug]: Parsing ["*.rb"] with `ruby` parser
[debug]: Parsing hello.rb
[debug]: Serializing to .yardoc/objects/root.dat
[debug]: Re-generating object ...
[debug]: Re-generating object Hello...
[debug]: Generating asset js/jquery.js
[debug]: Serializing to doc/js/jquery.js
[debug]: Generating asset js/app.js
[debug]: Serializing to doc/js/app.js
[debug]: Generating asset js/full_list.js
[debug]: Serializing to doc/js/full_list.js
[debug]: Generating asset css/style.css
[debug]: Serializing to doc/css/style.css
[debug]: Generating asset css/common.css
[debug]: Serializing to doc/css/common.css
[debug]: Generating asset css/full_list.css
[debug]: Serializing to doc/css/full_list.css
[debug]: Generating asset class_list.html
[debug]: Serializing to doc/class_list.html
[debug]: Generating asset method_list.html
[debug]: Serializing to doc/method_list.html
[debug]: Generating asset file_list.html
[debug]: Serializing to doc/file_list.html
[debug]: Generating asset frames.html
[debug]: Serializing to doc/frames.html
[debug]: Serializing to doc/_index.html
[debug]: Serializing to doc/index.html
[debug]: Serializing to doc/file.README.html
[debug]: Serializing to doc/top-level-namespace.html
[error]: Exception occurred while generating 'Hello.html'
[error]: CompatibilityError: incompatible character encodings: ASCII-8BIT and UTF-8
[error]: Stack trace:
        C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/templates/default/layout/html/layout.erb:21:in `_erb_cache_5'
        C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:289:in `erb'
        C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/templates/default/layout/html/setup.rb:62:in `layout'
        C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:369:in `render_section'
        C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:261:in `block (2 levels) in run'
        C:/Users/tthomas2/.gem/ruby/3.2.0/gems/yard-0.9.34/lib/yard/templates/template.rb:258:in `each'

Files:           1
Modules:         1 (    1 undocumented)
Classes:         0 (    0 undocumented)
Constants:       0 (    0 undocumented)
Attributes:      0 (    0 undocumented)
Methods:         1 (    0 undocumented)
 50.00% documented

The documentation "completes" but with an inline error message that doesn't stop the process. This time it's the superscript 3 in hello.rb that breaks.

I'm wondering if YARD is loading text from templates as ASCII and thus failing to combine UTF-8 and ASCII strings.

Expected Output

Expected YARD to generate the documentation without errors.

Environment details:

  • OS: Windows 10
  • Ruby version (ruby -v): ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x64-mingw-ucrt]
  • YARD version (yard -v): yard 0.9.34

I have read the Contributing Guide.

@lsegal
Copy link
Owner

lsegal commented Aug 26, 2024

I can't seem to reproduce this on ruby 3.3.x with yard 0.9.35. Is this still an issue for you?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants