Enable reusable Prism parse result

[koic]

Follow-up to Shopify/ruby-lsp#1849.

This feature utilizes ruby/prism#3478 to implement Shopify/ruby-lsp#1849.

By using ruby/prism#3478, this feature enables performance improvements by reusing Prism's parsed results instead of parsing the source code. Below is a sample code snippet, which is expected to improve performance by 1.3x in this case.

#!/usr/local/bin/ruby

require 'benchmark/ips'
require 'prism'
require 'rubocop-ast'

@source = File.read(__FILE__)
@parse_lex_result = Prism.parse_lex(@source)

def build_processed_source(parse_lex_result)
  RuboCop::AST::ProcessedSource.new(
    @source,
    3.4,
    __FILE__,
    parser_engine: 'parser_prism',
    prism_result: parse_lex_result
  )
end

Benchmark.ips do |x|
  x.report('source')                { build_processed_source(nil) }
  x.report('Prism::ParseLexResult') { build_processed_source(@parse_lex_result) }

  x.compare!
end

$ bundle exec ruby reusable_ast.rb
ruby 3.4.2 (2025-02-15 revision d2930f8e7a) +PRISM [x86_64-darwin23]
Warming up --------------------------------------
              source   116.000 i/100ms
Prism::ParseLexResult
                       151.000 i/100ms
Calculating -------------------------------------
              source      1.144k (± 8.4%) i/s -      5.684k in   5.018270s
Prism::ParseLexResult
                          1.460k (± 5.2%) i/s -      7.399k in   5.082102s

Comparison:
Prism::ParseLexResult:     1460.3 i/s
              source:     1144.4 i/s - 1.28x  slower

Achieving 1.3x speedup with such this simple modification is a significant improvement for Ruby LSP and its users.

Compatibility

By using parser_class.instance_method(:initialize).parameters.assoc(:key), the implementation checks whether Prism#initialize supports the :parser keyword. If it does not, the system falls back to the conventional parsing method. This ensures backward compatibility with previous versions of Prism.

Development Notes

Since this feature is specifically designed for Prism, the keyword name prism_result is meant for Prism. While it may be possible to achieve similar functionality with the Parser gem, there are currently no concrete use cases. Therefore, for clarity, we have chosen prism_result as the keyword.

Additionally, Prism has two types of results: Prism::ParseResult and Prism::ParseLexResult, but the prism_result keyword argument is meant to receive Prism::ParseLexResult. Since prism_parse_lex_result would be too long, the name prism_result was chosen to align with the superclass Prism::Result.

cc @bbatsov @marcandre @vinistock

[bbatsov]

I'm OK with the proposal, but I'd suggest to document this optimization both here and eventually in RuboCop.

By using ruby/prism#3478, this feature enables performance improvements by reusing Prism's parsed results instead of parsing the source code.

One thing that's not very clear to me - when/how does this pre-parsing happen in Prism?

[bbatsov]

Ah, after reviewing the RubyLSP thread it seems they are already using Prism.parse_lex and they want to pass the results directly to RuboCop to avoid having it to parse the source again itself. From what I understand we won't be able to leverage this to improve RuboCop's performance directly, right?

[bbatsov]

Are you sure that wording is correct here? (between multiple language servers) I'd assume the benefit here is that the same result is reused for different operations within the lifetime of a specific instance of the language server.

[koic]

I've updated the text to focus on an objective expectation instead of providing an example for a use case that could potentially be misleading.

[koic]

I'd suggest to document this optimization both here and eventually in RuboCop.

I've added it to the index.adoc.

Ah, after reviewing the RubyLSP thread it seems they are already using Prism.parse_lex and they want to pass the results directly to RuboCop to avoid having it to parse the source again itself. From what I understand we won't be able to leverage this to improve RuboCop's performance directly, right?

Yes, that understanding is correct. For Ruby LSP, re-parsing the same source code in RuboCop, when it has already been parsed externally, is redundant processing. This is an optimization for Ruby LSP and does not directly speed up RuboCop itself.

[bbatsov]

Looks good to me. 🚀

[koic]

@bbatsov ruby/prism#3478 has been merged. Since it is compatible with both the old and new versions of Prism, once this PR is merged and RuboCop AST is released, I will update and rebase RuboCop's rubocop/rubocop#13899.

[koic]

@vinistock I think that this feature is likely valuable for Ruby LSP, independent of rubocop/rubocop#13899. Related development is currently in progress in Shopify/ruby-lsp#3252, so if the release of this feature is needed, please feel free to let me know.

cc @bbatsov @marcandre

[bbatsov]

@koic Seems to me we can go ahead and merge this without having to wait for @vinistock to respond.

[koic]

@bbatsov Yah, I'll merge this implementation as it doesn't cause any incompatibilities. Thank you!

[marcandre]

Thanks. @koic do you need a release for this?

[koic]

@marcandre When released, it will help with the development of rubocop/rubocop#13899 and Shopify/ruby-lsp#3252. By the way, #363 is expected to work from the next release onward :-)