JET doesn't analyze expression given to @btime

[bergel]

Consider this example:

julia> JET.report_text("""
       using BenchmarkTools
       @btime (println("This ran!");1+"1") samples=1 evals=1
       """, concretization_patterns=[:x_])
This ran!
═════ 1 toplevel error found ═════
┌ @ top-level:2
│ MethodError: no method matching +(::Int64, ::String)
...

The example above shows that JET executes the macro in addition to running the usual checks. Is there a way for JET to not evaluate specific macros but only analyze the expression the macro operates on?

[aviatesk]

The concretization_patterns=[:x_] option tells JET to actually execute all provided (in order to to achieve better analysis accuracy). This means JET will execute code like the following, regardless of whether or not macros are being used:

julia> report_text("""
       println("This ran!")
       1 + "1"
       """; concretization_patterns=Any[:x_])
This ran!
...

To prevent this, I suggest you might want to consider not specifying concretization_patterns at all.

[bergel]

Thanks @aviatesk for the prompt reply.
However, it seems that removing the concretization_patterns prevents JET from finding obvious error. E.g.,

julia> JET.report_text("""
              using BenchmarkTools
              @btime (println("This ran!");1+"1") samples=1 evals=1
              """; target_defined_modules = true)
No errors detected

It should complain with 1+"1"

[aviatesk]

Indeed, the code passed to @btime is internally represented as an Expr object, and BenchmarkTools.jl fundamentally executes this using Core.eval. JET doesn't analyze these Expr objects, becauset Julia's metaprogramming features can generate an unlimited number of such invalid Expr objects, and it would be endless to report every single one. From the static analysis point of view, the difficulty of analyzing your case is essentially same as the one of analyzing code using Core.eval.

[bergel]

Thanks for your reply. It seems there is no solution to my initial problem then.

[aviatesk]

Indeed. Well, alternatively, JET could consider specially analyzing expressions provided to frequently used idiom such as @btime as a special case. To enable this, a sort of plugin system would be required.