Parse error found via module_roundtrip_naive

[jasikpark]

Sorry for the lack of context for the moment, I just want to get a WIP issue up:

thread '<unnamed>' panicked at fuzz_targets/module_roundtrip_naive.rs:34:9:
input: `f&(/K/d

)()`
output1: `f & /K/d()`

This parse should not error because it was just parsed above
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
==14270== ERROR: libFuzzer: deadly signal
    #0 0x106b6d470 in __sanitizer_print_stack_trace+0x28 (librustc-nightly_rt.asan.dylib:arm64+0x59470)
    #1 0x104f61a78 in fuzzer::PrintStackTrace()+0x30 (module_roundtrip_naive:arm64+0x100299a78)
    #2 0x104f54a0c in fuzzer::Fuzzer::CrashCallback()+0x54 (module_roundtrip_naive:arm64+0x10028ca0c)
    #3 0x19172b580 in _sigtramp+0x34 (libsystem_platform.dylib:arm64+0x4580)
    #4 0x36310001916fac1c  (<unknown module>)
    #5 0xcd21000191607a2c  (<unknown module>)
    #6 0x6c7b800105763a14  (<unknown module>)
    #7 0x1057b6aac in std::process::abort::h45a052e445b72460+0x8 (module_roundtrip_naive:arm64+0x100aeeaac)
    #8 0x104f5393c in libfuzzer_sys::initialize::_$u7b$$u7b$closure$u7d$$u7d$::hcccba7bfa5d7507b+0xb8 (module_roundtrip_naive:arm64+0x10028b93c)
    #9 0x10575a8c4 in std::panicking::rust_panic_with_hook::h8d0c9bb48096fa77+0x5b4 (module_roundtrip_naive:arm64+0x100a928c4)
    #10 0x10575a2d8 in std::panicking::begin_panic_handler::_$u7b$$u7b$closure$u7d$$u7d$::h51d4c7ea379d7ca8+0x94 (module_roundtrip_naive:arm64+0x100a922d8)
    #11 0x105757ef8 in std::sys_common::backtrace::__rust_end_short_backtrace::h3ffb6a655eb0d365+0x8 (module_roundtrip_naive:arm64+0x100a8fef8)
    #12 0x10575a048 in rust_begin_unwind+0x30 (module_roundtrip_naive:arm64+0x100a92048)
    #13 0x1057b8724 in core::panicking::panic_fmt::hc04a814f639f8411+0x28 (module_roundtrip_naive:arm64+0x100af0724)
    #14 0x104efe168 in module_roundtrip_naive::do_fuzz::h9a7a4cecd9c12737 module_roundtrip_naive.rs:34
    #15 0x104efe978 in rust_fuzzer_test_input lib.rs:297
    #16 0x104f4dfb4 in std::panicking::try::do_call::hf4788212a0733068+0xc4 (module_roundtrip_naive:arm64+0x100285fb4)
    #17 0x104f53bb8 in __rust_try+0x20 (module_roundtrip_naive:arm64+0x10028bbb8)
    #18 0x104f52ff4 in LLVMFuzzerTestOneInput+0x16c (module_roundtrip_naive:arm64+0x10028aff4)
    #19 0x104f562d0 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long)+0x150 (module_roundtrip_naive:arm64+0x10028e2d0)
    #20 0x104f74020 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long)+0xe0 (module_roundtrip_naive:arm64+0x1002ac020)
    #21 0x104f79444 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long))+0x1e5c (module_roundtrip_naive:arm64+0x1002b1444)
    #22 0x104f86bd4 in main+0x24 (module_roundtrip_naive:arm64+0x1002bebd4)
    #23 0x1913720dc  (<unknown module>)
    #24 0xec397ffffffffffc  (<unknown module>)

NOTE: libFuzzer has rudimentary signal handlers.
      Combine libFuzzer with AddressSanitizer or similar for better crash reports.
SUMMARY: libFuzzer: deadly signal
────────────────────────────────────────────────────────────────────────────────

Error: Fuzz target exited with exit status: 77

I'll try to see what's causing the failing parsing or printing in a bit

[kaleidawave]

It is because RegExp literals need special lexing. The RegExp lexing state is set when the previous token is an expression prefix (and the next character is == '/').

TSXToken::BitwiseAnd is missing from this list:

ezno/parser/src/tokens.rs

Lines 405 to 424 in 4e19531

	pub fn is_expression_prefix(&self) -> bool {
	matches!(
	self,
	TSXToken::Keyword(TSXKeyword::Return | TSXKeyword::Case | TSXKeyword::Yield | TSXKeyword::Throw | TSXKeyword::TypeOf | TSXKeyword::Await)
	| TSXToken::Arrow
	// for `const x = 2; /something/g`
	| TSXToken::SemiColon
	| TSXToken::OpenParentheses
	| TSXToken::OpenBrace
	| TSXToken::JSXExpressionStart
	| TSXToken::QuestionMark
	| TSXToken::Colon
	| TSXToken::LogicalNot
	| TSXToken::LogicalAnd
	| TSXToken::LogicalOr
	| TSXToken::Multiply
	| TSXToken::Add
	| TSXToken::Subtract
	| TSXToken::Divide
	) || self.is_assignment()

(this is_expression_prefix is also how JSX lexing works vs generic type arguments and inequalities)

will fix in the ongoing #158

I wonder if there is a way to reuse this logic?

ezno/parser/src/expressions/operators.rs

Line 369 in 4e19531

impl TryFrom<&TSXToken> for BinaryOperator {

Also I guess you are you finding these issues by running the fuzzing locally? Is it finding these quick?

[jasikpark]

Yep, I'm running it locally on an M1 Max macbook pro with lots of ram, takes only a minute or so of running to find these

[kaleidawave]

This specific issue fixed in #158