How does pest generate parsing codes for different rule types (modifiers)?

[WHMHammer]

I am trying to understand the following piece of codes for #520:

pest/generator/src/generator.rs

Lines 279 to 329 in a4c93a7

	match rule.ty {
	RuleType::Normal => quote! {
	#[inline]
	#[allow(non_snake_case, unused_variables)]
	pub fn #name(state: #box_ty<::pest::ParserState<'_, Rule>>) -> ::pest::ParseResult<#box_ty<::pest::ParserState<'_, Rule>>> {
	state.rule(Rule::#name, |state| {
	#expr
	})
	}
	},
	RuleType::Silent => quote! {
	#[inline]
	#[allow(non_snake_case, unused_variables)]
	pub fn #name(state: #box_ty<::pest::ParserState<'_, Rule>>) -> ::pest::ParseResult<#box_ty<::pest::ParserState<'_, Rule>>> {
	#expr
	}
	},
	RuleType::Atomic => quote! {
	#[inline]
	#[allow(non_snake_case, unused_variables)]
	pub fn #name(state: #box_ty<::pest::ParserState<'_, Rule>>) -> ::pest::ParseResult<#box_ty<::pest::ParserState<'_, Rule>>> {
	state.rule(Rule::#name, |state| {
	state.atomic(::pest::Atomicity::Atomic, |state| {
	#expr
	})
	})
	}
	},
	RuleType::CompoundAtomic => quote! {
	#[inline]
	#[allow(non_snake_case, unused_variables)]
	pub fn #name(state: #box_ty<::pest::ParserState<'_, Rule>>) -> ::pest::ParseResult<#box_ty<::pest::ParserState<'_, Rule>>> {
	state.atomic(::pest::Atomicity::CompoundAtomic, |state| {
	state.rule(Rule::#name, |state| {
	#expr
	})
	})
	}
	},
	RuleType::NonAtomic => quote! {
	#[inline]
	#[allow(non_snake_case, unused_variables)]
	pub fn #name(state: #box_ty<::pest::ParserState<'_, Rule>>) -> ::pest::ParseResult<#box_ty<::pest::ParserState<'_, Rule>>> {
	state.atomic(::pest::Atomicity::NonAtomic, |state| {
	state.rule(Rule::#name, |state| {
	#expr
	})
	})
	}
	},
	}

The bodies of the generated methods are chains of function calls as listed below:

• Normal: rule -> #expr
• Silent: #expr
• Atomic: rule -> atomic -> #expr
• CompoundAtomic -> atomic -> rule -> #expr
• NonAtomic -> atomic -> rule -> #expr

What makes me wonder is that for Atomic, atomic is called inside rule, but for CompoundAtomic and NonAtomic, rule is called inside atomic. Is there a reason that the calling sequence for Atomic is different from CompoundAtomic and NonAtomic? Does the calling sequence matter?

Thank you.

[NoahTheDuke]

I don't know the history. Have you used git blame to find out when and how those were added? That might shed some light.

[WHMHammer]

Thank you @NoahTheDuke, the change history is really interesting.

In the very first version of the generate_rule function, although the Compound Atomic modifier wasn't there, the calling sequences of the Atomic and Non-Atomic rules are the same: they are both atomic -> rule -> #expr.

In a later commit, the calling sequence in both Atomic and Non-Atomic (Compund Atomic still did not exist yet) were flipped, both becoming rule -> atomic -> #expr. The commit message was Added tests and fixed some issues., but the author didn't specify the details of the issue nor the purpose of changing the call sequence.

And then in another commit, the Non-Atomic case was removed, and the calling sequence of Compound Atomic changed from rule -> atomic -> #expr back to atomic -> rule -> #expr. The commit message was Fixed a couple of atomic-related bugs., still without any details.

When Compound Atomic was added back, its calling sequence became atomic -> rule -> #expr as well, different from the sequence of rule -> atomic -> #expr when it was removed in the commit I mentioned in the previous paragraph. The commit message was Fixed compound atomic's cascading effect., without furter details.

I still don't understand the reasons behind these changes yet, but I will continue my research.

Hi @dragostis, you made all 4 commits above. Would you mind kindly explain the decisions and provide me with some guidance on implementing _@, _$, and _! modifiers, as mentioned in #520? I would really appreciate your help! Thank you and thank you for building this wonderful library!

[tomtau]

@WHMHammer I think the calling sequence may matter for the state scoping (how the interior rules should behave / whether they produce inner tokens or not).