[BUG] Verilog preprocessor, ifdef around include statement

[Risto97]

• I have reviewed this project's contribution guidelines

Describe the bug

In the following code:

`ifdef SOME_MACRO
`include "inc.rdl"
`endif

addrmap test {
  reg {
    field {
    hw = rw; sw =rw;
    }f1 [`RANDOM] = 0;
  } reg1;
};

I want to include inc.rdl only if SOME_MACRO is defined.
However this doesn't work, as the PerlPreprocessor runs before VerilogPreprocess and expands include directives without considering ifdef, if, else, ... directives.

systemrdl-compiler/systemrdl/preprocessor/__init__.py

Lines 12 to 19 in 191f4dd

	# Run file through Perl preprocessor
	ppp = PerlPreprocessor(env, path, search_paths)
	preprocessed_text, seg_map = ppp.preprocess()
	included_files = ppp.included_files
	# ... then through the Verilog preprocessor
	vpp = VerilogPreprocessor(env, preprocessed_text, seg_map, defines=defines)
	preprocessed_text, seg_map = vpp.preprocess()

Expected behavior
I expect PerlPreprocessor to ignore include directives if they are surrounded by ifdef which should disable it.

Additional context
Add any other context about the problem here.

[Furetur]

I have also encountered this issue

[amykyta3]

This is intentional due to the Perl preprocessor being a step that is executed completely separately and prior to Verilog preprocessor.
This is partly due to poor specification by the SystemRDL language spec. I spoke to one of the committee members to clarify this and they confirmed that there are some gaps in the spec regarding how includes are handled in a two-layer preprocessor. According to one of the committe members, apparently some of the commercial/proprietary tools work around this mis-specification by reordering some operations to yield less surprising results.
In my implementation i had taken a similar approach. I write about the rationale in more detail here if you are interested: https://systemrdl-compiler.readthedocs.io/en/stable/dev_notes/preprocessors.html

Unfortunately stacking multiple preprocessors like RDL does causes some inherent contradictions which result in quite a messy situation.

[Risto97]

I tried a commercial SystemRDL compiler, and it seems like it works for the example I provided initially, but would fail if the include file contains any perl directives.
The commercial compiler forced me to have ifdef around include directives, as it cannot compile multiple SystemRDL files, in order to be compatible with both.

I think the following example demonstrates the issue about perl variables in include files mixed with verilog preprocessor:

test2.rdl

<% $foo = "FIRST_MACRO"; %>

`ifdef <%=$foo%>
`include "h2.rdl"
`endif

`ifdef <%=$bar%>
<% $macro = "THIRD_MACRO"; %>
`endif

addrmap test {
`ifdef <%=$macro%>
  reg {field {hw=rw;}f1 [32] = 0;
  }r1;
`endif
};

inc/h2.rdl

<% $bar = "SECOND_MACRO"; %>

When I call peakrdl with:

peakrdl html -I./inc2  -DSECOND_MACRO -DTHIRD_MACRO -o here test2.rdl

The question is if the h2.rdl should be processed by perl and verilog preprocessor.
The only way I see it is to do incremental preprocessing through the file, switching between two preprocessors.
But this might be slow to implement, and standard does not define it this way?