Introduce constant-folding preprocessor

[ethanuppal]

As an IL, calyx is intended as a generation target. Still, there are instances in which we write calyx code directly, and due to a lack of constant and constant-folding support, we have to resort to magic numbers. These come with all the classic magic number issues about readability and maintainability. If you have 16 everywhere in your code to represent the number of bits an entry takes up, adding a single bit tag to one of these yields a random 17 in the code, which is an entirely different magic number. Why not $define ENTRY_BITS 16 and then use $(ENTRY_BITS + 1)?

I propose to solve this with a very light constant-folding preprocessor. I've mocked up a working demo on the preproc branch (under tools/preprocessor). I cargo run on the initial file to produce the preprocessed file. Notice how line numbers are preserved (so that the LSP will still display correctly-positioned errors (it doesn't, but that's a separate bug)). It simply reads line-by-line from stdin and writes to stdout, which I think is the kind of simplicity that suffices for this task.

I find this useful in writing a cache (a highly parametrized construction), but I'd like to know if this is useful to anyone else.

Some further thoughts:

• I think this deserves its own tool because semantically defining and constant folding should not be the job of the IL. We don't want calyx to be system verilog.
• The current "import" system is more like a C "include" system (note: in the preprocessor); I'd expect an import system with access to an AST to perform some kind of AST union (if not done already, I haven't looked into Workspace too hard). Moving the imports to the preprocessor would also allow customization in terms of defining macros for the import. I've said this quite verbosely but imagine $import<WIDTH 32, INIT 0> "ram.futil" or dim.
• We'd need to integrate this into the LSP somehow. The preprocessor as written works on a line-by-line basis (fn process(&mut self, line: String) -> ContextResult<String>), so it's not especially suited for direct input to tree-sitter, I believe. I tried to modify the LSP to capture the stdin and wrap it in a new reader that applies the preprocessor to every line, but I wasn't able to get it working and ended up not committing it to the branch.

[rachitnigam]

Thanks for writing this up! I would generally like to frame this as a language built on top of Calyx, i.e., similar to the python eDSL, it's okay to make opinionated choices about how the parameterization should work and what kinds of things it should allow. A crucial implication of this distinction is that the Calyx compiler and language are not familiar with the effects of the parameterization on the optimization of the programs. I would also consider how far the effect of parameterization goes: do you want to extend the control language to support use of parameterized values? Should you be able to generate new groups based on parameterization (i.e. compile-time meta-programming), etc.

[ethanuppal]

I was emphatically not considering this to be a language* on top of calyx or to support parametrization -- if you think that this is a good idea, namely, a preprocessing layer geared toward parametrization, that would send the preprocessor down another direction. I feel as though parametrization should be handled in higher-level DSLs (e.g., Filament). I'm also worried about enabling stuff like meta-programming, which I feel might be too powerful (given that the main focus on language extensions should be geared toward generators). I think my idea with this preprocessor is just to get rid of magic numbers via a superset of calyx that doesn't affect anything existing. These are just my thoughts, though; I'll go ahead with whatever you feel is the best direction.

*not in the sense that it is valid calyx, but in the sense that it provides more/different functionality than calyx.

[ethanuppal]

Update: I've integrated the preprocessor with the LSP:

[sgpthomas]

Looks awesome! Just fyi, error locations are working better in this PR: #1950. I'll get around to finishing this up soon.

[ethanuppal]

If people like this idea, I'll rewrite my mockup with actually good code.

[rachitnigam]

I was emphatically not considering this to be a language* on top of calyx or to support parametrization

If this is not the case, then I'd rather just use m4? What is the specific value of rolling out our own define constant folder?

[ethanuppal]

Sure. I'm hesitant because I don't see a use for any features beside constants and constant folding, especially since calyx isn't designed as a frontend.

[ethanuppal]

I don't think what I want is a preprocessor. I think I found and implemented what I actually wanted. I'll close this if no one has any objections (and as Rachit pointed out, there are plenty of existing preprocessor in addition to m4 we can use if this need arises). I'll send an explanation in the Slack.

EDIT: See #2157