ast, read_verilog: ownership in AST, use C++ styles for parser and lexer

[widlarizer]

The purpose of this PR is to clarify resource ownership (responsibility for construction and deletion) in the AST by distinguishing between owning pointers (std::unique_ptr<AstNode>) and non-owning pointers (AstNode*).

Core idea

struct AstNode now holds these members:

std::vector<std::unique_ptr<AstNode>> children;
std::map<RTLIL::IdString, std::unique_ptr<AstNode>> attributes;

There are no more new AstNode and corresponding delete statements present in the codebase. std::unique_ptr<AstNode> falling out of scope triggers ~AstNode, destroying its children, too. Assigning nullptr to it behaves the same way. This should eliminate AstNode allocations being reported as leaking on yosys termination by valgrind.

Damage

This implied returning std::unique_ptr<AstNode> from parse rules. Prior to this PR, the return data type for parse rules is auto-generated with the bison %union directive. The auto-generated code handling it relies on each union member type having an T& operator=(T&);, but std::unique_ptr<...> deletes its own for good reason.

That's why I refactored the parser and lexer with their C++ modes causing a greater code change.

Locations in AST and parser values now use the same custom data type with std::shared_ptr<std::string> filename;. This reduces filename copying in the frontend greatly. I'm seeing 6-20% memory usage improvements in read_verilog as a result, at a cost of a 1% performance regression, which may be avoidable.

Superior location tracking without global state will allow proper column ranges in errors in a simple followup PR.

Somehow, syntax errors now don't quote characters, so you get ERROR: syntax error, unexpected ; instead of ERROR: syntax error, unexpected ';'. fixed

Testing

This PR has a high risk of creating new yosys segfaults in edge cases not hit by our test suite. More extensive testing will be needed. However, I would argue that crashes are more desirable than a mix of "everything somehow works out" and straight up silent data corruption.

TODOs

• Source locations are temporarily completely broken. This breaks tests
• Extend source location diagnostics testing on main prior to merging this. See logger: add -expect types prefix-log, prefix-warning, prefix-error #5183
• Can global variables be eliminated from the parser by moving them into the generated parser class?
• Ensure there are no Makefile dependency errors
• Extend ownership in ast and parser to attribute lists and strings. Containers already basically are owned pointers, but maybe should be still pointed to with std::unique_ptr to avoid accidental copies strings done, attribute lists low prio followup
• Play around with memory and address sanitizers to see if we're cutting down on the leaks - address sanitizer clean, memory sanitizer complains
• Try reduce the perf regression likely caused by mandatory source location construction for all nodes
• Fix the internal docs for the verilog frontend
• Workaround for older bison versions due to parse.error ? Declare correct required bison version

[widlarizer]

@whitequark I have a problem with flex headers not being available in the WASI build. Since the build runs on our regular linux CI image, it should have those installed if the linux bulid passes. I don't have experience with the WASI SDK. Does it not see system installed headers? Is there some reason even zlib and readline are disabled? Any suggestions appreciated. The run in question: https://github.com/YosysHQ/yosys/actions/runs/16206897063/job/45759169322

[whitequark]

Does it not see system installed headers?

No. Building for WASI uses a cross-compilation environment. Everything that you touch must be built for your target (WASI in this case) and installed in a cross-prefix. Doing things otherwise (particularly in C/C++) would be misguided: even if sometimes it works, that would be only by happy coincidence.

Readline will not work on WASI (for several reasons, including "WASI has no notion of escape codes" and "readline requires a working setjmp/longjmp which I currently cannot ship"). Zlib would not be too difficult to build but typically YoWASP builds are used with smaller netlists where it's just not really necessary. (Nobody asked for it in the entire time I maintained YoWASP).

Don't we have a pair of flex lexers in-tree already? I haven't checked the code/logs but I don't understand why flex headers would be necessary.

[whitequark]

OK so it looks like it needs this header whenever you use the C++ mode. There are two options:

• Do the whole configure-make-makeinstall dance for flex after downloading the flex sources first.
• Copy this one header out of the flex installation and update it every 4 years or whenever they add a new field.

I have no real preference.

[KrystalDelusion]

As a proof of concept, WASI and VS builds can pass if the FlexLexer.h file is given to them. In practice, I think telling the user to ln -s a file if they want to build for WASI seems ill-advised, so hopefully there's a bit more work here before it gets merged.

[widlarizer]

Doing things this way is the least likely to break, the header doesn't depend on anything and has no arch specific ifdefs. This way you will always get the header of the same version as the actual lexer generator expects. I can't guarantee that otherwise

I thought WASI builds get distributed from CI, but also, we are already effectively advising people to just figure out how they have to install dependencies on their system, the README example for some specific ubuntu distro is just illustrative

[KrystalDelusion]

This way you will always get the header of the same version as the actual lexer generator expects. I can't guarantee that otherwise

That's a good point actually.

I thought WASI builds get distributed from CI

Uhhh, pass. I think the WASI build on CI is mostly for checking it compiles, and it doesn't actually upload anything. I think the main user is YoWASP, though that is probably predominantly distributed from CI builds.

[whitequark]

Uhhh, pass. I think the WASI build on CI is mostly for checking it compiles, and it doesn't actually upload anything. I think the main user is YoWASP, though that is probably predominantly distributed from CI builds.

This is correct, although there are definitely non-YoWASP users (since the Emscripten build was deprecated and I know some people use it).

I'm not quite sure what you mean by "distributed from CI builds" specifically. Are you referring to the fact that people are unlikely to build their own? If so, yes, that's most likely the case (if people are building YoWASP binaries on their own they don't tell me).

[whitequark]

I've added a flex build/install step to the CI workflow since it was faster to do than to keep discussing this.

[whitequark]

Also, it looks like the MSVC build is failing too for the same reason, so this problem isn't exclusive to WASI.

[whitequark]

The build passes locally with act so I'm going to assume it will pass on GitHub too.

edit: yep, works

[KrystalDelusion]

I added the commits to get the VS build to pass; I also double checked the header file includes the necessary license notice to redistribute it (it does). This doesn't resolve the problem with the wheels building on Windows, but they are already disabled because of an issue with FFI, so it's not a blocker.