Retargeted to rust #10

db48x · 2020-12-09T10:56:48Z

All the tests pass this time around

This adds the Rust backend, along with a command-line switch to choose it over the cpp backend. The rust directory contains a manual port of a generated lexer to Rust; once it compiles and works I'll be able to make the corresponding changes to the rust-flex.skl and rust_backend.c.

turns out that two of the gotos cannot simply be replaced by loop labels

This requires an explicit break after the 'find_action loop, so that most code paths don't actually loop after entering 'yy_match.

…and expand a few C macros

and add a FILE type that so that we can implement Eq for it

so there's no need for a special method to do it. Also, this method presumably breaks things if you accidentally delete a buffer that is in the buffer stack but isn't the current buffer. The detail about not deallocating the buffer contents if we don't own it can be handled at the type level in Rust; currently we just have an owned Vec<u8> for that, but I can go back and make it generic, with a bound of AsRef<&[u8]>, so that we can handle both the owned and unowned cases automatically.

Still need to go in and remove yy_buffer_stack_top, since the Vec we store the stack in already knows its own length.

We're storing the stack in a Vec, which already has that information.

Also, make the input and output streams be Option<FILE>.

It's actually buffer.init() now.

Using Scan::wrap instead.

as previously suspected

and add a Debug impl to everything

Now that I've pretty much finished porting everything, I think that I have enough information to replace this with ordinary reads from the rust standard library, which would prevent this type of brain damage.

I put that in scare quotes because this is still pretty stupid, considering that the standard library is right there… Also, we need to actually store the number of bytes that we read.

Rust doesn't have them; use += and -= instead.

Just to be consistent between different m4 branches.

so it must therefore be a method

fixes ccl_rust

…rust working

Fixes both the yymorearraybol_rust and yymorearray_rust tests.

and add some m4_dnl to clean up the blank lines

when M4_NO_MODE_YYMORE_USED is defined.

Mostly the changes are in ruleset.sh, which is printing out rules to build the "Go" tests using gcc, since the Go backend isn't quite producing Go code yet.

now that they are generated as part of the ruleset.

by tossing in a bunch of m4_dnl to delete newline characters.

the .l.go and .l.rs rules are quite important.

db48x added 30 commits October 14, 2020 09:28

misc minor fixes

03a11b6

ported up to the input method

d26d3ac

port lex, mostly

c669f67

turns out that two of the gotos cannot simply be replaced by loop labels

fix the 'yy_match loop by extending it around the 'find_action loop

1ee5426

This requires an explicit break after the 'find_action loop, so that most code paths don't actually loop after entering 'yy_match.

mutable version of current_buffer method, fix loop break conditions, …

515ebfc

…and expand a few C macros

get Scan::input compiling

d706757

implement Scan::get_next_buffer and Scan::get_previous_state

4661031

implement Scan::try_NUL_trans

05320f7

implement Scan::unput

af9b402

port Scan::input

ea3a70b

port restart and switch_to_buffer

e437d42

and add a FILE type that so that we can implement Eq for it

BufferState::{new,init,flush}

28fabc5

port {push,pop}_buffer_state

d3e4587

Still need to go in and remove yy_buffer_stack_top, since the Vec we store the stack in already knows its own length.

remove buffer_stack_{top,max} members

c95c117

We're storing the stack in a Vec, which already has that information.

make buffer_stack a plain Vec, and remove ensure_buffer_stack

f2a22fa

Scan::{new,init}

d264e42

Also, make the input and output streams be Option<FILE>.

clean up stray init_buffer call

5c72837

It's actually buffer.init() now.

clean up yywrap function

f204a74

Using Scan::wrap instead.

clean up accessor methods, utility methods, malloc/realloc/free methods

ae5efde

no, BufferState::init cannot take an &mut Scan because borrow checker

766d782

as previously suspected

Scan is a dumb name; Scanner is better

2db1657

on second thought, make FILE store a *mut libc::FILE

1fd524c

and add a Debug impl to everything

a few minor things suggested by clippy

74b46a5

set the start state correctly

f2b4ff9

set the new buffer size correctly

f3aafb9

initialize an empty buffer correctly

6a5586f

fix dumb mistakes while reading

0ee1948

Now that I've pretty much finished porting everything, I think that I have enough information to replace this with ordinary reads from the rust standard library, which would prevent this type of brain damage.

use a "safer" way of reading into our buffer

a5da8e2

I put that in scare quotes because this is still pretty stupid, considering that the standard library is right there… Also, we need to actually store the number of bytes that we read.

db48x added 30 commits December 9, 2020 02:35

fixes for code implementing M4_MODE_LINENO

07e355e

change the lineno test so that it doesn't use increment and decrement

0a21269

Rust doesn't have them; use += and -= instead.

allow redundant_semicolons too

b18f8b9

s/num_to_read/number_to_read/

11dafb5

Just to be consistent between different m4 branches.

state_buf is a stack, so use push/pop instead of an index

5fb0d2b

array indices are usize in REJECT impl

f22de70

yymoreaddaybol test needs to use M4_TEST_ASSERT macro

3772108

yyless cannot be a macro after all

8c38e4e

so it must therefore be a method

fix iterator in more method

5a3b38f

fix adding and subtracting more_offset to leng

7ac2c54

more_flag is boolean

9b7f03b

less method needs to call self.more_adj

3b5ce91

characters are u8

9a3e199

m4 hack for special handling of unput

65652b1

fix bol test

cb3ea0a

handle eof (reading zero bytes) correctly

cfebad5

fixes ccl_rust

fix compile errors in all tests using lineno, and to get vartrailing_…

ceeab27

…rust working

recombine less method, now that it doesn't need any conditions

546fffb

handle copying into text_buf correctly (I think)

0dd2ad0

Fixes both the yymorearraybol_rust and yymorearray_rust tests.

accidentally deleted a ]])

e9c3cbd

and add some m4_dnl to clean up the blank lines

hide some minor warnings

efd4b76

compute range correctly in do_before_action

55aa82a

when M4_NO_MODE_YYMORE_USED is defined.

introduce test macros for some C/Rust/Go differences

1497901

bring the Go backend up to scratch so that its tests pass

26a96e8

Mostly the changes are in ruleset.sh, which is printing out rules to build the "Go" tests using gcc, since the Go backend isn't quite producing Go code yet.

Merge remote-tracking branch 'esr/retargeting' into retargeted-to-rust

d3f9173

remove obsolete file rust_backend.c

df897e4

remove a debugging printf

46c9ac5

remove some rules from the Makefile that are no longer needed

2853025

now that they are generated as part of the ruleset.

make the rust skeleton produce fewer random empty lines

444548a

by tossing in a bunch of m4_dnl to delete newline characters.

was slightly overagressive at removing unnecessary make productions

29f9fdd

the .l.go and .l.rs rules are quite important.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Retargeted to rust #10

Retargeted to rust #10

db48x commented Dec 9, 2020

Retargeted to rust #10

Are you sure you want to change the base?

Retargeted to rust #10

Conversation

db48x commented Dec 9, 2020