Replace basic Numba FSM functions with Rust implementations

[brandonwillard]

This PR introduces a simple Rust extension that replaces the index construction code.

We can base our work towards a primarily Rust project (i.e. #4) on these changes, which will allow us to make use of outlines-core in outlines much sooner and permit parallel work on more code conversion. We should also be able to apply straightforward changes to this setup that make the Python bindings and PyO3 dependencies optional (e.g. for Rust-only use of the underlying crate).

• Replace Python implementations with Rust extension
• Make sure tests pass with Rust extension
• Make sure benchmarks pass with Rust extension

[brandonwillard]

FYI: The failure is only due to missing coverage, which could be related to a distinct configuration issue: dottxt-ai/outlines#1105.

[rlouf]

Looks good to me, especially the benchmarks vs main:

[ErikKaum]

Nice, thank you Brandon 👍

As far as I can tell it looks good. Small nits on naming that don't need to be addressed now, but rather when there's a clearer separation between rust crate and bindings.

• rust_fsm --> maybe outlines-core-rs
• _walk_fsm_internal as to not mix python & rust naming conventions

I didn't find any obvious bugs in the fsm implementation itself, but I'm not as experienced as you either. In other words, good if someone else can double check it.

[rlouf]

I didn't find any obvious bugs in the fsm implementation itself, but I'm not as experienced as you either. In other words, good if someone else can double check it.

Note that the tests that were in Outlines all pass with this implementation.

[brandonwillard]

• rust_fsm --> maybe outlines-core-rs
• _walk_fsm_internal as to not mix python & rust naming conventions

Done

[drbh]

LGTM! and benchmarks look really good. Great job @brandonwillard

not sure if the "Tests / Combine & check coverage. (pull_request)" are important?

[brandonwillard]

LGTM! and benchmarks look really good. Great job @brandonwillard

Most of this is adapted from our initial internal work by @kc611, which was exclusively focused on obtaining parity with Numba. There are many more big improvements to make, but it does have parity and AOT compilation, so it's a worthy replacement for the code in outlines.

not sure if the "Tests / Combine & check coverage. (pull_request)" are important?

It's not a problem; see #7 (comment).