Skip to content

Token -> Tokenizer

Pre-release
Pre-release
Compare
Choose a tag to compare
@mathewsanders mathewsanders released this 02 Jan 20:57
· 32 commits to master since this release
Change from token type to tokenizer type (#2)

* Update API

Switch internal terminology from use of `Token` to `Tokenizer`.
Change external API from `matches(:)` back to the original `tokens(:)`.
Add a convenience method `components(:)` that returns just substrings
rather than array of `Token`.

* Update file name.

* Update README to show `Token` type.

* Include `import Mustard` in code snippet.