Issue with conversations with utf8

[arashsa]

! var name = ælda

+ [*] yo [*]
- <bot name>

+ *
% <bot name>
- wow

This does not work. There seems to be an issue with utf8 and conversations.

[kirsle]

This might be related to a similar issue in the JS implementation (aichaos/rivescript-js#147) which also involved optionals with Unicode symbols. The root cause identified there was with the word boundary sequence \b not working with non-ASCII symbols.

[arashsa]

Is there a way to solve this? I'm not familiar enough with the code to figure out where the issue might be.

[kirsle]

It will take some debugging and figuring out if I can fix the root cause of the issue (in the regular expressions).

On a near-term TODO list, I'll add a new command to RiveScript to allow for writing a raw regular expression in place of the trigger. I'm finding that Unicode is full of all kinds of little gotcha's, and regular expression engines aren't created equally between programming languages, so allowing for writing a raw regular expression might be the best way to work around these edge cases.

Example of what I mean:

// The new ~ command would allow for a raw regexp
// instead of using the + command
~ my name is (.+?)
- Nice to meet you, <star1>.

// that would be equivalent to...
+ my name is *
- Nice to meet you, <star1>.

[kirsle]

Closing this issue in favor of tracking the ~Regexp feature in aichaos/rivescript-wd#6