Add ?Keyword command to work around Unicode keyword bug

[kirsle]

This feature alleviates a huge pain point that a lot of users have experienced with Unicode keyword triggers.

• Fixes regex doesnt work for UTF8 #147
• Fixes wildcards failing for Japanese #253
• Fixes Problem with [*] for non English  #254

The Bug

Users wanted to be able to write things like this in their RiveScript code:

// (this means "hello" in Chinese)
+ [*] 你好 [*]
- 你好!

The intention is that the word 你好 appearing anywhere in a message should be able to match that trigger, as it does when using standard ASCII characters. However, the regexps that make "optional wildcards" like [*] work have problems matching against Unicode characters outside the ASCII range.

The Feature

This adds a new ?Keyword command to complement the +Trigger as a workaround for this very common use case:

? 你好
- 你好!

Under the hood, the ?Keyword is re-written into a +Trigger which includes the keyword text in every variation of wildcard, optional wildcard, and verbatim text. This makes sure that the keyword matches as expected in all messages that contain it, in a way that works with Unicode.

The JavaScript version gets this feature first because it's the most popular edition of RiveScript.

This will be included in the v1.19.0 release of rivescript-js.

See Also

• rivescript-wd#6 that became the central hub ticket for this issue.
• rivescript-python#78 had the same problem in Python.
• rivescript-go#22 had the same problem in Go.

[kirsle]

@dcsan @Lewikster @alecl @ahmader @kimmanlui @saqibmujtaba @lijiarui

You all participated in the various bug tickets about this issue, so I'd like you to know about RiveScript-JS v1.19.0 which adds a ? Keyword command for matching Unicode keywords in a user's message.

The ?Keyword will search for its text anywhere in a user's message, and works with Unicode in places where doing + [*] keyword [*] didn't. It is an alias to a +Trigger command that rewrites the keyword into a form that will ensure it matches in all cases.

// This didn't work as a trigger:
+ [*] 你好 [*]
- It didn't match. :(

// But you can now do this instead:
? 你好
- It matched!

The Python and Go versions will be getting this feature soon, too. If any of you would like to help, you can refer to this pull request to implement the same changes in the other versions and send me a PR. 😄

[dcsan]

hi -

thanks we'll check this out! So this is in the v1.19.0 which will be out via NPM?
does that version also use the new method to work with Async? I think there was an experimental branch but we never had time to properly try that out.

[kirsle]

@dcsan not yet with the async stuff. Where I left off, I had trouble getting the code to build in a backwards compatible way to older versions of Node. The CoffeeScript2 creates beautiful JS code, but, after Babel and such had its way it actually created broken JS with syntax errors in its attempt to rewrite the async functions. I shelved it for later and hadn't gotten back to trying to figure it out yet.

[dcsan]

Since node is controlled by yourself at server side, and async/await is such a huge feature addition, most people I would think are using a version that supports that directly? ( > node 7.x ?). I guess node 10 will be another LTS version so then people not running node8 are really not taking things too seriously... https://nodejs.org/en/download/releases/