You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Since world tokenizer training code is not available as far as I know, those of us who need a custom tokenizer train HF tokenizer (pip rwkv package, RWKV-LM trainer and json2binidx_tool all support it).
Currently it doesn't work with ai00_server:
[ai00_server::middleware] reload model failed: failed to parse vocabulary: invalid value: expected key to be a number in quotes at line 2 column 3
The text was updated successfully, but these errors were encountered:
Since world tokenizer training code is not available as far as I know, those of us who need a custom tokenizer train HF tokenizer (pip rwkv package, RWKV-LM trainer and json2binidx_tool all support it).
Currently it doesn't work with ai00_server:
The text was updated successfully, but these errors were encountered: