review tokenizer algorithm

[allan-simon]

right we're doing a simple greedy algo that does not look backward

instead we should do a

1 - try to get all possible tokenization

2 - apply on it filtering rules to remove impossible combination (based on a "impossible tokenization" rules fils ?)

3 - weight remaining tokenization using a weighting function (maybe 2 can be mixed with this step, admitting that 'impossible' stuff would get a very low score)

4 - keep the highest score (add a mechanical arm to the server to flip a coin for tokenization with same weight, you see I've thought about everything, smartass )