plain input/output prompt strategy w/o chat templates

[winglian]

No description provided.

[hamelsmu]

Thanks @winglian for starting this. Some important things for discussion:

1. We need multi-turn support (you are currently using sharegpt for this, I would want to handle single-turn as just a subset of multi-turn. My suggestion is not to add separate prompt strategies for multi and single turn and just only have multi-turn.
2. Can you wire it up to the config etc so I can test it? I can help writing docs and tests.
3. Do you have any opinions on the parameters add_eos_token and strip_bos_token for the tokenizer? Do we need to set those (especially considering the multi-turn case)? I think it makes sense to perhaps do something like this (psuedo-code below)

# If there is only 1 message in the thread, need both EOS and BOS
if len(messages) == 1:
... = tokenize(add_eos_token=True, strip_bos_token=False)

# If first message in the thread with many messages, then we need BOS but not EOS 
elif first message in the thread and len(messages) > 1:
 .... =  tokenize(add_eos_token=False, strip_bos_token=False)

# inputs don't need an EOS token
if msg.type='input' and NOT the first message in the thread: 
 .... =  tokenize(add_eos_token=False, strip_bos_token=True)

# outputs should have an EOS token always?
elif msg.type='output' and NOT first message in the thread:
.... = tokenize(add_eos_token=True, strip_bos_token=True)

Please check as sometimes I get this wrong - the actual code could be made more succint and readable, I just wrote it out long form to get the idea accross

EDIT: I talked with @winglian and we think that its better not to do any magic at all (so completely forget #3), and hand off all responsibility to the user to add EOS,BOS,EOT, etc. as well as whitespaces or newlines in b/w things. This will not only simplify the code, but give users extreme freedom w/strings