Thought: Training an "Outlines" optimized model

[ManuelFay]

I am just wondering what sort of features would be interesting if we wanted to train a specific LLM optimized for outlines ?

I am guessing maybe causality in most LMs is the biggest issue, maybe training with a Fill-In-The-Middle objective may help performance for templated generation ?

Maybe there are other training objectives that may help models generating under the Outlines framework ?

Wondering if anyone has insights on that, I'd be down to experiment a bit

[amir-in-a-cynch]

I am just wondering what sort of features would be interesting if we wanted to train a specific LLM optimized for outlines ?

I am guessing maybe causality in most LMs is the biggest issue, maybe training with a Fill-In-The-Middle objective may help performance for templated generation ?

Maybe there are other training objectives that may help models generating under the Outlines framework ?

Wondering if anyone has insights on that, I'd be down to experiment a bit

Don't know if you're still interested in this. But i've been doing something similar with focus on customizing an LLM to 1 specific grammar.

1. SINGLE GRAMMAR CUSTOMIZATION VIA GENERATION: What I do is start with the grammar, and then generate large volumes of strings from this. (Because you can convert any grammar into a generator, just by doing a random walk on the productions, starting from the start state). After that you can train the model with this, and then reuse that grammar as the "outlines" constraint.

2. GRAMMAR UNDERSTANDING VIA INSTRUCTION TUNING: The question now is if you want to generalize from just 1 specific grammar, to a more generic set of grammars, and make the LLM really good at this even without being "trained" on the target grammar. So I think what I would do is maybe "instruction tune" a LLM to take in a grammar as part of the prompt, and produce strings that respect it's rules in "unconstrained" generation. Once you've done this over a huge number of grammars, you should be able to include the grammar in a prompt in general and it'll be more likely to produce logits that respect the rules in the prompt (usually).

3. GENERATING A LARGE DATASET OF GRAMMARS AND THEIR GENERATIONS: But where does the dataset for (2) come from in the first place? Well, you could do the following - take ChatGPT or some other LLM and ask it to generate a large set of grammars in chomsky form. Next, apply some random permutations to these grammars - delete rules, swap their ordering, insert symbols, select rules from grammar B and mix them with grammar C, etc, etc. You should be able to get a pretty huge set from this combinatorial explosion. Finally, one can reuse the method from (1) above to generate a large number of outputs from the method of (1) above.

I do think this can only go so far due to architectural constraints - transformers aren't great at learning recursive type structures or parenthesis matching even, which is a very simple context free grammar. Something like a mamba is possibly more suited. But I hypothesize even a transformer would get considerably better.

Also, sorry if this was more of a research program than a light weight experiment. Item (1) above is probably good enough, if you want to "warm up" your target LLM to work with your grammar of preference.