Logprobs view – show even when aborted

[aleksusklim]

I'm playing with the new logprobs functionality. I see in the 1.77 release it is mentioned that they won't work with streaming. In fact, the link (View Logprobs) correctly opens the pop-up with the table, but is completely absent if I'd click [ABORT] during any generation.

Since I want to explore model's behavior against sampling settings and chosen tokens, I have to abort and Retry many times in a row!
The only possible option is to set a very low amount of tokens to generate (like the min default of 8 or set it even less) to be able to see the resulting logprobs.

Opened /api/extra/last_logprobs in another tab always gives me logprobs: null no matter streaming is enabled or not, is this normal?

The second question: for each row of the table, I can see percentage values in XX:XX%, but most of the rightmost cells are empty. Are they empty because of sampler settings (for example, they got discarded by min_p/top_p/top_k) or because the absolute value is too low?
Why there are only 6 (or 5?) columns?

Can you make the number of columns configurable (in the server initialization config is fine), defaulting to 6 ? Can you paint near-zero ones in gray instead of omiting them?

Also, is this technically possible to still print zero logprobs for tokens that got dropped by samplers, but in their correct order?
For example, with top_k=1 I always get 100% for everything, and nothing else. Can you "not drop" the tail, and output it with 0.00% but with actual tokens that were there?

[LostRuins]

Yes, the logprobs are not actually sent during streaming. Kobold Lite does a separate request for them after streaming ends. if you abort the stream, this will not be done.

In order for /api/extra/last_logprobs to work, a genkey must be provided (same genkey used during the generation). Check the API docs for usage samples.

most of the rightmost cells are empty

Yes, this is due to samplers. Try setting it to a more creative one (e.g. top_p = 1, top_k = 300, temp=1) and you should see plenty of alternatives.

Right now, its limited to top 5 probabilities, this is not currently configurable.

top_k = 1, by definition, drops everything except the top token. So you will only see 1 option, because everything has been discarded. It's a truncation sampler.

[aleksusklim]

How about outputting RAW logits values before sampling (for a configurable amount, like yours 5) separately? This way, the client can join the probabilities map (token→percent) with logits array (pairs of token+logit) to show the real effect of the sampler:

• Strictly left to right by decreasing raw logit value;
• Two numbers in white for those which survived the sampler, percent and logit;
• Green color for the chosen one in its natural column;
• Gray color for discarded tokens, with logit-only values (that have 0% probability)

Does it sound good to you? This would allow exploring the effect of samplers like XTC which discards from the head and not the tail.

Also, I didn't understand why logits are lost when aborting, if you have a separate endpoint to get them? Is it an upstream limitation?

[LostRuins]

The logits are not "lost" per se, but they will be overwritten in the next request. You can still fetch them from the dedicated /api/extra/last_logprobs endpoint up to that time using the original Genkey. But usually the logic for abort is to "halt" the request and not make any further requests.

The unmodified logits are not stored anywhere currently, since the probability calculation and storage only happens after sampling is complete.

[aleksusklim]

So-o-o? Could Lite still (try to) fetch them even after abort?
Could you make a dedicated field with logits values before sampling (maybe via a server option to store them)?

I just want this feature to be as useful as I imagined it to be!

[aleksusklim]

Also: can you keep the previous logprobs list while a new request is being generated? I mean, in Lite: always download the table just after the generation finishes – and keep it available until the next generation finishes.

So the user could look at the table while the next prompt (or the same one, if Retry was pressed) is being generated in the background?
Because now it is kinda blocking the game: when logprops are opened I cannot schedule the generation, but if I'll close and start it – then View Logprobs shows "Not Available" already!

The workaround is to always duplicate the tab between turns…

[LostRuins]

Hmm the reason why I clear it on start of generation instead of at the end, is to deal with all the edge cases where generation fails, gets aborted, gets stalled, logprobs failed to return etc, I don't want cases where the logprobs shown are for the wrong/incomplete generation. Thus I clear it when a new generation starts.