Super-preferred shortcuts

[psads-git]

Suppose that we have the two following shortcuts:

p, that expands to Paul

ps, that expands to Paul Something

If p were a super-preferred shortcut, it should be always the first suggestion. This would be very useful, as one could type

p+1

and be sure that one would get the super-preferred shortcut and not Paul Something.

[mike-fabian]

Would it be enough if the “learning” of how often which shortcuts have been used worked correctly?

When I read your issue first, I wondered why this is needed because I thought that the learning mechanism should put the most used shortcuts on top automatically, o.e. if “Paul Something” had been used more often, this would appear on top when typing only p, amd if “Paul” had been used more often, “Paul” would appear on top when only typing “p” (If “ps” is typed, of course only the shortcut for “Paul Something” matches).

Then I tried this out and found to my surprise that it doesn’t work that way at the moment.

I consider this a bug.

Would it be enough to just fix this bug?

Detailed reason why this learning doesn’t work correctly for shortcuts at the moment:

I created the two shortcuts you mentioned when creating the issue and tested a while and got this:

$ python3 tabstatistics.py -r  | grep Paul
  75801       1 2022-07-27 15:35:26.593667 '' 'paul something' 'p' 'Paul Something' 
  75804       1 2022-07-27 15:43:11.655511 'i' 'like' 'p' 'Paul Something' 
  75800       5 2022-07-27 15:38:01.600243 '' '' 'p' 'Paul Something' 
  75797 1000001 2022-07-27 15:41:45.317870 '' '' 'ps' 'Paul Something' 
  75799 1000002 2022-07-27 15:38:52.350897 '' '' 'p' 'Paul' 

Shortcuts have a count of one million (“1000000”) when they are created. That’s how they are distinguished from regular entries in the database. That is a weird hack, I used this hack to avoid adding an extra column to the database. Maybe that was a mistake, maybe I should have added an extra column to the database only for shortcuts, but it seemed to work well enough and I avoided to have a complicated migration to a newer database version when updating ibus-typing-booster. It seems to work well enough though, the only problem seems to be that it is theoretically possible to create a shortcut by typing the exact same thing (and with emtpy context!) one million times. And that is hardly possible in practice.

In the output of my tabstatistics.py tool I see that:

• One time I typed “ps” and selected “Paul Something” while the context was empty, this increased the count to 1000001.
• Two times I typed “p” and selected “Paul” while the context was empty, this increased the count to 1000002.
• Five times I typed “p” and selected “Paul Something” while the context was empty, this created the line
  75800 5 2022-07-27 15:38:01.600243 '' '' 'p' 'Paul Something'
  but this did not increase the count of the shortcut “Paul Something”. I think it should have increased that count by 5.
• Two more times I typed “p” and selected “Paul Something” while the context was not empty, this did not increase the count of the shortcut “Paul Something” either. Instead it created the lines:
  75801 1 2022-07-27 15:35:26.593667 '' 'paul something' 'p' 'Paul Something'
75804 1 2022-07-27 15:43:11.655511 'i' 'like' 'p' 'Paul Something'

Creating that extra lines is OK, but in addition to that, each time “Paul Something” was selected, the count of the shortcut should have been increased.

I.e. the final count of “Paul Something” shortcut should have been 1000010.

So this will certainly need to get fixed.

If this is fixed, the shortcut which is used most often will be on top of the candiate list.

I will fix this first and you can then test whether this fixes the problem for you.

If this does not yet solve the problem for you perfectly, then I could still add a “priority” column to the page in the setup tool where shortcuts are defined. Maybe like this:

                        Priority
p   Paul                5
p   Pauline             2
p   Pauli               2
p   Paula               0
ps  Paul Something      3
ps  Paul Something Else 0

The default priority would be 0 but the user could specify a priority higher than 0 if desired.

In the above example, when you type “p”, “Paul” would always be on top no matter how often you select “Pauline”, “Pauli”, or “Paula”. And when typing “ps” then “Paul Something” would always be on top no matter how often you select “Paul Something Else”.

For shortcuts with the same priority, like

p   Pauline             2
p   Pauli               2

The one selected most often would appear above the shortcuts with the same priority selected less often.

[mike-fabian]

I created a new issue for the bug I described above, I’ll fix that first and then we can see whether it is “good enough” or whether we need to add the possibility to manually prioritize shortcuts.

[psads-git]

Thanks, Mike! I suspect it will be enough to fix the bug.

Perhaps, in addition, it would be helpful have the shortcuts expanding in inline mode. For instance, typing ps should automatically expand to

Paul Something

although without auto-committing it.

[mike-fabian]

https://copr.fedorainfracloud.org/coprs/mfabian/ibus-typing-booster/builds/ has test builds of 2.18.5 with a fix for the bug #353

[mike-fabian]

Perhaps, in addition, it would be helpful have the shortcuts expanding in inline mode. For instance, typing ps should automatically expand to

Paul Something

I don’t think inline completion is useful if it does not really only complete but change what was already typed.

Currently, I show completions only inline if the completed word starts exactly with the characters which have been typed.

I see no way at the moment to shown inline completions if this is not the case without causing a lot of confusion.

Maybe there are some possibilities, but I don’t have any good ideas about that at the moment.

[psads-git]

Thanks, Mike!

[mike-fabian]

Does the improved learning of the shortcuts help?

If you have any exact ideas how inline completion mode could be improved, I am very interested of course.
Exact descriptions like “if these letters are typed and the suggested text is this, what should be displayed in the standard black colour and what should be displayed in the inline completion colour and what should happen if one more letter is typed and what should happen if one letter is deleted with backspace.”

[psads-git]

Thanks, Mike. Yes, now it seems that the shortcuts are working fine!

My suggestion of expanding shortcuts in inline mode aimed to have a way to simplify the committing of shortcuts. For instance, when I write the shortcut ps, I am sure that I want to commit

Paul Something.

In circumstances where one is certain that the shortcut is to be committed, as ibus-typing-booster is now, it takes two steps to commit it:

1. Typing the shortcut.

2. Press TAB + SPACE.

So, this way is a little inefficient, as the shortcut could be committed straightway.

[mike-fabian]

We have his new feature to automatically select the first candidate when it is very likely.

Currently, “very likely” is defined as

• context of a least 2 words exists
• context and first candidate can be found in a single dictionary
• first candidate differs from he typed word only by accents
• first candidate has a rather high score

Do you use this option?

If the rules above match, a candidate table pops up even if the inline completion mode “Yes, without fallback to popup” is used. Because this cannot be reasonably displayed inline, the word has already been fully typed and the candidate changes only the accents of the already typed word. That would be confusing. But if these rules really match, then the suggestion is so good that it is warranted to make an exception and pop up the candidate list anyway, I thought.

I am thinking about changing the meaning of “very likely” to mean

• the rules above

or

• the first candidate is a user shortcut expansion

In that case

1. Typing the shortcut

would be enough to pop up the candidate list and select the most often used matching and

2. SPACE

would already commit and you would save one key (the TAB).

If you have many shortcuts starting with regular letters, this would make the candidate list pop up quite often even with inline completion mode “Yes, without fallback to popup”. For every word starting with “p” you would get a popup …

Not only do you get annoying popups you might not want by just typing any word starting with “p” even when you don’t have the intention to use a shortcut but just want to type a regular word. Also, shortcuts starting with letters interfere quite badly with predictions of regular words. If you have shortcuts starting with “p”, you never get nice context sensitive predictions for words starting with “p” because the shortcuts will always win.

But a while ago we introduced the possibility to start shortcut definitions with characters which are not letters. So instead of defining shortcuts “p”, “ps”, ... I think it is better to define shortcuts like “\p”, “\ps”, ... using a start character for shorcuts which is not a letter and which is rarely used. This avoids interference with word predictions and avoids popping up candidate lists when they are not wanted.

[psads-git]

Thanks, Mike! Sounds quite fine what you are proposing. Only, I am not very sure whether the meaning of “very likely” should separate, for the user, the proposed two options. Thus, the user could select option1 or option2 or both simultaneously.

[mike-fabian]

No, I don't want to add a new option, just autoselect shortcut expansions automatically (only sometimes!), if the “Automatically select best candidate” option is set to “Yes, but only when extremely likely”.

I wrote “only sometimes!” because autoselecting always when a shortcut expansion is the first candidate will not work well.

Because when a candidate is selected and then typing continues, the selected candidate is copied into preedit and the new typing is appended. For example, if you type “r” and have a candidate “read” and select it by typing the right number of Tab keys, and then continue type “ing”, the result is “reading”. That makes possible to select candidates which complete partially what you want and continue typing the rest, which is often very useful.

But that means that if you have shortcuts

“p” “Paul”
“ps” “Paul Something”

and you type “p” then it is not a good idea to automatically select whatever shortcut happens to be on top (the most often used one) because that would make it difficult to type the other one. Let’s say you type “p” and “Paul” is on top and you type “s” next you would get “Pauls” and if “Paul Something” is on top you would get “Paul Somethings”. Which is probably not wanted. And f that is not wanted, one would need to cancel the selection with Escape before typing the “s”. It would also make it hard to type regular words starting with “p”. If you try to type ”power” and the “p” already automatically selects “Paul” and you don't cancel with Escape, then you get “Paulower”. This would be extremely annoying.

I need to think more whether here are cases left when autoselection of shortcut is possible without causing annoying side effects.

At least I would need to add another constraint

• the first candidate is a user shortcut expansion
• this is the only shortcut expansion in the candidate list ( as I explained above without this constraint it is difficult to select other shortcut expansions in the same candidate list)

And probably that is still not enough, one probably needs to add something to avoid conflicts regular words. like maybe

• There are no candidates which are not shortcut expansions and which complete the typed letters to a regular word.

I will continue to think a bit about this, but I think it is quite hard to figure out whether this shortcut expansion is so likely correct that it can be automatically selected.

[psads-git]

Thanks, Mike. I think the way you are thinking about how to make shortcuts automatically selected is correct. If you will eventually implement such a partially automatic shortcut selection, the user can optimize the shortcuts such that they are eligible by the rules of automatic selection -- and this will be a nice ibus-typing-booster improvement.

[mike-fabian]

I found a weird bug in the shortcut tab of the setup tool: #355

[psads-git]

Thanks , Mike, for letting me know that.

[mike-fabian]

I have another idea which might be better because it might make it possible to automatically select shortcuts more often at the cost of a little inconsistency when continuing to type while something is selected in the candidate list.

As I explained above, if something is selected in the candidate list and more keys are typed which do not cause commits (like space for example) but add input, then the selection is first copied to the preedit and the new input appended.

This makes it possible to type “r”, get the suggestion “read”, select it, type “i” and get “read” moved to the preedit and get the input “i” appended. Now there is “readi” in the preedit and probably the suggestion “reading” will be shown.

I find this very useful when typing normal words.

But, maybe this is not so useful for shortcut expansions. Does one really often want to add additional letters to a shortcut expansion and try a completion again with the added letters included? At least for the shortcuts I am using, it does not seem to make much sense to append anything immediately without a space in between.

If that is the case not only for me but also for you and most other users, then I could make the shortcut expansions behave slightly differently when typing more letters:

• selection is a normal word:
  -- additional` typing copies the selection to preedit and appends the additional typing
• selection is a shortcut expansion:
  -- additional typing does not copy the selection to the preedit, instead the additional typing is directly added to the preedit and a new candidate list is calculated

[mike-fabian]

That would make it possible to auto select shortcut expansions even if there are more than one in candidate list, it would still be possible to get a shortcut expansion which is not at the first position by typing more letters and it would still be possible to type normal words by typing more letters.

For example, if you type p and the first candidate is the shortcut expansion “Paul“ and the second is the shortcut expansion “Paul Something”, they you could still add an s to get “Paul Something” selected (instead of getting “Pauls”) and you could still continue to type ower to get “power” instead of “Paulower”.

This would work much more often than my previous idea, one would save the Tab key to select much more often.

The problem is the slight inconsistency, adding input when a shortcut expansion is selected would then behave differently from when something which is not a shortcut expansion is selected.

[psads-git]

Formidable idea, Mike! And if it does not break consistency too much, maybe it is worth trying.