Highlight only visible text

[Moosems]

Here's the current plan:

1. Create a new tag named "MLCDS" to remember where Docstrings (DS), Multi Line Comments (MLCs), and backstick code areas (markdown) start and end. This tag should not start with "Token" so that it is not deleted later.
2. Make a method that takes the tokens and tags and areas to remove and returns the tags without those areas.
3. When highlight() runs, check if anything has changed in the text widget. If not, return. (Things to look for include viewable area, text, and so on.)
4. Update the code to add MLCDS tags to the text widget where necessary. If a line starts or ends with a Docstring or MLC, add an MLCDS tag to the entire line.
5. When the visible area of the text widget changes, get the visible area, visible text, and line offset, as before.
6. Work with the MLCDS tags to add the necessary starts and ends to the visible text and then lex it using Pygments.
7. Splice the tags to remove parts that are not visible (columns to the left or right of the visible area) and remove tags from emojis.
8. Add the remaining tags to the visible text and display it in the text widget.
9. Not in method: update the docs when this is done.
10. Also not in method: Make sure that when typing, the typing area is viewable (see()).

In summary, the changes will make the text widget display only the visible area, while adding the necessary context outside the viewport using MLCDS tags.

[Moosems]

Could you review the plan and give your thoughts/ideas/suggestions/changes?

[rdbende]

I like your plan. The only thing I have concerns about is the emoji-in-tag-range
checking thingy. I'm not sure how would it perform.

[Moosems]

Check for a set of characters in the text, if any ranges are found, remove it from any relevant tags (might make a method that takes the tokens and tags and areas to remove and returns the tags without those areas).

[rdbende]

Check for a set of characters in the text, if any ranges are found, remove it from any relevant tags (might make a method that takes the tokens and tags and areas to remove and returns the tags without those areas).

But that would run on every keypress?

[rdbende]

Oh no

[Moosems]

Yes, this runs on every keypress. Maybe I should add a check at the beginning of the method to check if anything has changed and if something has, only then will it all run. Another check could be if the return key was pressed and only highlight the whole thing in that case because otherwise only the line with the cursor needs to be changed.

[Moosems]

A method to scroll to where someone types that works properly is also in order.

[Moosems]

@rdbende, this runsbefore the highlighting. The issue? The check for MLCDS tags relies on checking for "Token.Literal.String.Doc", "Token.Comment.Multiline", "Token.Literal.String.Backtick" Tokens. What would you recommend? My thought is lex the line with the cursor if it is an insert, removal, or replacement event and use that to determine in that case if the MLCDS tags need updating.

[Moosems]

I wonder if it's the lexing or tags that takes up the computation time. If it's just the tags then it wouldn't be that hard to simply add the tags that are in the visible area and lex the whole files such that MLCDS isn't an issue otherwise it's going to be the whole arduous process.