feat: lsp semantic tokens provider

[He1pa]

lsp semantic tokens provider, i.e., semantic highlighting in editor

1. Does this PR affect any open issues?(Y/N) and add issue references (e.g. "fix #123", "re #123".):

• N
• Y [Enhancement][Track] KCL VSCode IDE extension and lsp enhancement #524

2. What is the scope of this PR (e.g. component or file name):

lsp

3. Provide a description of the PR(e.g. more details, effects, motivations or doc link):

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Other

4. Are there any breaking changes?(Y/N) and describe the breaking changes(e.g. more details, motivations or doc link):

• N
• Y

5. Are there test cases for these changes?(Y/N) select and add more details, references or doc links:

• Unit test
• Integration test
• Benchmark (add benchmark stats below)
• Manual test (add detailed scripts or steps below)
• Other

[Peefy]

@CodiumAI-Agent /review

[CodiumAI-Agent]

PR Analysis

• 🎯 Main theme: Implementing semantic tokens provider for the language server protocol (LSP) in the KCL language.
• 📝 PR summary: This PR adds a semantic tokens provider for the KCL language's LSP. It includes changes to multiple files, mainly in the kclvm/tools/src/LSP/src directory, to provide semantic highlighting in the editor. It also includes a test file to verify the functionality.
• 📌 Type of PR: Enhancement
• 🧪 Relevant tests added: Yes
• ⏱️ Estimated effort to review [1-5]: 4, because the PR involves changes in multiple files and the addition of a new feature, which requires a good understanding of the LSP and the KCL language.
• 🔒 Security concerns: No security concerns found

PR Feedback

💡 General suggestions: The PR is well-structured and the changes are logically grouped. The addition of semantic tokens provider is a valuable enhancement for the KCL language's LSP. It would be beneficial to include more comments in the code to explain the logic and functionality of the new feature.

🤖 Code feedback:

---

relevant file	kclvm/tools/src/LSP/src/semantic_token.rs
suggestion	Consider adding error handling for the case when gs.get_symbols().get_symbol(*symbol_ref) returns None. This could prevent potential panics in the code. [important]
relevant line	"+ if let Some(symbol) = gs.get_symbols().get_symbol(*symbol_ref) {"

---

relevant file	kclvm/tools/src/LSP/src/semantic_token.rs
suggestion	It might be more efficient to use a hashmap for LEGEND_TYPE instead of an array, to speed up the lookup time from O(n) to O(1). [medium]
relevant line	"+pub const LEGEND_TYPE: &[SemanticTokenType] = &["

---

relevant file	kclvm/tools/src/LSP/src/request.rs
suggestion	It would be good to handle the case where snapshot.get_db(&path.clone().into()) returns None, to avoid potential panics. [important]
relevant line	"+ let db = snapshot.get_db(&path.clone().into())?;"

---

relevant file	kclvm/tools/src/LSP/src/semantic_token.rs
suggestion	Consider refactoring the semantic_tokens_full function to reduce its complexity and improve readability. You could extract some of the logic into separate helper functions. [medium]
relevant line	"+pub(crate) fn semantic_tokens_full(file: &str, gs: &GlobalState) -> Option {"

---

✨ Usage tips:

---

Tag me in a comment '@CodiumAI-Agent' and add one of the following commands:

• /review: Request a review of your Pull Request.
• /describe: Update the PR title and description based on the contents of the PR.
• /improve [--extended]: Suggest code improvements. Extended mode provides a higher quality feedback.
• /ask <QUESTION>: Ask a question about the PR.
• /update_changelog: Update the changelog based on the PR's contents.
• /add_docs 💎: Generate docstring for new components introduced in the PR.
• /generate_labels 💎: Generate labels for the PR based on the PR's contents.
• /analyze 💎: Automatically analyzes the PR, and presents changes walkthrough for each component.

See the tools guide for more details.
To edit any configuration parameter from the configuration.toml, add --config_path=new_value.
For example: /review --pr_reviewer.extra_instructions="focus on the file: ..."
To list the possible configuration parameters, add a /config comment.

[coveralls]

Pull Request Test Coverage Report for Build 7346371503

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage remained the same at 87.787%

---

Totals
Change from base Build 7346367346:	0.0%
Covered Lines:	2293
Relevant Lines:	2612

---

💛 - Coveralls

[coveralls]

Pull Request Test Coverage Report for Build 7382719301

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage decreased (-15.0%) to 72.777%

---

Totals
Change from base Build 7346367346:	-15.0%
Covered Lines:	43287
Relevant Lines:	59479

---

💛 - Coveralls

[Peefy]

LGTM

[Peefy]

LGTM