Add Python stub file generation with documentation parsing #388

andrewleech · 2025-06-06T12:10:03Z

Summary

This PR adds comprehensive Python stub file (.pyi) generation for the LVGL MicroPython bindings, providing full IDE support with autocompletion, type hints, and rich documentation.

Key Features:

🚀 Fast Parallel Processing: 6 seconds vs. minutes (uses all CPU cores)
📝 Rich Documentation: Automatic extraction from 1400+ LVGL functions
🎯 IDE Integration: Full autocompletion and type hints (.pyi files)
⚡ Separate Build: Doesn't slow down main MicroPython builds
🔧 Smart Formatting: Bullet points, text wrapping, proper Python conventions
🔗 Source Navigation: File:line references to original C implementation

The implementation includes:

Stub Generation: Creates .pyi files with proper Python type hints
Documentation Parsing: Extracts Doxygen comments from C headers using parallel processing
Smart Parameter Handling: Converts obj to self for class methods
Performance Optimization: Processes 209 header files in ~6 seconds using all CPU cores
Source References: Adds file:line references for navigation to C implementation

Testing

Tested on Unix port with full stub generation:

Processes 209 LVGL header files using parallel processing
Extracts documentation from 1423 functions
Generates type hints for 41 widget classes and 64 enums
Produces comprehensive .pyi files for IDE consumption

The generated stubs provide full autocompletion and documentation in modern Python IDEs like VS Code, PyCharm, etc.

Trade-offs and Alternatives

Trade-offs:

Adds ~6 seconds to generate full documentation (but as separate optional target)
Increases repository size slightly with documentation files

Alternatives considered:

External documentation parsing libraries (rejected to minimize dependencies)
Manual stub file maintenance (rejected due to maintenance burden)
No documentation extraction (rejected as it provides significant developer value)

The implementation uses custom regex-based Doxygen parsing to avoid external dependencies while providing exactly the functionality needed for LVGL's documentation format.

🤖 Generated with Claude Code

Co-Authored-By: Claude [email protected]

PGNetHun

Hello!

Thanks for all these changes, the stubs will be a big help during MicroPython development, I have left only a few comments.

PGNetHun · 2025-07-13T07:19:02Z

micropython.mk

 LVGL_MPY: $(LVGL_MPY)

+# Generate Python stub files with documentation (slow - parses 200+ header files)
+lvgl-stubs: $(LVGL_STUBS_FILE)


Please make target upper-case, and follow naming convention

Suggested change

lvgl-stubs: $(LVGL_STUBS_FILE)

LVGL_STUBS: $(LVGL_STUBS_FILE)

Or is there other reason (that I don't know) that target is lowercase lvgl-stubs ?

No I didn't realise there was a convention in place, happy to fix.

PGNetHun · 2025-07-13T07:25:41Z

CLAUDE.md

I guess this file is generated using AI.

This is a guidance file for Claude yes, it's important during ongoing development with Claude but I wouldn't expect it be included in the repo here (I usually gitignore it for repos I don't own).
I'll remove it before taking PR out of draft.

Great, thanks!

PGNetHun · 2025-07-13T07:27:50Z

LVGL_DEVELOPMENT_NOTES.md

+**Quick Build Command:**
+```bash
+# From micropython root directory
+make -j -C ports/unix USER_C_MODULES=$(pwd)/lib


I think this is incorrect: lib folder contains only Python files, not the user C modules.

I guess this file was generated using AI.

Sorry for the confusion; during development with Claude Code the biggest benefits come from Claude being about to build and test the code it writes.

During development I had this repo checked out as a submodule in micropython/lib folder then this was the command used to build micropython with lvgl as a user C module.

I didn't intend this doc to be included in the PR, it was just a reference for Claude itself to use.

No problem, don't need to say sorry, I'm happy that you do all these stuff, thanks, I just review and add comments, and as you mentioned, this is just a draft PR.
Keep up the good work! 👍

PGNetHun · 2025-07-13T07:28:22Z

LVGL_DEVELOPMENT_NOTES.md

Please avoid generating everything with AI, as that usually products incorrect text and code.

A few months ago I generally said the exact same thing, however with Claude Code and a bit of practice / refinement of my process I've had a dramatic increase in my personal quality and quantity of work.

With its assistance I've been able to tackle a significant number of large features / issues that have been on my personal backlog for years due to lack of the time to take them on!

It's not a silver bullet and does take practice and discipline to get finished work out and tested, but it's changing the shape of embedded development for me.

But yes everything needs real review and testing, which is incomplete here certainly.

PGNetHun · 2025-07-14T08:14:08Z

gen/gen_mpy.py

+
+    # Find all header files
+    header_files = []
+    search_dirs = [


Could you please move these search dirs list to the beginning of file as an array (constant)?
So later we can easily extend or update the list of dirs, and no need to search for this in the 4000+ lines of code.
HEADERS_SEARCH_DIRS = ["src/widgets", "src/core", "src/misc", "src/draw"]

PGNetHun · 2025-07-14T08:19:08Z

gen/gen_mpy.py

Could you please move all stubs generation implementation into a separate gen_stubs.py module?
Then call it from this gen_mpy.py.
Just to make code cleaner, and code/stubs generators separate.

But if your stubs generation implementation can run independent from gen_mpy.py (because no gen_mpy functions are used), then I vote for having gen_stubs.py file as a separate runnable script, and call it from makefile.
So, basically separate gen_mpy.py and gen_stubs.py

That's a good idea. I'd initially thought the stubs would be easier to generate as the functions are being parsed but yeah it ended up working better as a separate pass looking for different details. It should be straightforward now to split it out into its own file and I'll see how it goes making it standalone!

andrewleech · 2025-07-14T08:47:17Z

Thanks for the feedback, though I must admit you caught me off guard. I left this PR in draft because I intended to do a manual cleanup before prompting for discussion.

Yes this has been put together with significant assistance from Claude Code AI, I have been getting very high quality code design acceleration with it both professionally and on open source work.

This PR is actually the first in a string of improvements I've been intermittently working on; these stubs after intended to help developers certainly, but I also used them as an input for Claude to reference in a follow up branch to fix and modernise the examples and bring them all up to valid v9 syntax.

Extends the LVGL bindings generator to automatically create Python stub files (.pyi) that provide type hints and enable IDE autocompletion for LVGL MicroPython bindings. Features: - Type mapping from C types to Python type hints - Single stub file with all widgets, functions, enums, and constants - Detailed documentation header with content statistics The generated lvgl.pyi file includes 40+ widget classes, 300+ functions, and comprehensive type information for better development experience. Include C function names, location and description in Python stub docstrings. The docstrings include a "C function: <name>" line before the "Source: <file>:<line>" reference. For example: - Python method `delete()` shows "C function: lv_obj_delete" - Python method `add_flag()` shows "C function: lv_obj_add_flag" Create a proper Python package structure for LVGL type stubs: - Create stubs/pyproject.toml with setuptools-scm versioning - Add stubs/lvgl-stubs/ package directory with __init__.py and py.typed - Update gen_mpy.py to output stubs to package directory by default - Add .gitignore to exclude generated .pyi files but include package structure - Update README.md with comprehensive IDE support documentation Signed-off-by: Andrew Leech <[email protected]>

Add automatic LVGL API JSON generation to both makefile and CMake build systems: - Generate JSON with LVGL's gen_json.py script for complete API metadata - Pass JSON to gen_mpy.py via -J flag for enhanced callback structure analysis - Add stub generation targets to CMake (delegates to makefile for now) - JSON provides missing structure field information for proper callback binding This ensures the MicroPython binding generator always has access to complete LVGL API metadata for proper callback structure analysis and user_data handling. Signed-off-by: Andrew Leech <[email protected]>

Refactor Python stub generation into dedicated module system: - Create gen_stubs.py: Standalone script for generating Python type stubs - Create gen_utils.py: Shared utilities between gen_mpy.py and gen_stubs.py - Update gen_mpy.py: Remove stub generation code, import shared utilities - Update pyproject.toml: Remove setuptools-scm, use __version__.py approach The new system generates complete Python type stubs with: - All LVGL widget classes with proper typing - Function signatures with parameter names and types - Enum definitions and constants - Documentation extracted from C headers - Validation to ensure stub-API consistency Signed-off-by: Andrew Leech <[email protected]>

andrewleech mentioned this pull request Jun 6, 2025

Add Python stub generation. #387

Closed

andrewleech force-pushed the python-stub-generation branch from f72bde8 to 32986d0 Compare June 10, 2025 02:32

PGNetHun reviewed Jul 14, 2025

View reviewed changes

PGNetHun added the enhancement New feature or request label Jul 14, 2025

pi-anl added 3 commits July 17, 2025 11:58

andrewleech force-pushed the python-stub-generation branch from 32986d0 to 0e8f6ea Compare July 18, 2025 05:18

	lvgl-stubs: $(LVGL_STUBS_FILE)
	LVGL_STUBS: $(LVGL_STUBS_FILE)

Add Python stub file generation with documentation parsing #388

Are you sure you want to change the base?

Add Python stub file generation with documentation parsing #388

Uh oh!

Conversation

andrewleech commented Jun 6, 2025

Summary

Testing

Trade-offs and Alternatives

Uh oh!

PGNetHun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewleech Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PGNetHun Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PGNetHun Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewleech commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

PGNetHun left a comment •

edited

Loading

andrewleech Jul 14, 2025 •

edited

Loading

PGNetHun Jul 14, 2025 •

edited

Loading

PGNetHun Jul 14, 2025 •

edited

Loading

andrewleech commented Jul 14, 2025 •

edited

Loading