Skip to content

brupelo/py2bin

BranchesTags

Repository files navigation

Estimated reading time: 5 minutes

1 🚀 py2bin

Status: Proof of Concept (POC)

py2bin is an ambitious project aimed at transforming Python code into standalone executables. By transpiling Python bytecode to LLVM Intermediate Representation (IR) and generating native machine code, py2bin bridges the gap between Python's simplicity and the performance of compiled languages.

Whether you're passionate about Python, compilers, or cutting-edge programming tools, py2bin offers an exciting opportunity to contribute to a groundbreaking initiative.

1.1 🌟 Why py2bin?

In a world where Python dominates, deploying standalone executables without requiring a runtime is a game-changer. py2bin:

  • Eliminates the need for Python environments in production.
  • Aims to produce optimized, fast, and portable executables.
  • Opens up new possibilities for Python in resource-constrained environments.

1.2 🚧 Current Status

py2bin is currently in the Proof of Concept (POC) stage. The focus is to validate the approach, test feasibility for real-world applications, and explore various use cases.

This is an ideal time to jump in and help shape py2bin's direction!

1.3 🎯 Roadmap and Milestones

The project is structured into clear milestones, each designed to break down development into manageable phases. Contributors of all experience levels can find meaningful ways to get involved.

1.3.1 Milestone 1: Foundations 🚀

Status: ✅ Mostly Complete

This milestone establishes the core structure of the project, allowing parallel contributions.

  • Bootstrap the repository with a basic transpiler covering all bytecode handlers.
  • Create a reporting mechanism to efficiently test the transpiler using AI and LLMs.
  • Ensure basic Python snippets generate native executables.
  • Transition to a standalone repository architecture, eliminating the need for an external py2bin-tests folder.
  • Enable parallel contributions without conflicts through a structured architecture.
  • Achieve a zero-failure rate on internal test suites.
  • Establish baseline code coverage.
  • Assign small tasks to new contributors, simplifying onboarding and providing clear documentation.
  • Implement all bytecode handlers in the standalone architecture, even if not fully optimized.
  • Establish a steady development rhythm so new contributors can become core members and help the repo grow organically.
  • Generate executable files from basic Python snippets.

Next Steps:

  • Refine the internal test suite for greater robustness.
  • Improve contributor onboarding with detailed guides and hands-on examples.
  • Encourage new contributors to take ownership of specific areas of the codebase.

1.3.2 Milestone 2: Bytecode Handling 🔧

Status: 🚧 In Progress

Ensuring a solid bytecode foundation by expanding testing coverage and validation.

  • Ensure bytecode handlers are well-tested, covering more than just happy paths.
  • Expand test coverage to include edge cases and complex bytecode scenarios.
  • Ensure seamless integration of new bytecode handlers into the existing architecture.

Next Steps:

  • Prioritize rigorous testing and validation of bytecode handlers.
  • Document best practices for bytecode handling to support contributors.
  • Establish automated tests for edge cases to maintain reliability.

1.3.3 Milestone 3: Mid-Level Python Code 🧩

Status: ⏳ Upcoming

Adding support for mid-level Python constructs to extend the transpiler’s capabilities.

  • Successfully transpile mid-level Python code (e.g., Shedskin dataset).
  • Develop a robust mechanism for handling Python constructs like classes, decorators, and generators.
  • Integrate mid-level code testing into the internal test suite.

Next Steps:

  • Identify and address gaps in handling mid-level Python features.
  • Collaborate with contributors to expand test cases for mid-level code.
  • Provide examples of transpiled mid-level Python code for reference.

1.3.4 Milestone 4: External Dependencies 🌐

Status: ⏳ Upcoming

Enabling support for third-party dependencies to broaden use cases.

  • Ensure mid-level Python code using external dependencies (e.g., numpy, requests) transpiles correctly.
  • Develop a dependency resolution mechanism for external libraries.
  • Test and validate transpilation with popular third-party packages.

Next Steps:

  • Research and implement strategies for handling external dependencies efficiently.
  • Create a curated list of supported libraries for early adopters.
  • Develop a mechanism to identify and flag unsupported dependencies early.

1.3.5 Milestone 5: Production-Level Packages 📦

Status: ⏳ Upcoming

Moving toward real-world applications by ensuring compatibility with production-grade projects.

  • Transpile production-level Python packages with accuracy and reliability.
  • Implement a command to build full pyproject.toml Python projects (partially complete).
  • Ensure compatibility with common packaging tools (e.g., setuptools, poetry).
  • Validate transpilation with real-world production packages.

Next Steps:

  • Expand the pyproject.toml build command to handle complex project structures.
  • Collaborate with the community to test and refine production-level transpilation.
  • Develop a benchmark system to compare performance across different package configurations.

1.3.6 Milestone 6: Optimization and Benchmarking 📊

Status: ⏳ Upcoming

Optimizing performance and benchmarking against existing solutions.

  • Benchmark performance and compare py2bin with alternative tools (e.g., Cython, Nuitka).
  • Begin optimization for real-world usage, focusing on execution speed and resource efficiency.
  • Publish performance metrics and improvement plans.

Next Steps:

  • Identify key performance bottlenecks and prioritize optimization efforts.
  • Engage the community in benchmarking and performance challenges.
  • Develop a feedback loop to continuously measure and improve performance.

1.4 🔍 How Does py2bin Compare?

py2bin stands among a variety of tools that address Python deployment and optimization in different ways. Here’s a list of related tools and projects that have inspired py2bin or operate in adjacent spaces. Each has its unique approach, and py2bin aims to complement the ecosystem with its focus on transpiling Python bytecode into LLVM Intermediate Representation (IR) for standalone executables.

1.4.1 Related Tools

  • bazel A build tool that supports Python, among other languages, with a focus on reproducible and fast builds.

  • cx-freeze A tool for creating standalone executables from Python scripts.

  • cython A static compiler for Python that transforms Python code into C or C++ extensions.

  • docker Containerization technology for packaging and distributing applications, including Python environments.

  • mojo The Mojo Programming Language

  • Nuitka A Python compiler that converts Python scripts into standalone executables or extension modules.

  • numba A just-in-time (JIT) compiler for Python, specializing in numerical and array-heavy code.

  • pex A tool for creating Python executables as .pex files that encapsulate a Python environment.

  • py14 Archived

  • py2app A tool for packaging Python scripts as standalone macOS applications.

  • PyCer PyCer, a Python to C++ compiler written in Python. Heavily based on Polyakov Konstantin's py2c project.

  • py2exe A similar tool for packaging Python scripts as standalone Windows executables.

  • py2many Transpiler of Python to many other languages

  • pyinstaller A popular tool for bundling Python applications into standalone executables for various platforms.

  • pynsist A Windows installer builder for Python applications.

  • PyRun A minimal Python runtime environment for running Python applications.

  • shedskin A Python-to-C++ compiler focused on statically-typed Python subsets.

  • shiv A tool for creating self-contained Python executables using zipapp.

  • sy_py2c Translation Python source into C.

  • xar A binary packaging format that can embed Python interpreters and libraries.

1.4.2 🌟 Inspiration and Complementary Tools

py2bin draws inspiration from the above tools, each addressing specific needs in the Python ecosystem. By focusing on transpiling Python bytecode to LLVM IR, py2bin aims to bring something unique to the table—targeting scenarios where lightweight, optimized, and standalone binaries are essential. Our goal is not to replace any tool but to add a new option for developers to explore.

If you're familiar with any of these tools or excited by their possibilities, py2bin invites you to contribute and help expand the boundaries of Python deployment!

1.5 🙌 How You Can Contribute

py2bin is currently in a technical Proof of Concept (POC) stage and requires expertise in bytecode handling, LLVM Intermediate Representation (IR), and compiler/transpiler development. Contributions from experienced developers are essential at this phase.

1.5.1 Ways to Contribute

1.5.1.1 Developers

  • Implement bytecode handlers.
  • Optimize LLVM IR generation.
  • Enhance the performance and functionality of the transpiler.

1.5.1.2 Testers

  • Provide datasets or test cases.
  • Test py2bin with real-world Python code and report issues.

1.5.1.3 Documentation & Community (Future Need)

  • Non-coding contributions (guides, tutorials) will be prioritized once py2bin is production-ready.

1.5.2 Contribution Guidelines

  1. Development-Only Issues – Only technical issues related to development and functionality will be considered.
  2. Experience Required – py2bin involves IR generation, transpilation, and compiler design. Ensure familiarity with these before contributing.
  3. No General Support – As an experimental project, general support will be provided once it matures.

1.6 📂 Setting Up and Running the Repo

1.6.1 Prerequisites

Ensure you have the following installed:

  • Just – Task runner.
  • Uv – Virtual environment and dependency manager.
  • Ruff – Python linter and formatter.
  • Clang – Required for compiling .ll (LLVM IR) code into executables.

To verify Clang installation, run:

clang --version

1.6.2 Setting Up the Environment

Run the following in the repo's root directory:

just b .

This sets up a virtual environment and installs dependencies. Once done, py2bin will be available in your PATH.

1.6.3 Using py2bin

Run:

py2bin --help

Available commands:

╭─ Commands ─────────────────────────────────────────────────────────────────────────────────────────╮
│ batch       Process all Python files in the specified input directory.                             │
│ build       Build an entire Python project into an executable from pyproject.toml (WIP).          │
│ check       Transpile a Python file to LLVM IR and print its content for verification.             │
│ test        Process a single Python file, transpile it, and test the output.                       │
│ transpile   Transpile a single Python file to an executable.                                       │
╰────────────────────────────────────────────────────────────────────────────────────────────────────╯

Key Commands:

  • py2bin test – Transpiles a single Python file and tests the output. This is the main command for debugging.
  • py2bin batch – Runs transpilation and tests on multiple scripts, ensuring broader verification before submitting changes.

Notes:

  • The build command is a work in progress and not ready for use.
  • The test command is a subset of transpile with additional validation, making it the primary tool for iteration.
  • While transpile alone is useful, test and batch will be the most frequently used commands.

1.6.4 Before Submitting a PR

Ensure the following pass:

just lint  # Linter should exit with code 0
just test --cov-report=term  # Tests should exit with code 0
cd tests/done && py2bin batch .  # Should exit with code 0
cd tests/doing/<your-folder> && py2bin batch .  # Should exit with code 0

1.6.5 Folder Structure

TESTS
├───doing/    # Contributor-specific folders for work-in-progress
├───done/     # Successfully transpiled scripts
└───todo/     # Scripts needing fixes

This structure ensures clear organization and collaboration.

1.7 🚀 Maximizing Productivity with AI

We encourage the use of tools like ChatGPT, Claude, DeepSeek, and Gemini to:

  • Understand transpiler architecture.
  • Debug issues efficiently.
  • Generate optimizations or new feature ideas.

However, AI tools should complement, not replace, technical expertise. Foundational knowledge of Python bytecode, LLVM IR, and transpilers will help craft better prompts and interpret AI-generated outputs effectively.

Best Practices:

  • Experiment within your tests/doing/<your-folder>.
  • Improve unit tests in py2bin/handlers/tests or py2bin/core/tests.
  • Use AI tools for small, precise tasks while applying engineering judgment.

1.8 🎯 Final Thoughts

py2bin is an ambitious experiment exploring:

  • Compiler Design – Deep dive into how code is translated into executables.
  • Transpilers & Bytecode – Learn how Python bytecode is transformed and optimized.
  • LLVM IR – Work with one of the most powerful compiler infrastructures.
  • Advanced Systems Knowledge – Bridge the gap between high-level programming and system execution.

Whether we succeed or not, the journey itself will be a valuable learning experience. Let’s push the boundaries of Python deployment together!

1.9 ⚖️ Legal

1.9.1 🚧 Proof of Concept Status

py2bin is currently in its Proof of Concept (POC) stage. While I am fully committed to helping contributors make tangible progress over the next few months, I must emphasize that this project is experimental in nature. If I find the project becomes stagnant or the approach proves infeasible, I may decide to shut it down.

For this reason, a fork-based approach is highly recommended for all contributors. By maintaining forks of the repository, you ensure that even if this project is discontinued, the work can continue and be improved independently. Please understand that no responsibility will be held by myself or any contributors if the project does not achieve its goals.

1.9.2 🔐 Legal Disclaimer

By contributing to this repository, you acknowledge and agree to the terms outlined in the Legal Disclaimer.

1.9.3 📄 License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.

About

No description, website, or topics provided.

Resources

Readme

License

Apache-2.0 license
Activity

Stars

0 stars

Watchers

2 watching

Forks

0 forks
Report repository

Releases

1 tags

Packages

No packages published

Languages