dev-intro.md

Introduction for Mypyc Contributors

Supported Features

Only a small subset of Python is supported. If you try to compile something that is not supported, you are not likely to get a good error message.

Here's a summary of what should work:

• Top-level functions with required positional-only arguments.
• Calls to top-level functions defined in the same file.
• Types:
  • int
  • bool
  • List[...]
  • Tuple[...] (only a few tuple operations)
  • None as return type
• Some integer operations:
  • Basic integer arithmetic: + - * // %
  • Integer comparisons
• Some list operations:
  • [e, ...] (construct list)
  • l[n]
  • l[n] = x
  • l.append(x)
  • len(l)
  • l * n (multiply list by integer)
• Simple assignment statement var = x (only local variables).
• If/else/elif statement.
• While statement.
• Expression statement.
• Return statement.
• and and or in a boolean context.
• for x in range(n): ... (for convenience only).
• break and continue statements.

High-level Overview

Mypyc compiles a Python module to C, and compiles that to a Python C extension module.

It has these passes:

• Type check the code using mypy and infer types for variables and expressions.
• Translate the mypy AST into a mypyc-specific intermediate representation (IR).
  • The IR is defined in mypyc.ops.
  • The translation happens in mypyc.genops.
• Insert exception handling (mypyc.exceptions).
• Insert explicit reference count inc/dec opcodes (mypyc.refcount).
• Translate the IR into C (mypyc.emit*).
• Compile the generated C code using a C compiler.

Tests

The test cases are defined in the same format (.test) as used in the mypy project. Look at mypy developer documentation for a general overview of how things work. Test cases live under test-data/.

Type-checking Mypyc

One of the tests (test_self_type_check) type checks mypyc using mypy.

Overview of Generated C

Mypyc uses a tagged pointer representation for integers, char for booleans, and C structs for tuples. For most other objects mypyc uses the CPython PyObject *.

Mypyc compiles a function into two functions:

• The native function takes a fixed number of C arguments with the correct C types. It assumes that all argument have correct types.
• The wrapper function conforms to the Python C API calling convention and takes an arbitrary set of arguments. It processes the arguments, checks their types, unboxes values with special representations and calls the native function. The return value from the native function is translated back to a Python object ("boxing").

Calls to other compiled functions don't go through the Python module namespace but directly call the target native function. This makes calls very fast compared to CPython.

The generated code does runtime checking so that it can assume that values always have the declared types. Whenever accessing CPython values which might have unexpected types we need to insert a type check. For example, when getting a list item we need to insert a runtime type check (an unbox or a cast operation), since Python lists can contain arbitrary objects.

The generated code uses various helpers defined in lib-rt/CPy.h. The header should only contain inline or static functions, since we don't compile the C helpers into a separate object file.

Other Important Limitations

All of these limitations will likely be fixed in the future:

• There's currently no way to run the compiler other than through tests (test-data/run.test has end-to-end tests -- use these sparingly since they are expensive to run).

• We don't detect infinite recursion.

• We don't handle Ctrl-C in compiled code.

• We don't detect undefined local variables.

• There's no way to access most stdlib functionality.

Hints for Implementing Typical Mypyc Features

This section gives an overview of where to look for and what to do to implement specific kinds of mypyc features.

Syntactic Sugar

Syntactic sugar that doesn't need additional IR operations typically only requires changes to mypyc.genops. Test cases are located in test-data/genops-*.test and the test driver is in mypyc.test.test_genops.

You may also need to add some definitions to the stubs used for builtins during tests (test-data/fixtures/ir.py). We don't use full typeshed stubs to run tests since they would seriously slow down tests.

Built-in Operation for an Already Supported Type

If you want to add support for a new primitive operation for a type that mypyc already supports in some fashion, you generally have to do at least these steps:

• Add a new operation to mypyc.ops. Often you only need to add a suboperation to PrimitiveOp or Branch instead of defining a new Op subclass. We don't have test cases specifically for operations.

• Generate the new operation in mypyc.genops. Also add test cases (see Syntactic Sugar for more information).

• Implement C generation for the new operation in mypyc.emitfunc. Test cases are located in mypyc.test.test_emitfunc. They are normal Python unit tests instead of data-driven test cases.

• Test that your new operation works by adding a test case to test-data/run.test and verifying that it passes. You don't always need to commit the new test. If your operation is pretty straightforward, you can omit a test in run.test and just add a note with your PR mentioning that you've verified that your change works end-to-end.

If your operation compiles into a lot of C code, you may also want to add a C helper function for the operation to make the generated code smaller. Here is how to do this:

• Add the operation to lib-rt/CPy.h. Usually defining a static function is the right thing to do, but feel free to also define inline functions for very simple and performance-critical operations. We avoid macros since they are error-prone.

• Add unit test for your C helper in lib-rt/test_capi.cc. We use Google Test for writing tests in C++. The framework is included in the repository under the directory googletest/. The C unit tests are run as part of the pytest test suite (test_c_unit_tests).

A New Primitive Type

Some types such as int and list are special cased in mypyc to generate operations specific to these types.

Here are some hints about how to add support for a new primitive type (this may be incomplete):

• Decide whether the primitive type has an "unboxed" representation (a representation that is not just PyObject *).

• Create a new instance of RPrimitive to support the primitive type. Make sure all the attributes are set correctly and also define <foo>_rprimitive and is_<foo>_rprimitive.

• Update mypyc.genops.Mapper.type_to_rtype().

• Update emit_box in mypyc.emit.

• Update emit_unbox or emit_cast in mypyc.emit.

• Update emit_inc_ref and emit_dec_ref in mypypc.emit if needed. If the unboxed representation does not need reference counting, these can be no-ops. If the representation is not unboxed these will already work.

• Update emit_error_check in mypyc.emit for unboxed types.

• Update emit_gc_visit and emit_gc_clear in mypyc.emit if the type has an unboxed representation with pointers.

The above may be enough to allow you to declare variables with the type and pass values around. You likely also want to add support for some primitive operations for the type (see Built-in Operation for an Already Supported Type for how to do this).

If you want to just test C generation, you can add a test case with dummy output to test-data/module-output.test and manually inspect the generated code. You probably don't want to commit a new test case there since these test cases are very fragile.

Add a test case to test-data/run.test to test compilation and running compiled code. Ideas for things to test:

• Test using the type for an argument.

• Test using the type for a return value.

• Test passing a value of the type to a function both within compiled code and from regular Python code. Also test this for return values.

• Test using the type as list item type. Test both getting a list item and setting a list item.