Passing slice as type parameters, and returning dynamic classes

[mwaddoups]

In the Cython project, I've been discussing an issue with how they implement their type annotations (cython/cython#6272). Cython does some absolute magic where they hook into the annotations for certain specific types written in Python code, which when passed to the cython builder re-interpret the python code as cython code.

There are two specific issues I've noticed with writing type hints for their code so far (I've not used their pure python side until recently).

Issue 1 - Cython memoryviews

One is with regards to Cython memoryviews. It is valid "pure Python" Cython code to write the following function

import cython

def f(x: cython.double, y_arr: cython.double[:]):
    y_arr[0] = x

The wanted annotations here are something like x: float, y_arr: Buffer.

But I am having trouble trying to write a type stub that allows you to use cython.double as an alias for one type, and cython.double[:] as an alias for another type.

1. Aliasing cython.double to a generic type gives "Expected type expression but received 'slice'" issues for cython.double.
2. Aliasing to typing.Any does stop type checkers throwing the error, but not sure if this is really expected behaviour or just lucky short-circuiting.
3. I can also get type-checkers to stop complaining if I set (in stubs) that cython.double = CythonDouble, where

class CythonDouble:
    def __class_getitem__(self, key):
        pass

x: CythonDouble
y: CythonDouble[:]

But in this case, both x and y have revealed type of CythonDouble, so trying to get different type semantics is hard. And it feels super hacky.

As I mentioned in the Cython issue, I'm not sure if SomeType[:] is ever intended type expression syntax, so they might be fighting a losing battle. But Cython has an old codebase and there is a lot of code that uses this, so if there's a way to make it work that's better than just using typing.Any , that would be great.

Issue 2 - dynamically created classes

Cython also supports the instantiation of C structs, and their use as return types. The following is valid Cython

SomeStruct = cython.struct(x=cython.double, y=cython.double)

def f() -> SomeStruct:
    return SomeStruct(3.0, 4.0)

But I have not been able to find a way to type cython.struct to allow this behaviour. Even returning typing.Any gives a "Variable not allowed in type expression" error in the function return value annotation.

Any ideas on how this could be achieved?

[bzoracler]

Slices as type parameters could be very neat for expressing multi-dimensional array types in the scientific Python stack, but I think it needs buy-in from e.g. the NumPy typing folks and a formalised proposal. I don't think this'll happen for a while, so I'd draft up one or both of the following as a proof-of-concept:

1. An alternative syntax for Cython memoryviews, maybe using a typing.Protocol copying the API of builtins.memoryview or whatever PEP 3118 is supposed to expose in the Python side, and a PR in Cython for recognition and code generation;
2. A mypy plugin to recognise things like cython.int[:, :, :]. I don't think mypy's currently documented plugin hooks expose a way to process this syntax - this'll require an AST transform at the earliest stages to something more in line with Python typing.

As for dynamically-created classes/types, a hook is exposed under the mypy plugin system for this (see get_dynamic_class_hook) which can work out-of-the-box (unlike what the mypy documentation implies, specifying the name on both the l-value and in the call argument X = dynamic_class("X", []) is not necessary).

---

A more serious issue would be that Cython performs a lot of type coersions for both C-Python-C and C++-Python-C++, so it is unclear whether a passing e.g. Python ints to a function parameter typed as cython.int should allow access to memoryview/buffer attributes inside the function body. mypyc special-cases their native integers differently from typeshed for a similar reason.

I guess the conclusion is, getting Cython to work robustly isn't possible in type-checkers without a plugin system, and since mypy seems to be the only widely-used type-checker with plugins, there's nothing to be done here for other type-checkers, which would just have to deal with typing.Anys.