install fails under PyPy

[belm0]

see also crossbario/autobahn-python#968 (comment), python-trio/trio-websocket#119

  Running setup.py install for wsaccel ... error
    Complete output from command /usr/local/Cellar/pypy3/7.0.0/bin/pypy3 -u -c "import setuptools, tokenize;__file__='/private/var/folders/ww/51xbs7nx6t16x7mh225q9x1h0000gn/T/pip-install-dc6tdoqk/wsaccel/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/var/folders/ww/51xbs7nx6t16x7mh225q9x1h0000gn/T/pip-record-8umwani7/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    creating build
    creating build/lib.macosx-10.13-x86_64-3.6
    creating build/lib.macosx-10.13-x86_64-3.6/wsaccel
    copying wsaccel/__init__.py -> build/lib.macosx-10.13-x86_64-3.6/wsaccel
    running build_ext
    building 'wsaccel.utf8validator' extension
    creating build/temp.macosx-10.13-x86_64-3.6
    creating build/temp.macosx-10.13-x86_64-3.6/wsaccel
    gcc -pthread -arch x86_64 -DNDEBUG -O2 -fPIC -I/usr/local/Cellar/pypy3/7.0.0/libexec/include -c wsaccel/utf8validator.c -o build/temp.macosx-10.13-x86_64-3.6/wsaccel/utf8validator.o
    wsaccel/utf8validator.c:235:11: warning: 'PyMethod_New' macro redefined [-Wmacro-redefined]
      #define PyMethod_New(func, self, klass) ((self) ? PyMethod_New(func, self) : PyInstanceMethod_New(func))
              ^
    /usr/local/Cellar/pypy3/7.0.0/libexec/include/pypy_decl.h:487:9: note: previous definition is here
    #define PyMethod_New PyPyMethod_New
            ^
    wsaccel/utf8validator.c:983:49: warning: passing 'void **' to parameter of type 'const void **' discards qualifiers in nested pointer types [-Wincompatible-pointer-types-discards-qualifiers]
      __pyx_t_1 = PyObject_AsReadBuffer(__pyx_v_ba, ((void **)(&__pyx_v_buf)), (&__pyx_v_buf_len)); if (unlikely(__pyx_t_1 == -1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 104; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
                                                    ^~~~~~~~~~~~~~~~~~~~~~~~~
    /usr/local/Cellar/pypy3/7.0.0/libexec/include/object.h:333:64: note: passing argument to parameter here
    PyAPI_FUNC(int) PyObject_AsReadBuffer(PyObject *, const void **, Py_ssize_t *);
                                                                   ^
    2 warnings generated.
    gcc -pthread -shared -undefined dynamic_lookup build/temp.macosx-10.13-x86_64-3.6/wsaccel/utf8validator.o -o build/lib.macosx-10.13-x86_64-3.6/wsaccel/utf8validator.pypy3-70-darwin.so
    clang: warning: argument unused during compilation: '-pthread' [-Wunused-command-line-argument]
    building 'wsaccel.xormask' extension
    gcc -pthread -arch x86_64 -DNDEBUG -O2 -fPIC -I/usr/local/Cellar/pypy3/7.0.0/libexec/include -c wsaccel/xormask.c -o build/temp.macosx-10.13-x86_64-3.6/wsaccel/xormask.o
    wsaccel/xormask.c:235:11: warning: 'PyMethod_New' macro redefined [-Wmacro-redefined]
      #define PyMethod_New(func, self, klass) ((self) ? PyMethod_New(func, self) : PyInstanceMethod_New(func))
              ^
    /usr/local/Cellar/pypy3/7.0.0/libexec/include/pypy_decl.h:487:9: note: previous definition is here
    #define PyMethod_New PyPyMethod_New
            ^
    wsaccel/xormask.c:2165:36: error: no member named 'curexc_traceback' in 'struct _ts'
            PyObject* tmp_tb = tstate->curexc_traceback;
                               ~~~~~~  ^
    wsaccel/xormask.c:2168:21: error: no member named 'curexc_traceback' in 'struct _ts'
                tstate->curexc_traceback = tb;
                ~~~~~~  ^
    1 warning and 2 errors generated.
    error: command 'gcc' failed with exit status 1

    ----------------------------------------
Command "/usr/local/Cellar/pypy3/7.0.0/bin/pypy3 -u -c "import setuptools, tokenize;__file__='/private/var/folders/ww/51xbs7nx6t16x7mh225q9x1h0000gn/T/pip-install-dc6tdoqk/wsaccel/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/var/folders/ww/51xbs7nx6t16x7mh225q9x1h0000gn/T/pip-record-8umwani7/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/var/folders/ww/51xbs7nx6t16x7mh225q9x1h0000gn/T/pip-install-dc6tdoqk/wsaccel/

[methane]

I think PyPy is faster enough without wsaccel.
If you really want to use wsaccel with PyPy, would you try building C source with new Cython?

[belm0]

It's not necessarily something I have control over-- wsaccel is a transitive dependency of my build. Any package with a wsaccel dependency which doesn't have a pypy exclusion will break projects which try to use the package under PyPy.

[belm0]

This is an aside, but have you considered getting wsaccel out of the C extension business? There is a pure Python implementation which outperforms it in this blog post: https://www.willmcgugan.com/blog/tech/post/speeding-up-websockets-60x/

Perhaps wsaccel could change its implementation to that, and all current dependees would magically get a lighter weight dependency.

From my own measurement the implementation using bytearray.translate() (python2 compatible) is about 10% faster than wsaccel on 10K of data:

https://nbviewer.jupyter.org/gist/belm0/2b610cc405dd3dac977f34650cb187cc

[openmark2]

Try installing wsaccel directly from source files:
pip install git+https://github.com/methane/wsaccel.git
This assumes you have Cython installed and available in your PyPy environment:
pip install Cython
From my understanding, the package downloaded from pip comes with .c files which fail to be compiled by gcc, the source files, in contrast, contain .pyx files which are converted to .c files on-the-go by your local Cython installation and then compiled. I had a similar error and installing from sources solved it.

[methane]

From my own measurement the implementation using bytearray.translate() (python2 compatible) is about 10% faster than wsaccel on 10K of data:

You are comparing orange and egg. This project implements XorShifter, which is drop-in-replacement of Autobahn's implementation. It is incremental masker, not one-shot masker.

---

I created this because of very poor performance of Tornado and Autobahn, and ws4py.
Tornado has own speedup module already.
Autobahn will have own speedup module too.
ws4py has been stopped.

I don't have motivation to improve this project. Let's rest this project in pease.