Is it possible to do cross-compilation to build windows wheels?

[Congee]

According to jq's wiki, https://github.com/stedolan/jq/wiki/Cross-compilation , you may be able to build wheels for Windows from Linux. I would like to help but have no idea about how you are currently building wheels? Should I make a PR to change the setup.py or write a CI (continuous integration, github actions would be good) script?

[mwilliamson]

Wheels are currently built using Travis: https://github.com/mwilliamson/jq.py/blob/master/.travis.yml

[Congee]

I ma note down what I tried for future reference.

[Congee]

I am able to cross-compile jq and oniguruma on Ubuntu.

[Congee]

I am able to build oniguruma on windows with mingw on both my windows box and travis' windows ci image. No luck with jq on windows + mingw. Maybe msys2 makes it possible, but I wouldn't try it.

[Congee]

I'm trying to do it in two stages in travis. First by cross-compiling jq + oniguruma on Linux, then build with cython on windows.

Persisting artifacts between stages seems tricky. https://docs.travis-ci.com/user/build-stages/#data-persistence-between-stages-and-jobs
Is it possible to cross-compile with cython from linux from windows as well?

[Congee]

Obviously MSVC is not happy about GNU stuff

#ifdef __GNUC__
#define JV_PRINTF_LIKE(fmt_arg_num, args_num) \
  __attribute__ ((__format__( __printf__, fmt_arg_num, args_num)))
#define JV_VPRINTF_LIKE(fmt_arg_num) \
  __attribute__ ((__format__( __printf__, fmt_arg_num, 0)))
#endif

/* ... */

jv jv_string_vfmt(const char*, va_list) JV_VPRINTF_LIKE(1);
jv jv_string_fmt(const char*, ...) JV_PRINTF_LIKE(1, 2);

I'm stuck here :(

[Congee]

Redefining those macros seems getting me around. Seeing
libjq.a(execute.o) : error LNK2001: unresolved external symbol printf

The cross-compilation of jq must have been failed!

[Congee]

A little investigation shows me both the official jq.exe and my libjq.a have common symbols like pthread_*, printf. I'm not blocked by failed cross-compilation but linking to the c runtime on windows.

[Congee]

Got a jq.cp38-win_amd64.pyd with PyInit_jq but cannot import it

ImportError: DLL load failed while importing jq: The specified module could not be found.

Also, I have to add crap here

Not elegant, not elegant.

[Congee]

Why does mingw produces a library that requires its runtime rather than mere msvcrt.dll? This does not seem to be cross-compilation to me.

[Congee]

Anyway, I found prebuilt libjq.a and libonig.a from a community. A good news

https://packages.msys2.org/package/mingw-w64-x86_64-jq?repo=mingw64
https://packages.msys2.org/package/mingw-w64-x86_64-oniguruma?repo=mingw64

They resemble what I built. Making use of them instead of build it myself could save time

[Congee]

The pyd could not be loaded because two of its dependencies libgcc_s_seh-1.dll and libwinpthread-1.dll could not be loaded. The finding was thanks to Process Monitor.

The dumpbin tool did not tell me a dependency of pthread for jq.cp38-win_amd64.pyd, because it's indirect. In fact, libgcc_s_seh-1.dll depends on libwinpthread-1.dll.

C:\Users\cwu\AppData\Local\Temp\jq.py\build\lib.win-amd64-3.8>dumpbin /dependents jq.cp38-win_amd64.pyd
Microsoft (R) COFF/PE Dumper Version 14.25.28612.0
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file jq.cp38-win_amd64.pyd

File Type: DLL

  Image has the following dependencies:

    msvcrt.dll
    python38.dll
    KERNEL32.dll
    VCRUNTIME140.dll
    api-ms-win-crt-runtime-l1-1-0.dll
    api-ms-win-crt-stdio-l1-1-0.dll
    api-ms-win-crt-heap-l1-1-0.dll
    api-ms-win-crt-string-l1-1-0.dll
    api-ms-win-crt-utility-l1-1-0.dll
    api-ms-win-crt-time-l1-1-0.dll
    api-ms-win-crt-math-l1-1-0.dll
    api-ms-win-crt-environment-l1-1-0.dll
    api-ms-win-crt-filesystem-l1-1-0.dll
    libgcc_s_seh-1.dll
    SHLWAPI.dll

C:\Users\cwu\AppData\Local\Temp\jq.py\build\lib.win-amd64-3.8>dumpbin /dependents libgcc_s_seh-1.dll
Microsoft (R) COFF/PE Dumper Version 14.25.28612.0
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file libgcc_s_seh-1.dll

File Type: DLL

  Image has the following dependencies:

    KERNEL32.dll
    msvcrt.dll
    libwinpthread-1.dll

Of course, the first thing I suspected was the PATH that python searches for dlls. I did add the path of those two wanted dlls, but with no luck? Weird, why?

[Congee]

Anyway, it's promising that we can use jq.py on Windows. @mwilliamson What am I supposed to do to get this building process consolidated into the github repo?

[Jiehong]

Wouldn't it be easier to just built everything on a windows-ci travis host, and have the wheels fully generated? This would avoid this need for cross compilation, right?

[Congee]

Travis CI is no longer the go-to for open source projects due to its pricing changes. And, their Windows support is really inferior to Circle CI. If you need to grab a wheel, see https://github.com/Congee/jq.py/releases

[mwilliamson]

Closing in favour of #20.