CI: Build on Windows #374

vt-alt · 2021-12-02T13:40:18Z

Правда куча ошибок при сборке. Гораздо больше чем тут #370
Может я как-то не так собрал openssl.

vt-alt · 2021-12-02T15:33:58Z

Да, надо, как минимум, no-deprecated убрать.

beldmit · 2021-12-02T15:39:18Z

Ну идея собирать ENGINE с no-deprecated - это оксюморон.

vt-alt · 2021-12-02T16:17:15Z

Теперь стали более релевантные ошибки. Но, осталось слишком много варнингов [deprecated] Since OpenSSL 3.0.

vt-alt · 2021-12-02T16:38:20Z

Наверное, /WX тоже стоит отключить? Или попытаться сделать что-то типа /wd4996.

vt-alt · 2021-12-02T16:52:22Z

Может, этот коммит отдельным PR, если тесты на Linux пройдут?

С фиксами для MSVC? Я надеюсь, что смогу это пофиксить. Тогда будет логично принять сразу оба коммита.

vt-alt · 2021-12-03T09:51:23Z

Чем больше фикшу — тем больше новых ошибок вылезает. Пока дособиралось до sign.c, то есть основная либа уже собралась и остались тесты.

vt-alt · 2021-12-04T09:54:28Z

Поправил сколько смог.

Отключил запуск ctest так как все тесты завершаются с Exit code 0xc0000135 (в интернетах пишут, что это STATUS_DLL_NOT_FOUND) и разбираться с этим через GA уж слишком запарно и мой лимит энтузиазма иссяк. Я и так излишне оптимистично думал, что будет проще в 10 раз.

Test project D:/a/gost-engine/gost-engine
      Start  1: digest-with-engine
 1/14 Test  #1: digest-with-engine ...............Exit code 0xc0000135
***Exception:   0.01 sec

Зато компиляция почти всего проходит успешно, кроме test_tls и benchmark/sign — пришлось добавить грубое отключение их сборки на Windows в CMakeLists.txt.

Теперь хотя бы можно тестировать корректность кода engine.

beldmit · 2021-12-04T10:55:59Z

Автотесты ругаются по делу

vt-alt · 2021-12-05T11:04:31Z

Не осиливаю как сделать запуск тестов. Установил правильно PATH, OPENSSL_ENGINES и OPENSSL_CONF, но тесты ведут себя как будто энжина не загрузилась. openssl.exe engine gost выводит такое сообщение:

C4070000:error:1280006A:DSO support routines:win32_bind_func:could not bind to the requested symbol name:crypto\dso\dso_win32.c:184:symname(bind_engine)
C4070000:error:1280006A:DSO support routines:DSO_bind_func:could not bind to the requested symbol name:crypto\dso\dso_lib.c:176:
C4070000:error:13000068:engine routines:dynamic_load:DSO failure:crypto\engine\eng_dyn.c:435:
C4070000:error:13000074:engine routines:ENGINE_by_id:no such engine:crypto\engine\eng_list.c:430:id=gost

ps. Если что - dso, dynamic-engine, engine, shared - включены.

beldmit · 2021-12-05T17:10:16Z

А функция объявлена как __dllexport? Я сто лет как под винду не пытался собирать DLL-ки.

vt-alt · 2021-12-06T03:04:51Z

Вроде я начал понимать. На виндах cmake не информативно пишет при сборке чего угодно Building Custom Rule D:/a/gost-engine/gost-engine/libprov/CMakeLists.txt, но если посмотреть что создается после этого то видно, что сначала собирается

gost_engine.vcxproj -> D:\a\gost-engine\gost-engine\bin\Debug\gost.dll

а потом

lib_gost_engine.vcxproj -> D:\a\gost-engine\gost-engine\bin\Debug\gost.dll

Думаю, новая сборка engine в виде библиотеки (из коммита d9a2b29) перезаписывает поверх старую сборку gost.dll в виде dlopen объекта. В "библиотеке" уже нет символа bind_engine, там другой механизм загрузки.
На unix они собираются в разные файлы gost.so и libgost.so, надо сделать или чтоб и на виндах они собирались во что-то разное, или не знаю что.

ps. Не собирать "библиотеку", например. Наверное так и делаю, так как на виндах это не актуально.
pps. Да, это помогло. Теперь тесты запускаются вручную, но не через ctest.
ppps. Запускаются. Но теперь gostprov не грузится.

vt-alt · 2021-12-06T06:52:29Z

Пока итог такой - gostprov.dll не работает как надо на windows - openssl.exe list -providers не показывает gostprov.

> $env:PATH = "$env:PATH;$pwd\_dest\Program Files\OpenSSL\bin"
> $env:OPENSSL_MODULES = "$pwd\bin\Debug"
> $env:OPENSSL_CONF = "$pwd\test\provider.cnf"
> & '.\_dest\Program Files\OpenSSL\bin\openssl.exe' list -providers -verbose
Providers:
  default
    name: OpenSSL Default Provider
    version: 3.1.0
    status: active
    build info: 3.1.0-dev
    gettable provider parameters:
      name: pointer to a UTF8 encoded string (arbitrary size)
      version: pointer to a UTF8 encoded string (arbitrary size)
      buildinfo: pointer to a UTF8 encoded string (arbitrary size)
      status: integer (arbitrary size)

Наверное, я отключу сборку gostprov для windows, а после просветления можно будет включить.

ps.

> & '.\_dest\Program Files\OpenSSL\bin\openssl.exe' list -provider gostprov
list: unable to load provider gostprov
Hint: use -provider-path option or OPENSSL_MODULES environment variable.
E0150000:error:1280006A:DSO support routines:win32_bind_func:could not bind to the requested symbol name:crypto\dso\dso_win32.c:184:symname(OSSL_provider_init)
E0150000:error:1280006A:DSO support routines:DSO_bind_func:could not bind to the requested symbol name:crypto\dso\dso_lib.c:176:
E0150000:error:078C0105:common libcrypto routines:provider_init:init fail:crypto\provider_core.c:906:name=gostprov

Может тоже не хватило __dllexport?

vt-alt · 2021-12-06T07:20:21Z

Может тоже не хватило __dllexport?

Видимо да. dumpbin /export gostprov.dll показывает, что символов нет. Просто так __dllexport к OSSL_provider_init не добавить, так как эта функция определена глобально в core.h без __dllexport. А если добавить в одном месте, только в gost_prov.c, то ошибка:

D:\a\gost-engine\gost-engine\gost_prov.c(147,5): error C2375: 'OSSL_provider_init': redefinition; different linkage [D:\a\gost-engine\gost-engine\gost_prov.vcxproj]

Не понимаю как тогда у них прошли тесты на providers на виндах. Это не должно работать.

levitte · 2021-12-06T11:02:48Z

I understood from openssl/openssl#17203 what's going on, or at least that part. I wonder if Cmake has some sort of mechanism for doing symbol exports in a platform agnostic way

beldmit · 2021-12-06T11:03:57Z

@levitte, yes. We consider the problem is lack of __dllexport OSSL_provider_init. Can it be so?

vt-alt · 2021-12-06T11:23:46Z

Btw, When I added OPENSSL_EXPORT to OSSL_provider_init as experiment, it's loaded successfully. So, it seems dllexport related.

levitte · 2021-12-06T11:44:48Z

It is dllexport related indeed.
I have found the Cmake solution: https://cmake.org/cmake/help/latest/module/GenerateExportHeader.html. It's been around since Cmake 3.0, but support for C projects came in 3.12. Considering gost-engine current requires 3.18 at a minimum, no update of the required Cmake version is needed.

levitte · 2021-12-06T12:14:20Z

I'm giving that Cmake mechanism a go

vt-alt · 2021-12-10T13:57:58Z

GA cache issues are badly debuggable (for example it does not show from which cache restore is performed when prefixed keys are used). I think I will change cache logic a bit so such issues are more understandable (without use of prefixed restore keys).

Uses actions/cache to speed up engine rebuilds. Signed-off-by: Vitaly Chikunov <[email protected]>

`/wd4996' disables deprecated declarations warning, this is analogous to `-Wno-error=deprecated-declarations'. Error example: gost_pmeth.c(39,17): warning C4996: 'EVP_PKEY_get0': Since OpenSSL 3.0 Signed-off-by: Vitaly Chikunov <[email protected]>

Since they are now global. Error message: cl : command line error D8021: invalid numeric argument '/Wno-error=deprecated-declarations' Fixes: 5dfb598 ("CMakeLists.txt: Workaround openssl deprecation of HMAC and CMAC primitives") Signed-off-by: Vitaly Chikunov <[email protected]>

Error message: gost_crypt.c(652,36): warning C4018: '<': signed/unsigned mismatch Signed-off-by: Vitaly Chikunov <[email protected]>

Errors are like this: gost_pmeth.c(188): warning C4702: unreachable code gost_grasshopper_cipher.c(909): warning C4702: unreachable code Signed-off-by: Vitaly Chikunov <[email protected]>

Error example: gost_md.c(54,45): error C2059: syntax error: ':' gost_md.c(67,5): error C2059: syntax error: 'if' gost_md.c(69,6): error C2143: syntax error: missing '{' before '->' gost_md.c(69,6): error C2059: syntax error: '->' gost_md.c(70,5): error C2059: syntax error: 'return' gost_md.c(71,1): error C2059: syntax error: '}' Signed-off-by: Vitaly Chikunov <[email protected]>

Un-inline `cipher_gost_grasshopper_ctracpkm'. Error message: gost_core.lib(gost_omac_acpkm.obj) : error LNK2019: unresolved external symbol cipher_gost_grasshopper_ctracpkm referenced in function CMAC_ACPKM_Init gost.dll : fatal error LNK1120: 1 unresolved externals Signed-off-by: Vitaly Chikunov <[email protected]>

Error message example: gost_prov_cipher.c(237,63): error C2059: syntax error: '}' Signed-off-by: Vitaly Chikunov <[email protected]>

These planned to be removed later anyway. Error message: gosthash2012_ref.h(15,18): warning C4081: expected '('; found 'string' Signed-off-by: Vitaly Chikunov <[email protected]>

vt-alt · 2021-12-10T14:59:26Z

Now that GA cache issue is (hopefully) solved it should build correctly (without workarounds). At least it does on my branch. Extra thanks to @levitte for OpenSSL fixes! 🙏

beldmit · 2021-12-11T12:28:05Z

@vt-alt Do you currently consider it mergeable?

No need external library (Ws2_32.lib) if we can easily implement it. Error messages: gost_core.lib(gost_keyexpimp.obj) : error LNK2019: unresolved external symbol htonl referenced in function gost_kdftree2012_256 gost.dll : fatal error LNK1120: 1 unresolved externals Signed-off-by: Vitaly Chikunov <[email protected]>

Error message: test_ciphers.c(37,9): warning C4068: unknown pragma 'GCC' Signed-off-by: Vitaly Chikunov <[email protected]>

MSVC is not C99 compatible[1]. Errors: test_ciphers.c(312,25): error C2057: expected constant expression test_ciphers.c(312,25): error C2466: cannot allocate an array of constant size 0 test_ciphers.c(312,26): error C2133: 'c': unknown size Link: https://docs.microsoft.com/en-us/cpp/c-language/ansi-conformance Signed-off-by: Vitaly Chikunov <[email protected]>

Error messages: test_ciphers.c(329,5): error C2059: syntax error: '{' test_ciphers.c(329,5): error C2059: syntax error: '}' test_ciphers.c(329,5): error C2059: syntax error: ')' Signed-off-by: Vitaly Chikunov <[email protected]>

test_digest.c(513,2): warning C4389: '==': signed/unsigned mismatch test_digest.c(820,5): warning C4389: '==': signed/unsigned mismatch test_sign.c(241,22): warning C4389: '==': signed/unsigned mismatch test_params.c(1131,16): warning C4018: '<': signed/unsigned mismatch test_sign.c(241,22): warning C4389: '==': signed/unsigned mismatch Signed-off-by: Vitaly Chikunov <[email protected]>

Error message: gost12sum.c(13,10): fatal error C1083: Cannot open include file: 'unistd.h': No such file or directory gost12sum.c(80,23): warning C4013: 'getopt' undefined; assuming extern returning int test_keyexpimp.c(7,10): fatal error C1083: Cannot open include file: 'arpa/inet.h': No such file or directory Signed-off-by: Vitaly Chikunov <[email protected]>

Error message: test_keyexpimp.c(111,11): warning C4013: 'setenv' undefined; assuming extern returning int Signed-off-by: Vitaly Chikunov <[email protected]>

Signed-off-by: Vitaly Chikunov <[email protected]>

Error message: OPENSSL_Uplink(00007FF9FB00F600,08): no OPENSSL_Applink Link: https://www.openssl.org/docs/faq.html#PROG3 Signed-off-by: Vitaly Chikunov <[email protected]>

Library form overwrites module form due to both having the same name `gost.dll'. As temporary workaround do not build library form on Windows, until we invent how to solve it properly. Currently, there is no known need of engine in the library form on Windows. Signed-off-by: Vitaly Chikunov <[email protected]>

This will skip tests on Windows, due to prove being `.bat' file and cmake will be unable to run it. Also, `HAVE_TEST2_V0' is renamed to be more meaningful. Signed-off-by: Vitaly Chikunov <[email protected]>

This will fix setting of OPENSSL_ENGINES in ctest on Windows. Signed-off-by: Vitaly Chikunov <[email protected]>

`OSSL_provider_init' requires dllexport attribute to be visible (and thus loadable) in DLL. Link: openssl/openssl#17203 Signed-off-by: Vitaly Chikunov <[email protected]>

vt-alt · 2021-12-11T12:42:32Z

I did small last minute change (added static to be32, and replaced SKIP_LINUX_ONLY with MSVC).
It if compiles green then it's mergeable.

beldmit · 2021-12-11T16:45:28Z

Merged. Many thanks!

vt-alt force-pushed the windows branch from e35e62b to 6fff337 Compare December 2, 2021 13:41

vt-alt force-pushed the windows branch from 6fff337 to b6cd0f1 Compare December 2, 2021 16:01

vt-alt force-pushed the windows branch 2 times, most recently from 35161d6 to 231f659 Compare December 2, 2021 16:50

vt-alt force-pushed the windows branch 4 times, most recently from c507043 to 39c6ac8 Compare December 3, 2021 09:43

vt-alt force-pushed the windows branch from 39c6ac8 to 4a612f9 Compare December 4, 2021 09:45

vt-alt force-pushed the windows branch from 4a612f9 to 58efea9 Compare December 4, 2021 09:57

vt-alt force-pushed the windows branch from 58efea9 to e5afcb5 Compare December 5, 2021 00:11

vt-alt mentioned this pull request Dec 6, 2021

External provider does not load on Windows, because OSSL_provider_init is not dllexport'ed openssl/openssl#17203

Closed

vt-alt added 9 commits December 10, 2021 17:50

CI: Build and test on Windows (MSVC)

bd8cc41

Uses actions/cache to speed up engine rebuilds. Signed-off-by: Vitaly Chikunov <[email protected]>

MSVC: Fix signedness in comparison

4bea3b4

Error message: gost_crypt.c(652,36): warning C4018: '<': signed/unsigned mismatch Signed-off-by: Vitaly Chikunov <[email protected]>

MSVC: Fix unreachable code warning

4348f7b

Errors are like this: gost_pmeth.c(188): warning C4702: unreachable code gost_grasshopper_cipher.c(909): warning C4702: unreachable code Signed-off-by: Vitaly Chikunov <[email protected]>

MSVC: Replace tentative arrays with pointers

6fee6ae

Error message example: gost_prov_cipher.c(237,63): error C2059: syntax error: '}' Signed-off-by: Vitaly Chikunov <[email protected]>

MSVC: Remove GCC pragma messages

352b641

These planned to be removed later anyway. Error message: gosthash2012_ref.h(15,18): warning C4081: expected '('; found 'string' Signed-off-by: Vitaly Chikunov <[email protected]>

vt-alt force-pushed the windows branch from 9a41a81 to 0ec25a2 Compare December 10, 2021 14:56

vt-alt added 13 commits December 11, 2021 15:39

MSVC: Ifdef GCC pragmas

4a66f06

Error message: test_ciphers.c(37,9): warning C4068: unknown pragma 'GCC' Signed-off-by: Vitaly Chikunov <[email protected]>

MSVC: No support for GCC's compound statement expressions

7195a0d

Error messages: test_ciphers.c(329,5): error C2059: syntax error: '{' test_ciphers.c(329,5): error C2059: syntax error: '}' test_ciphers.c(329,5): error C2059: syntax error: ')' Signed-off-by: Vitaly Chikunov <[email protected]>

MSVC: Fix absence of 'setenv'

57b996b

Error message: test_keyexpimp.c(111,11): warning C4013: 'setenv' undefined; assuming extern returning int Signed-off-by: Vitaly Chikunov <[email protected]>

MSVC: Skip building Linux-only binaries

f599b8f

Signed-off-by: Vitaly Chikunov <[email protected]>

MSVC: Fix 'no OPENSSL_Applink' error

24a48b0

Error message: OPENSSL_Uplink(00007FF9FB00F600,08): no OPENSSL_Applink Link: https://www.openssl.org/docs/faq.html#PROG3 Signed-off-by: Vitaly Chikunov <[email protected]>

CMake: Test presence of prove for perl tests

22a2775

This will skip tests on Windows, due to prove being `.bat' file and cmake will be unable to run it. Also, `HAVE_TEST2_V0' is renamed to be more meaningful. Signed-off-by: Vitaly Chikunov <[email protected]>

CMake: Add CONFIG to the output dir if MSVC

c156592

This will fix setting of OPENSSL_ENGINES in ctest on Windows. Signed-off-by: Vitaly Chikunov <[email protected]>

MSVC: Add dllexport to OSSL_provider_init

9d2cfe3

`OSSL_provider_init' requires dllexport attribute to be visible (and thus loadable) in DLL. Link: openssl/openssl#17203 Signed-off-by: Vitaly Chikunov <[email protected]>

vt-alt force-pushed the windows branch from 0ec25a2 to 9d2cfe3 Compare December 11, 2021 12:40

beldmit merged commit 1c34b34 into gost-engine:master Dec 11, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI: Build on Windows #374

CI: Build on Windows #374

vt-alt commented Dec 2, 2021

vt-alt commented Dec 2, 2021

beldmit commented Dec 2, 2021

vt-alt commented Dec 2, 2021

vt-alt commented Dec 2, 2021 •

edited

Loading

vt-alt commented Dec 2, 2021 •

edited

Loading

vt-alt commented Dec 3, 2021

vt-alt commented Dec 4, 2021

beldmit commented Dec 4, 2021

vt-alt commented Dec 5, 2021 •

edited

Loading

beldmit commented Dec 5, 2021

vt-alt commented Dec 6, 2021 •

edited

Loading

vt-alt commented Dec 6, 2021 •

edited

Loading

vt-alt commented Dec 6, 2021

levitte commented Dec 6, 2021

beldmit commented Dec 6, 2021

vt-alt commented Dec 6, 2021

levitte commented Dec 6, 2021

levitte commented Dec 6, 2021

vt-alt commented Dec 10, 2021

vt-alt commented Dec 10, 2021

beldmit commented Dec 11, 2021

vt-alt commented Dec 11, 2021

beldmit commented Dec 11, 2021

CI: Build on Windows #374

CI: Build on Windows #374

Conversation

vt-alt commented Dec 2, 2021

vt-alt commented Dec 2, 2021

beldmit commented Dec 2, 2021

vt-alt commented Dec 2, 2021

vt-alt commented Dec 2, 2021 • edited Loading

vt-alt commented Dec 2, 2021 • edited Loading

vt-alt commented Dec 3, 2021

vt-alt commented Dec 4, 2021

beldmit commented Dec 4, 2021

vt-alt commented Dec 5, 2021 • edited Loading

beldmit commented Dec 5, 2021

vt-alt commented Dec 6, 2021 • edited Loading

vt-alt commented Dec 6, 2021 • edited Loading

vt-alt commented Dec 6, 2021

levitte commented Dec 6, 2021

beldmit commented Dec 6, 2021

vt-alt commented Dec 6, 2021

levitte commented Dec 6, 2021

levitte commented Dec 6, 2021

vt-alt commented Dec 10, 2021

vt-alt commented Dec 10, 2021

beldmit commented Dec 11, 2021

vt-alt commented Dec 11, 2021

beldmit commented Dec 11, 2021

vt-alt commented Dec 2, 2021 •

edited

Loading

vt-alt commented Dec 2, 2021 •

edited

Loading

vt-alt commented Dec 5, 2021 •

edited

Loading

vt-alt commented Dec 6, 2021 •

edited

Loading

vt-alt commented Dec 6, 2021 •

edited

Loading