Build on Windows with Clang and Flang

[isuruf]

This build blas and cblas with clang, and lapack with flang.

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[jakirkham]

Thanks for putting this together @isuruf!

Couple of thoughts. First, if we use VC 14, I guess that means we can't link Python 2.7 stuff with this. What would your proposal be in that case? Use m2w64-openblas? Second, if we go this route, I guess we can't support Fortran (yet). Any thoughts on how to handle things that want Fortran support? Admittedly this isn't an easy question. So maybe there is no good answer.

[isuruf]

What would your proposal be in that case? Use m2w64-openblas?

No idea

Second, if we go this route, I guess we can't support Fortran (yet). Any thoughts on how to handle things that want Fortran support?

Wait until Flang is ready or use CLAPACK.

[isuruf]

There's a PR for DYNAMIC_ARCH here

[scopatz]

Any movement on this? This would be really great to have. I am looking to windows support for COIN, which also needs a BLAS...

[isuruf]

@scopatz, do you need LAPACK or just blas?

[scopatz]

I think it needs LAPACK too

[isuruf]

Okay, then I need to fix flang-compiler/flang#288 then which should take anything between 1 week to months

[scopatz]

Ho-kay! Thanks for letting me know! And thanks for working on these issues!

[ghost]

@scopatz If you require an older LAPACK version (from 2009), you can just just the vcpkg BLAS. The performance is not good but it's something.

[powershell]
git clone https://github.com/microsoft/vcpkg.git
cd vcpkg
.\bootstrap.ps1
vcpkg install clapack:x64-windows

[isuruf]

conda-forge also has clapack

[ghost]

Is this ready to go (I assume using a patched cmake to build is okay for now, since it's only temporary), or will this be blocked due to the Python 2.7 issue?

[isuruf]

I'm going to package cmake tomorrow. (CMake 3.10 is coming out tomorrow and I'll have the flang windows fixes on top of that).

Afterwards I need to figure out what patches are needed. DYNAMIC_ARCH PR, Flang PR and maybe others.

[ghost]

If it were me, I'd just build OpenBLAS HEAD for windows only now, and then wait until the next version to release everything together. It's not ideal, but who knows what's in there that's enabling windows builds. It could be a mess (or it may not be).

[isuruf]

Btw, appveyor's 1 hour limit is not enough to build with both DYNAMIC_ARCH=1 and NOFORTRAN=0

[ghost]

Yes but conda gets two hours.

[ghost]

If you're trying to make it difficult to install Intel components when using flang, that's not being helpful. There is no reason to restrict this. Possibly it would be helpful to restrict having both MKL and openblas.

[msarahan]

Any exercise in restricting what gets installed with what is really not worthwhile until we've tested packages in mixed environments and run some test suites to see if anything breaks. I'm in agreement with @xoviat that we shouldn't arbitrarily restrict flang-compiled stuff to not coexist with intel-compiled stuff until we have evidence that it doesn't work.

[isuruf]

So, a flang feature would install flang compiled packages from conda-forge, but by default ifort compiled packages from defaults is preferred. Isn't that what we want? Am I missing something?

[isuruf]

Btw, the issue with vc is not applicable here, as we won't be adding flang-meta to the run requirements. (Adding vc to run requirements was the issue, but if it weren't added, then packages with the feature can't be installed without explicitly installing vc package. We are in luck here because that's exactly what we want. No flang compiled packages unless flang-meta is installed.)

[isuruf]

@msarahan, would you mind commenting on #34 (comment) ?

[msarahan]

It's Thanksgiving. I'll be back on Monday.

…

On Nov 23, 2017 12:20, "Isuru Fernando" ***@***.***> wrote: @msarahan <https://github.com/msarahan>, would you mind commenting on #34 (comment) <#34 (comment)> ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#34 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACV-TjrB1MU7k8LeKp7LlTJTu5D5Knsks5s5bdqgaJpZM4OncRe> .

[jakirkham]

Happy Thanksgiving Mike! 🦃

[msarahan]

So, a flang feature would install flang compiled packages from conda-forge, but by default ifort compiled packages from defaults is preferred. Isn't that what we want? Am I missing something?

The key is always having a self-consistent set of software that is compatible within itself. With the mess we've seen with features, I am wary of any new scheme to restrict sets of software until I have seen packages or test cases to prove theories about how it might work. As @xoviat pointed out above, it's not even completely clear that we need to restrict Flang-built software from ifort-built software. We should test that by trying to identify two fortran-built packages that we can mix in an environment, and run the test suites for both of those packages, hoping to catch any potential incompatibilities.

Scipy comes to mind, but I'm not aware of many (any?) other major fortran-using software. Can any of you suggest something? Perhaps Openblas compiled with flang, scipy compiled with ifort, and vice-versa, just for good measure? If you need our help making packages compiled with ifort to test, we'll be happy to assist.

[ghost]

Is there any way to merge this as a pre release so that we can test it?

[ghost]

Actually I compiled scipy with flang+mkl and it worked for the most part except for three failures caused by be flang bugs.

[isuruf]

As @xoviat said scipy compiled with flang+mkl almost works. I don't have access to ifort. @msarahan would you be able to test scipy compiled with ifort+openblas from this PR? (I think Intel is offering free licenses to open source contributors, I can check that out)

Btw, I've reverted the flang feature. Since this openblas is not interfering with defaults because there is no openblas for windows on defaults, any objections to merging ?

[isuruf]

@xoviat, what errors did you get? If you give me instructions I can try it out and see if there's anything I can do. (I've installed mkl using conda)

[ghost]

Just install my flang numpy branch:

pip install git+https://GitHub.com/xoviat/numpy.git@flang

then install scipy

pip install git+https//GitHub.com/scipy/scipy.git

Note that you must be in a conda environment with flang and mkl.

[ghost]

Compare that to the current Windows development procedure!

[mingwandroid]

You still need visual studio installed though here do you not?

[isuruf]

You still need visual studio installed though here do you not?

Yeah, but for development and packaging into conda, wheels, it's easier.

@conda-forge/openblas, any more comments on the PR?

[ghost]

Did you confirm that build procedure?

[ghost]

Actually it would be very nice if this was merged right now. I cannot upload the wheels that I built using MKL because it's too large (100MB!).

[ghost]

Trying to work around a missing conda OpenBLAS is just painful, considering that OpenBLAS takes over an hour to build.

[scopatz]

I think that this should go in if it can since it seems to be compatible with VS. I don't have merge rights, though, myself.

[ghost]

@isuruf has merge rights, but of course may be worried about objections from others. Maybe wait another day, and merge if no more comments?

[scopatz]

Sounds reasonable to me

[isuruf]

@xoviat, I ran the tests for scipy built with flang+mkl. 4 failed. 12768 passed. I'll report on the scipy issue tracker if I find anything.

[ghost]

The fortranfile issue is probably not that important, but there's precision loss that's happening somewhere; this causes an optimization routine to fail. It would be good if that could be fixed (though it's probably a bit hairy).

[ghost]

@isuruf Can you merge?

[ghost]

@isuruf Ping.