Allow and document building on windows

[dgrunwald]

Even with the information in #17, building cvise on windows was quite difficult for me as I am unfamiliar with cmake.

This pull request documents the steps I used and fixes some build errors.

[dgrunwald]

Note that I haven't had much success using the resulting cvise.

I tried with the interestingness test:

cl.exe /c test.cpp || exit 1

and the simple test.cpp:

int dummy() {
     return 1; 
}
int main() {
     return dummy(); 
}

Result:

C:\\temp\\cvise-test>python C:\tools\cvise\bin\cvise check.bat test.cpp -n 1 --debug
00:00:00 ERROR Prereqs not found for pass IfPass
00:00:00 DEBUG perform sanity check...
00:00:00 DEBUG sanity check tmpdir = C:\Users\...\AppData\Local\Temp\cvise-sanity-o_4urtoz
00:00:00 DEBUG sanity check successful
00:00:00 INFO ===< 19012 >===
00:00:00 INFO running 1 interestingness test in parallel
00:00:00 INFO INITIAL PASSES
00:00:00 DEBUG Creating pass root folder: C:\Users\dg\AppData\Local\Temp\cvise-IncludesPass-aa7yodn_
00:00:00 INFO ===< IncludesPass >===
00:00:01 DEBUG Creating pass root folder: C:\Users\dg\AppData\Local\Temp\cvise-CommentsPass-30ebektj
00:00:01 INFO ===< CommentsPass >===
00:00:01 ERROR Skipping IfPass
00:00:01 DEBUG Creating pass root folder: C:\Users\dg\AppData\Local\Temp\cvise-LineMarkersPass-xx98ovnb
00:00:01 INFO ===< LineMarkersPass >===
00:00:01 DEBUG Creating pass root folder: C:\Users\dg\AppData\Local\Temp\cvise-BlankPass-6u3zmcat
00:00:01 INFO ===< BlankPass >===
00:00:02 DEBUG Creating pass root folder: C:\Users\dg\AppData\Local\Temp\cvise-ClangBinarySearchPass-replace-function-def-with-decl-2xg5mlui
00:00:02 INFO ===< ClangBinarySearchPass::replace-function-def-with-decl >===
00:00:02 DEBUG available transformation opportunities for c++98: 2, took: 0.02 s
00:00:02 DEBUG available transformation opportunities for c++11: 2, took: 0.02 s
00:00:02 DEBUG available transformation opportunities for c++14: 2, took: 0.00 s
00:00:02 DEBUG available transformation opportunities for c++17: 2, took: 0.02 s
00:00:02 DEBUG available transformation opportunities for c++20: 2, took: 0.02 s
00:00:02 INFO using C++ standard: c++98 with 2 transformation opportunities
00:00:02 DEBUG granularity reduced to 1
00:00:02 DEBUG ***ADVANCE*** from 0 to 1 with chunk 1
00:00:02 DEBUG Creating pass root folder: C:\Users\dg\AppData\Local\Temp\cvise-ClangBinarySearchPass-remove-unused-function-yo_8s8_s
00:00:02 INFO ===< ClangBinarySearchPass::remove-unused-function >===
00:00:02 DEBUG available transformation opportunities for c++98: 0, took: 0.02 s
00:00:02 DEBUG available transformation opportunities for c++11: 0, took: 0.00 s
00:00:02 DEBUG available transformation opportunities for c++14: 0, took: 0.02 s
00:00:02 DEBUG available transformation opportunities for c++17: 0, took: 0.02 s
00:00:02 DEBUG available transformation opportunities for c++20: 0, took: 0.00 s
00:00:02 INFO using C++ standard: c++98 with 0 transformation opportunities
00:00:02 DEBUG Creating pass root folder: C:\Users\dg\AppData\Local\Temp\cvise-LinesPass-0-zp8py5q6
00:00:02 INFO ===< LinesPass::0 >===
00:00:02 DEBUG perform sanity check...
00:00:02 DEBUG sanity check tmpdir = C:\Users\dg\AppData\Local\Temp\cvise-sanity-1kon7djj
00:00:03 DEBUG sanity check successful
Traceback (most recent call last):
  File "C:\tools\cvise\bin\cvise", line 305, in <module>
    reducer.reduce(pass_group, skip_initial=args.skip_initial_passes)
  File "C:\Tools/cvise/share\cvise\cvise.py", line 143, in reduce
    self._run_additional_passes(pass_group['first'])
  File "C:\Tools/cvise/share\cvise\cvise.py", line 166, in _run_additional_passes
    self.test_manager.run_pass(p)
  File "C:\Tools/cvise/share\cvise\utils\testing.py", line 506, in run_pass
    self.state = self.current_pass.new(self.current_test_case, self.check_sanity)
  File "C:\Tools/cvise/share\cvise\passes\lines.py", line 54, in new
    self.__format(test_case, check_sanity)
  File "C:\Tools/cvise/share\cvise\passes\lines.py", line 36, in __format
    os.unlink(backup.name)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\temp\\cvise-test\\tmph4y1034u'

I would have expected replace-function-def-with-decl to succeed in a reduction, but somehow that didn't work.
Also there is a crash when another program like windows defender blocks the deletion of the directory.
Unfortunately I've spent enough time fighting/learning cmake that I've run out of time and won't continue from here in the near future.

[marxin]

Thank you very much for the attempt and I'm willing to merge your pull request!

[marxin]

I would have expected replace-function-def-with-decl to succeed in a reduction, but somehow that didn't work.

You can invoke that manually with: ./clang_delta x.c --transformation=replace-function-def-with-decl --counter=1 --to-counter=2 and see the output.

[marxin]

Also there is a crash when another program like windows defender blocks the deletion of the directory.

Hm, are you sure it's really windows defender?

[dgrunwald]

Hm, are you sure it's really windows defender?

No I'm not sure, that was just my first guess. There's a lot of background stuff in Windows that might be holding on to directory handles.
I've seen errors like this occur (with other tools) on machines where most other services (search indexer etc.) were disabled.

I've only seen ERROR_ACCESS_DENIED occur with files when another process has the file open. Not sure what that could have been.
Deleting empty directories tends to non-deterministically trigger ERROR_DIR_NOT_EMPTY (whenever there's another process trying to concurrently list the directory) -- this one is fundamentally unavoidable on Windows and requires a retry-loop.

[marxin]

Well, what C-Vise does during reduction: it spawns multiple processes with interestingness test and waits for the first one to finish. Then other processes are canceled and the top-level working directory is removed. That means there can be still running processes that access the files. Such an approach works fine on Linux but may cause troubles on Windows. (Un)fortunately, I don't use Windows system, so can't debug that ..

[dgrunwald]

The problems with file deletion itself being asynchronous in Windows can be worked around in a number of ways.
But I guess the process cancellation is also asynchronous, and other processes (e.g. children of the interestingness test) might still have files open without specifying FILE_SHARE_DELETE. It's also impossible to delete a directory if there is a running process using it as current working directory.
In that case there is no solution other than waiting for those processes to quit.

I guess cvise could catch the error and then retry deleting at a later point in time (e.g. after the next reduction step).

[marxin]

Looking at the backtrace:

cvise/cvise/passes/lines.py

Lines 31 to 36 in 2aa333e

	backup = tempfile.NamedTemporaryFile(mode='w+', delete=False, dir=tmp)
	shutil.copyfile(test_case, backup.name)
	shutil.move(tmp_file.name, test_case)
	try:
	check_sanity()
	os.unlink(backup.name)

It's strange that one can't unlink the temporary file. After the return from check_sanity, the file should not be used by another process.

[dgrunwald]

It's used by the same process. Mode w+ doesn't allow deletion (Python doesn't allow opening files without prohibiting deletion), and the NamedTemporaryFile instance still has an open handle.
A backup.close() call would help (though that brings up the question: why not use delete=True instead?).

[marxin]

A backup.close() call would help (though that brings up the question: why not use delete=True instead?).

Good point, I've just rewritten the code in the pass. Can you please test it now if it makes a progress? If so, I can rework other places..

[dgrunwald]

Weirdly enough, now the file copy fails:

00:00:03 INFO ===< LinesPass::0 >===
Traceback (most recent call last):
  File "c:\Tools\cvise\bin\cvise", line 305, in <module>
    reducer.reduce(pass_group, skip_initial=args.skip_initial_passes)
  File "C:\Tools/cvise/share\cvise\cvise.py", line 143, in reduce
    self._run_additional_passes(pass_group['first'])
  File "C:\Tools/cvise/share\cvise\cvise.py", line 166, in _run_additional_passes
    self.test_manager.run_pass(p)
  File "C:\Tools/cvise/share\cvise\utils\testing.py", line 506, in run_pass
    self.state = self.current_pass.new(self.current_test_case, self.check_sanity)
  File "C:\Tools/cvise/share\cvise\passes\lines.py", line 53, in new
    self.__format(test_case, check_sanity)
  File "C:\Tools/cvise/share\cvise\passes\lines.py", line 32, in __format
    shutil.copy(test_case, backup.name)
  File "C:\Python310\lib\shutil.py", line 417, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "C:\Python310\lib\shutil.py", line 256, in copyfile
    with open(dst, 'wb') as fdst:
PermissionError: [Errno 13] Permission denied: 'C:\\temp\\creduce\\tmp3ouuezhu'

Maybe the delete=True parameter somehow also influences the FILE_SHARE_WRITE mode?
It's a bit unclear how the Python API maps to the Win32 concepts...

[marxin]

Maybe the delete=True parameter somehow also influences the FILE_SHARE_WRITE mode? It's a bit unclear how the Python API maps to the Win32 concepts...

Hmm, that's strange. It seems the FILE_SHARE_WRITE mode is not easily accessible from Python :/

[dgrunwald]

Note that the copy was successful previously when you were using delete=False.
I think the problem is unrelated to FILE_SHARE_WRITE (which I think Python always uses). Rather, it's that with delete=True the file is marked as delete-on-close, which apparently prevents any future open() unless that future open also allows deletion (FILE_SHARE_DELETE).
So using delete=False and instead deleting the file in a custom try-finally should work (as long as the file handle is closed prior to the deletion).