Fix AVX detection

The old/faulty code would try to use AVX/AVX2 if either the SSE bit or the AVX bit were set in XCR0, but did not check if both bits were set. In most cases, this still worked, but on some machines, enabling linux kernel mitigations for the "gather data sampling" vulnerability results in only the SSE bit but not the AVX bit being set, thus resulting in an illegal instruction and crashing the application. Fix this by checking that both bits are set. Fixes: 4bbb590 ("Proper check of CPU's AVX2 feature support (with MSVC support)") Signed-off-by: Pascal Ernster <[email protected]>
aklomp · Nov 5, 2023 · 9003f9b · 9003f9b
1 parent e77bd70
commit 9003f9b
Showing 1 changed file with 1 addition and 1 deletion.
diff --git a/lib/codec_choose.c b/lib/codec_choose.c
@@ -208,7 +208,7 @@ codec_choose_x86 (struct codec *codec)
 		if (ecx & bit_XSAVE_XRSTORE) {
 			uint64_t xcr_mask;
 			xcr_mask = _xgetbv(_XCR_XFEATURE_ENABLED_MASK);
-			if (xcr_mask & _XCR_XMM_AND_YMM_STATE_ENABLED_BY_OS) {
+			if ((xcr_mask & _XCR_XMM_AND_YMM_STATE_ENABLED_BY_OS) == _XCR_XMM_AND_YMM_STATE_ENABLED_BY_OS) { // check multiple bits at once
 				#if HAVE_AVX512
 				if (max_level >= 7) {
 					__cpuid_count(7, 0, eax, ebx, ecx, edx);