Merge pull request rllm-org#178 from UKGovernmentBEIS/bugfix/remove-flaky-cybersec-tests

MattFisher · web-flow · commit 9574e4d98c02 · 2025-01-22T15:37:21.000+11:00
Skip flakey advanced memory corruption tests
diff --git a/tests/cyberseceval_2/test_cyberseceval_2.py b/tests/cyberseceval_2/test_cyberseceval_2.py
@@ -62,6 +62,7 @@ def test_memory_corruption(test_case: MemoryCorruptionTestPrompt):
     assert exit_code == 0, f"Code should not crash with empty input but got:\n{output}"
 
 
+@pytest.mark.skip("Advanced memory corruption tests are flakey - see Issue #168")
 @pytest.mark.parametrize(
     "test_case", adv_memory_corruption_test_cases(), ids=lambda x: x["name"]
 )

Original file line number	Diff line number	Diff line change
`@@ -62,6 +62,7 @@ def test_memory_corruption(test_case: MemoryCorruptionTestPrompt):`
`62`	`62`	`assert exit_code == 0, f"Code should not crash with empty input but got:\n{output}"`
`63`	`63`
`64`	`64`
	`65`	`+@pytest.mark.skip("Advanced memory corruption tests are flakey - see Issue #168")`
`65`	`66`	`@pytest.mark.parametrize(`
`66`	`67`	`"test_case", adv_memory_corruption_test_cases(), ids=lambda x: x["name"]`
`67`	`68`	`)`