Add ARC auto-sync architecture #2570

R33v0LT · 2024-12-05T14:33:36Z

Your checklist for this pull request

I've documented or updated the documentation of every API function and struct this PR changes.
I've added the necessary ARC files for the disassembler to work properly
I've added ARC to python bindings
I've added ARC to auto-sync scripts
Dev fuzzing is done without errors
I've added tests that prove my fix is effective or that my feature works (if possible)

Detailed description

This PR adds support for the ARC architecture in capstone. It is worth noting that this architecture is experimental in lvm, that is, full support for all instructions is not fully implemented. Also, MC tests in llvm (like llvm_root/llvm/test/MC/Disassembler/ARC/ldst.txt ) have a format different from other architectures, which is why tests auto-generated by MCUpdater.py does not add instruction bytes to yaml. I think this format will change when ARC is fully supported
To successfully pass the autosync test, a merge of the following PR is required: capstone-engine/llvm-capstone#69

Test plan

There are no differences in the way capstone is built with the ARC architecture
MC tests were generated by MCUpdater. Only instruction bytes are manually added. All other tests are added manually
At the moment, all tests are passing successfully, there are CI/CD results to confirm

Closing issues

None

R33v0LT · 2024-12-11T13:33:40Z

@Rot127, fuzzing, as I expected, shows that those instructions that are not yet supported for ARC in llvm lead to incorrect decoding and program crash. llvm-mc from the official github also fails with an error. Should we do something now or ignore it until llvm updates?

Rot127 · 2024-12-11T13:52:37Z

You can fix it in our fork of LLVM. And open an issue in LLVM with a reference to the fix commit. They can cherry-pick it. If you have the time, you can also open a PR with the fix yourself.

R33v0LT · 2024-12-11T15:20:38Z

The problem is solved now. The DecodeGPR32RegisterClass return value was not checked (in llvm disassembler also)
I must reran fuzzing again to check if all is good now. If so, I'll marked this PR as ready for review

arch/ARC/ARCDisassembler.c

Rot127 · 2024-12-17T13:09:16Z

Do you consider this done? Then I would review it this week.

R33v0LT · 2024-12-17T19:13:38Z

Do you consider this done? Then I would review it this week.

Yes. I think it is done now

Rot127

Continue later (at arc.h). But so far looks very good!

Rot127 · 2024-12-22T12:31:20Z

arch/ARC/ARCDisassembler.c

+			Result = decodeInstruction(DecoderTable64, Instr,
+						   Insn64, Address, NULL);
+			if (MCDisassembler_Success == Result) {
+				;


Could you an issue please that we should also covert the LLVM_DEBUG stuff?
It seems to become more and more common to use.
No need to fix it here, but it nice to track.

LLVM_DEBUG is already present in the patches

arch/ARC/ARCGenCSFeatureName.inc

Rot127 · 2024-12-22T12:39:58Z

arch/ARC/ARCMapping.c

+	MI->MRI = MRI;
+
+	ARC_LLVM_printInst(MI, MI->address, "", O);
+	// ARC_add_cs_groups(MI);


Suggested change

// ARC_add_cs_groups(MI);

Rot127 · 2024-12-22T12:40:25Z

arch/ARC/ARCMapping.c

+
+void ARC_init_cs_detail(MCInst *MI)
+{
+	if (detail_is_set(MI)) {


Invert condition and return early please.

Rot127 · 2024-12-22T12:42:28Z

arch/ARC/ARCMapping.h

+static inline void add_cs_detail(MCInst *MI,
+				 int /* arc_op_group */ op_group, ...)
+{
+	if (!MI->flat_insn->detail)


Suggested change

if (!MI->flat_insn->detail)

if (!details_is_set(MI))

cstool/cstool.c

Rot127 · 2024-12-22T12:52:39Z

bindings/python/capstone/arc_const.py

+ARC_INS_ST_DI_AW = 195
+ARC_INS_ST_DI = 196
+ARC_INS_ST = 197
+ARC_INS_SUB1_ = 198


Suggested change

ARC_INS_SUB1_ = 198

ARC_INS_SUB1_ = 198

These instruction ids ending in _ look fishy. I suspect they end in a . or with a {. But they for sure end with a non alphabetical character. Can you check if this is expected? For some archs a . actually belongs to the mnemonic (PPC). For some it doesn't (ARM).

If the mnemonic should not end in a special character, you can check the getNormalizedMnemonic function in the PrinterCapstone. You can remove special chars there.

It works like that because of instructions like sub1.$cc. I can replace last dot if the mnemonic cointains it

Yes, please remove it.

Rot127 · 2024-12-22T12:53:41Z

bindings/python/capstone/arc.py

+    _fields_ = (
+        ('type', ctypes.c_uint),
+        ('value', ARCOpValue),
+        ('access', ctypes.c_uint8)


Suggested change

('access', ctypes.c_uint8)

('access', ctypes.c_int)

It was an enum in arc.h. Please either make it an uint8 or change it here to an int

Rot127 · 2024-12-22T12:54:02Z

bindings/python/capstone/arc.py

+
+class ARCOp(ctypes.Structure):
+    _fields_ = (
+        ('type', ctypes.c_uint),


Suggested change

('type', ctypes.c_uint),

('type', ctypes.c_int),

enums are ints if I am not mistaken.

Rot127 · 2024-12-22T12:54:46Z

bindings/python/capstone/arc.py

+    _fields_ = (
+        ('op_count', ctypes.c_uint8),
+        ('operands', ARCOp * 8),
+        ('update_flags', ctypes.c_bool),


Suggested change

('update_flags', ctypes.c_bool),

Rot127

This was a very straight forward review! Excellent job, well done!

Please add ARC in cs_v6_release_guide.md und New features.
Simple two liner is enough. Just please mention about the memory operands.

Otherwise, please address the other small requests.

Regarding the not yet implemented memory operand type: Do you want to add them? It complicates the details a bit. But on the other hand there seems only the MemOperandRI type. So not too much work I think? Or was there another problem?

Nonetheless, we can merge it like this. But we should have them before v6 Gold.
cc @XVilka @kabeor

Rot127 · 2024-12-22T13:37:17Z

suite/auto-sync/src/autosync/Targets.py

@@ -36,8 +38,10 @@
    "LoongArch": "LoongArch",
    "SystemZ": "SystemZ",
    "Mips": "Mips",
+    "ARC": "ARC",


Suggested change

"ARC": "ARC",

Rot127 · 2024-12-22T13:37:48Z

suite/auto-sync/src/autosync/cpptranslator/CppTranslator.py

+                        # from autosync.cpptranslator.patches.Helper import get_text
+                        # if patch.get_main_capture_name() == "template_def":
+                        #     print(get_text(self.src, q[0].start_byte, q[0].end_byte).decode())


Suggested change

# from autosync.cpptranslator.patches.Helper import get_text

# if patch.get_main_capture_name() == "template_def":

# print(get_text(self.src, q[0].start_byte, q[0].end_byte).decode())

Rot127 · 2024-12-22T13:37:58Z

suite/auto-sync/src/autosync/cpptranslator/CppTranslator.py

+                # if "template_declaration" in pattern:
+                #     print(self.tree.root_node)


Suggested change

# if "template_declaration" in pattern:

# print(self.tree.root_node)

Rot127 · 2024-12-22T13:39:24Z

suite/auto-sync/src/autosync/cpptranslator/patches/BadConditionCode.py

Please add a test for this new Patch in test_patches.py.

Rot127 · 2024-12-22T13:40:36Z

suite/auto-sync/src/autosync/cpptranslator/patches/FieldFromInstr.py

+            if len(param_list_caller.named_children) == 1:
+                # If function just return fieldFromInstruction(...)
+                inst_param: Node = param_list_caller.named_children[0]
+            else:
+                inst_param = param_list_caller.named_children[1]


Also, please add a test for this edge case.

Rot127 · 2024-12-22T13:41:41Z

suite/cstest/include/test_detail_arc.h

@@ -0,0 +1,60 @@
+// Copyright © 2024 Rot127 <[email protected]>


Your copyright

Rot127 · 2024-12-22T13:42:53Z

suite/cstest/src/test_detail_arc.c

@@ -0,0 +1,99 @@
+// Copyright © 2024 Rot127 <[email protected]>


Also your copyright please

Rot127 · 2024-12-22T13:44:06Z

suite/test_corpus3.py

@@ -134,6 +135,8 @@ def test_file(fname):
        ("CS_ARCH_ALPHA", "CS_MODE_BIG_ENDIAN"): 56,
        ("CS_ARCH_HPPA", "CS_MODE_HPPA_11+CS_MODE_BIG_ENDIAN"): 57,
        ("CS_ARCH_HPPA", "CS_MODE_HPPA_20+CS_MODE_BIG_ENDIAN"): 58,
+        ("CS_ARCH_ARC", "CS_MODE_LITTLE_ENDIAN"): 59,
+        ("CS_ARCH_ARC", "CS_MODE_BIG_ENDIAN"): 60,


Suggested change

("CS_ARCH_ARC", "CS_MODE_BIG_ENDIAN"): 60,

Rot127 · 2024-12-22T13:46:53Z

tests/details/arc.yaml

Do these tests cover all the possible OP_GROUPS? If no, please add one test for each op group.
Otherwise, all good.

Also, please add tests which checks the implicit reads, writes and groups.

Rot127

Nice! Almost done. Just those few nitpicks.

Also, please mark the suggestions as resolved when done.

Rot127 · 2024-12-29T15:22:00Z

arch/ARC/ARCMapping.c

+	switch (op_group) {
+	default:
+		printf("ERROR: Operand group %d not handled!\n", op_group);
+		assert(0);


Suggested change

assert(0);

CS_ASSERT_RET(0);

Rot127 · 2024-12-29T15:25:06Z

include/capstone/arc.h

+	ARC_INS_ADD_,
+	ARC_INS_ADD_F,
+	ARC_INS_ADD,
+	ARC_INS_AND_,


Please don't forget about those ones.

Rot127 · 2024-12-29T15:26:22Z

suite/auto-sync/src/autosync/cpptranslator/patches/BadConditionCode.py

+class BadConditionCode(Patch):
+    """
+    Patch   return BadConditionCode
+    to      assert(0 && "Unknown condition code passed")


Suggested change

to assert(0 && "Unknown condition code passed")

to CS_ASSERT(0 && "Unknown condition code passed")

Rot127 · 2024-12-29T15:27:14Z

suite/auto-sync/src/autosync/cpptranslator/patches/BadConditionCode.py

@@ -0,0 +1,32 @@
+# Copyright © 2024 Dmitry Sibitsev <[email protected]>


Your email please :)

github-actions bot added python bindings CS-core-files auto-sync Auto-Sync-files Auto-Sync LLVM-generated-files Documentation Github-files Github related files labels Dec 5, 2024

R33v0LT marked this pull request as draft December 5, 2024 14:47

R33v0LT marked this pull request as ready for review December 13, 2024 09:44

Rot127 reviewed Dec 13, 2024

View reviewed changes

arch/ARC/ARCDisassembler.c Outdated Show resolved Hide resolved

XVilka mentioned this pull request Dec 14, 2024

Rewrite GPL-only code rizinorg/rizin#2704

Open

10 tasks

R33v0LT force-pushed the auto-sync-arc branch from aa93895 to e2fb2d7 Compare December 17, 2024 10:36

R33v0LT added 16 commits December 17, 2024 13:40

Add ARC files

fbb3991

Added ARC files for successful compilation

3b731d9

Refactor ARC files

2363031

Add ARC c/cs tests

173b587

Add ARC python test/bindings

02b498f

Add ARC to CI/CD

c94c5df

Avoid omitting parameter names

190516e

Update cs files

cef71a5

Fix ARC bugs

eeaff5d

Update ARC python bindings

cdec496

Refactor and update ARC test files

982f3f6

Add detail flag to arc test

8a515b9

Fix ARC test problems

8b35577

Fix ARCMapping compile error

4c094b8

Replace __CHAR_BIT__ to CHAR_BIT

d4cd8ba

Add credits and ARC info

f457e17

R33v0LT added 8 commits December 17, 2024 13:41

Update ARC to match the latest next

ee448e6

Python formatting

328ae07

Remove asserts on 'Unknown condition code passed'

6973284

Add ARC to some more documentation

e4aef04

Add ARC to Targets constants

0f29088

Add ARC support to llvm-tblgen

165c01b

Replace asserts & add Expr handling

d98e2be

Check DecodeGPR32RegisterClass return value

06afa37

R33v0LT force-pushed the auto-sync-arc branch from e2fb2d7 to 06afa37 Compare December 17, 2024 10:49

Fix fieldFromInstruction patch

d977b77

R33v0LT closed this Dec 17, 2024

R33v0LT reopened this Dec 17, 2024

Rot127 mentioned this pull request Dec 22, 2024

auto-sync progress tracker: Refactor and implement architectures #2015

Open

47 tasks

Rot127 requested changes Dec 22, 2024

View reviewed changes

R33v0LT added 3 commits December 24, 2024 18:34

Refactor ARC

0b6655d

Reformat python files

a69e9d1

Fix incorrect import

43da7fa

R33v0LT requested a review from Rot127 December 25, 2024 12:55

Rot127 requested changes Dec 29, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ARC auto-sync architecture #2570

Add ARC auto-sync architecture #2570

R33v0LT commented Dec 5, 2024 •

edited

Loading

R33v0LT commented Dec 11, 2024

Rot127 commented Dec 11, 2024

R33v0LT commented Dec 11, 2024

Rot127 commented Dec 17, 2024

R33v0LT commented Dec 17, 2024

Rot127 left a comment

Rot127 Dec 22, 2024

R33v0LT Dec 24, 2024

Rot127 Dec 22, 2024

Rot127 Dec 22, 2024

Rot127 Dec 22, 2024

Rot127 Dec 22, 2024 •

edited

Loading

R33v0LT Dec 24, 2024 •

edited

Loading

Rot127 Dec 28, 2024

Rot127 Dec 22, 2024

Rot127 Dec 22, 2024

Rot127 Dec 22, 2024

Rot127 left a comment

Rot127 Dec 22, 2024

Rot127 Dec 22, 2024

Rot127 Dec 22, 2024

Rot127 Dec 22, 2024

Rot127 Dec 22, 2024

Rot127 Dec 22, 2024

Rot127 Dec 22, 2024

Rot127 Dec 22, 2024

Rot127 Dec 22, 2024

Rot127 left a comment

Rot127 Dec 29, 2024

Rot127 Dec 29, 2024

Rot127 Dec 29, 2024

Rot127 Dec 29, 2024

	# from autosync.cpptranslator.patches.Helper import get_text
	# if patch.get_main_capture_name() == "template_def":
	# print(get_text(self.src, q[0].start_byte, q[0].end_byte).decode())

		# if "template_declaration" in pattern:
		# print(self.tree.root_node)

		@@ -0,0 +1,60 @@
		// Copyright © 2024 Rot127 <[email protected]>

		@@ -0,0 +1,99 @@
		// Copyright © 2024 Rot127 <[email protected]>

	to assert(0 && "Unknown condition code passed")
	to CS_ASSERT(0 && "Unknown condition code passed")

		@@ -0,0 +1,32 @@
		# Copyright © 2024 Dmitry Sibitsev <[email protected]>

Add ARC auto-sync architecture #2570

Are you sure you want to change the base?

Add ARC auto-sync architecture #2570

Conversation

R33v0LT commented Dec 5, 2024 • edited Loading

R33v0LT commented Dec 11, 2024

Rot127 commented Dec 11, 2024

R33v0LT commented Dec 11, 2024

Rot127 commented Dec 17, 2024

R33v0LT commented Dec 17, 2024

Rot127 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Rot127 Dec 22, 2024 • edited Loading

Choose a reason for hiding this comment

R33v0LT Dec 24, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Rot127 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Rot127 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

R33v0LT commented Dec 5, 2024 •

edited

Loading

Rot127 Dec 22, 2024 •

edited

Loading

R33v0LT Dec 24, 2024 •

edited

Loading