@kabeor @aquynh @Rot127 I propose to make the next release, with auto-sync changes a 6.0, not 5.1 because:

• There are slight API changes
• The amount of code changes is HUGE

https://github.com/orgs/capstone-engine/projects/1 - then it would need to be updated too.

please can you summarize the API changes here?

@aquynh

ARM

• Enum changes:
  • ARM_CC_ -> ARMCC
  • System registers are renamed to match C namespaces. Also group Banked and system registers into different groups.
  • Some instr. enum entries no longer exist (e.g. VPUSH, VPOP).
• Some instruction groups which are not part of LLVM were removed (e.g. GROUP_INT)
  • Groups like RET, INT should be added via Mapper separately.
• Feature groups like ARM_GRP_CRC are renamed to match LLVM nameing: ARM_FEATURE_HasCRC
• Features are now checked more strictly (V8, MCLASS, ARM, THUMB) because instruction aliases are supported now. And those alias might change depending on enabled features.
• The memory offset register or immediate are now always part of the memory operand. Offsets or index operands are no longer separated. Before, only offset ops which were within the [] brackets were added.
• writeback is part of detail and no longer of detail.arm.
• Register alias not defined in LLVM (r15 = pc etc.) are no longer printed as default. Must be enabled via CS_OPT_SYNTAX_CS_REG_ALIAS or -a for the cstool.
• The immediate value of operands is no of type uint32_t, no longer int32_t.

PPC

• Predicate enums members are renamed. They now use the LLVM name (e.g. PPC_BC_NU_PLUS -> PPC_PRED_NU_PLUS).
• Branch conditions are now saved in more detail in cs_ppc.bc.
• The base register of an PPC memory operand was not present if reg = r0. This is fixed now.
• ppc_ops_crx is removed (wasn't used).

AArch64

• Renamed all ARM64 -> AArch64 (for filenames, enums variable names). Necessary to be consistent with LLVM.
• SME operands changed (contin more detail, terminology is closer to the docs).
• System operands change (now categorized into SysAlias, SysImm, SysReg).

This list is also part of the PR.

@aquynh

ARM

• Enum changes:

  • ARM_CC_ -> ARMCC
  • System registers are renamed to match C namespaces. Also group Banked and system registers into different groups.
  • Some instr. enum entries no longer exist (e.g. VPUSH, VPOP).

• Some instruction groups which are not part of LLVM were removed (e.g. GROUP_INT)

  • Groups like RET, INT should be added via Mapper separately.

• Feature groups like ARM_GRP_CRC are renamed to match LLVM nameing: ARM_FEATURE_HasCRC

• Features are now checked more strictly (V8, MCLASS, ARM, THUMB) because instruction aliases are supported now. And those alias might change depending on enabled features.

• The memory offset register or immediate are now always part of the memory operand. Offsets or index operands are no longer separated. Before, only offset ops which were within the [] brackets were added.

• writeback is part of detail and no longer of detail.arm.

• Register alias not defined in LLVM (r15 = pc etc.) are no longer printed as default. Must be enabled via CS_OPT_SYNTAX_CS_REG_ALIAS or -a for the cstool.

• The immediate value of operands is no of type uint32_t, no longer int32_t.

PPC

• Predicate enums members are renamed. They now use the LLVM name (e.g. PPC_BC_NU_PLUS -> PPC_PRED_NU_PLUS).
• Branch conditions are now saved in more detail in cs_ppc.bc.
• The base register of an PPC memory operand was not present if reg = r0. This is fixed now.
• ppc_ops_crx is removed (wasn't used).

AArch64

• Renamed all ARM64 -> AArch64 (for filenames, enums variable names). Necessary to be consistent with LLVM.
• SME operands changed (contin more detail, terminology is closer to the docs).
• System operands change (now categorized into SysAlias, SysImm, SysReg).

This list is also part of the PR.

cant we avoid breaking compatibility?

@aquynh The short answer is no.

But let me go into more details also for others:

The problem with automatic Capstone updates is that due to the C and C difference we have many cases to handle when C is not equivalent to C .

To reduce those cases we need to be as close to the original C syntax and semantic as possible. Because every renaming (i.e. enum values), semantic and overall design changes, almost always add manual work during an update.

This is why those breaking changes are needed.
Each of them moves Capstone code semantically or syntactically closer to the LLVM definitions.

This is of cause a pain for compatibility, but it is definitely worth it in the long run.
Because:

• All auto-sync archs are semantically pretty much equivalent to LLVM.
  • Which gives more correct results.
  • Results are comparable to llvm-objdump.
  • Eases test generation.
  • Easier to extend CS in the future with other information known in LLVM (see ARM opcode & operand encoding information (based on auto-sync) #2045 which adds instruction encodings for all auto-sync archs without much trouble).
• Unifies how modules work, so we can share code between them (see the new Mapping.* files).
• Reduces the effort to update (less manual work, due to less edge cases). So hopefully more people update their archs.

Here more detail to each breaking change.

Enum changes:

Done to match LLVM naming. It saves us to change enum names over several files whenever we update.

ARM

Some instruction groups which are not part of LLVM were removed (e.g. GROUP_INT)

Capstone unique instr. groups (like RET, INT) are added via Mapper separately (is on the toddo list). Because they are not defined in LLVM, we can not generate them without adding exceptions again.

Features are now checked more strictly (V8, MCLASS, ARM, THUMB) because instruction aliases are supported now. And those alias might change depending on enabled features.

Simple necessity, because with the new instructions the same bytes have a valid decoding depending on the enabled features.

The memory offset register or immediate are now always part of the memory operand. Offsets or index operands are no longer separated. Before, only offset ops which were within the [] brackets were added.

Move closer to the LLVM logic. The disponent of a memory access doesn't need to be within the [] brackets (e.g. strt fp, [sp], 4). But the disponent is defined as part of the memory operand. This was incorrectly represented in CS before.

writeback is part of detail and no longer of detail.arm.

We support now the concept of Tied operands (the way LLVM describes writeback registers). So writeback information is now known for all auto-sync archs.

Register alias not defined in LLVM (r15 = pc etc.) are no longer printed as default. Must be enabled via CS_OPT_SYNTAX_CS_REG_ALIAS or -a for the cstool.

As said before, modules will be more equivalent to the llvm-objdump results. Also the register naming and the decoded asm string.

The immediate value of operands is not of type uint32_t, no longer int32_t.

See: #2056

PPC

Branch conditions are now saved in more detail in cs_ppc.bc.

Just a nice feature we are now able to provide.

The base register of an PPC memory operand was not present if reg = r0. This is fixed now.

A semantical fix. The base register should have been set.

ppc_ops_crx is removed (wasn't used).

Wasn't used.

AArch64

Renamed all ARM64 -> AArch64 (for filenames, enums variable names). Necessary to be consistent with LLVM.

This is a big one. But having two names for the same architecture in the code is a nightmare for generation. Also it just doesn't bring any value. Being closer to LLVM is the choice here.

SME operands changed (contain more detail, terminology is closer to the docs).

Again a nice feature addition because we save more detail. Being closer to the official docs when it comes to naming eases integrating Capstone in other projects.

System operands change (now categorized into SysAlias, SysImm, SysReg).

Again, move to LLVM semantic because:

• It wasn't correct before (system immediate and other alias were incorrectly identified as system registers or not categorized at all).
• This mimics the inheritance of system operands within the LLVM code.
• Eases generation.

Personally, I think that Capstone will become more and more irrelevant as disassembler engine if we:

• Do not modernize it (update archs, update testing)
• Provide people a relative easy way to add more features and architectures (e.g. the instruction encoding, instruction form information (for PPC) and others).

If we do not go through the pain of breaking compatibility once, to gain log term improvements, Capstone just won't be competitive to other disassemblers in the future.

lets take one example: we want to rename ARM_CC to ARMCC.

can we have compatibility by keeping ARM_CC, and add (new) ARMCC, so everyone is happy?

I answer your suggestions later. On the phone it is difficult to write well. As a general note though: I second @XVilkas here. I work on this now for half a year and opened the ARM PR as draft after two-three months. Exactly to ask for this kind of feedback, suggestions on design and other choices. This is also why I added the list of breaking changes to the PR description, updated it continuously and asked for feedback on the big ones. So no one needed to read through the code and can save time. Also, I think I made clear that I am more than happy to provide detailed answers and provide more details if requested. As I stated in the ARM PR, my time I can spend on this is limited (until end of July). And we want and need to start building on it in Rizin. With all due respect, but there were at least four and a half months to discuss a big decision like this. And we really need to carry on. 26 Jun 2023 17:23:51 Nguyen Anh Quynh @.>:
…
I never say it is not merged, Anton. On Mon, Jun 26, 2023, 23:12 Anton Kochkov @.> wrote: > If auto-sync work is not merged, I am afraid we have to fork the capstone. > It's your choice - you want updated architectures or not. > > — > Reply to this email directly, view it on GitHub > <#2015 (comment)>, > or unsubscribe > https://github.com/notifications/unsubscribe-auth/ABNQNYGJCQLEGSSBCAUYCGTXNGRFNANCNFSM6AAAAAAX3POTSU > . > You are receiving this because you were mentioned.Message ID: > @.***> > — Reply to this email directly, view it on GitHub[#2015 (comment)], or unsubscribe[https://github.com/notifications/unsubscribe-auth/AK5ET6CWWG5MAD4QVHRZWLTXNGSQLANCNFSM6AAAAAAX3POTSU]. You are receiving this because you were mentioned.[Tracking image][https://github.com/notifications/beacon/AK5ET6B4GEQVBVWGEALNKVLXNGSQLA5CNFSM6AAAAAAX3POTSWWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTS72PDV4.gif]

understood - we are all short of time, especially those who are maintaining this project without paying, in spare time.

we will try to merge this in July.

@aquynh I agree that the compatibility concern is very valid. But since I asked in the PRs and got no push back I assumed modernizing is more important.

I would propose to finish up the v5 release and add a note that v6 will bring big change.

If people try out the next branch and figure they rely desperately on some of those old stuff, we can think about how to make it compatible for them in a different branch. Or guide them to do this on their own.

can we have compatibility by keeping ARM_CC, and add (new) ARMCC, so everyone is happy?

This specifically is not just a syntax change, but also the values change (ARM_CC_invalid = 0 is removed with ARMCC_UNDEF = 15). Reversing this also means to:

1. Have two different CC enums (for CS and LLVM)
2. Having to translate between each of those enums.

But there is little meaning in keeping this complexity (other then compatibility reasons of cause).

Because it takes longer than I expected, I suggest targeting upcoming LLVM 17.0 release with a few nice updates in ARMv9 and RISC-V extensions: https://discourse.llvm.org/t/llvm-17-0-0-release-planning-and-update/71762

• July 25th - release/17.x branch created
• July 27th - 17.0.0-rc1 released
• August 9th - 17.0.0-rc2 released

https://llvm.org/docs/ReleaseNotes.html#non-comprehensive-list-of-changes-in-this-release

@XVilka In that way, should we merge #1949 after 17.x release?

Suggest to continue this topic at capstone-engine/llvm-capstone#11

@Rot127 As discussed in the rizin Mattermost I have some free time and would like to update the BPF support to auto-sync 🙌

Great! Please go ahead. Start with the documentation and let me know if you need help with something. If something is not clearly written or needs clarification in you opinion please let me know as well. The docs haven't been read by many people yet. So any fresh looks at it are welcome.

Also notify me when you have a fork pushed and a draft PR opened. So we can link it here. Draft PR is preferred, because we can comment on it more easily.