WIP: Refactor /ao for regex assembly search (#6663)

[Yashwin-i]

[x] I've read the guidelines for contributing to this repository.

[x] I made sure to follow the project's coding style.

[ ] I've documented every RZ_API function and struct this PR changes. (N/A - no RZ_API changes)

[ ] I've added tests that prove my changes are effective (required for changes to RZ_API). (WIP)

[ ] I've updated the Rizin book with the relevant information (if needed).

[x] I've used AI tools to generate fully or partially these code changes and I'm sure the changes are not copyrighted by somebody else.

Detailed description

This PR implements the core functionality for issue #6663 (Refactor /ao for regex assembly search). I have built a functional /ao command that correctly reads memory, disassembles instructions, and matches them against the user's regex.

(Note: The original issue requested multithreading, but per @wargio's feedback, RzArch is not yet thread-safe. Therefore, this has been refactored into a single-threaded static function inside cmd_search.c to respect the current architecture).

What is currently working:
-> Added a new static helper do_asm_regex_search directly in cmd_search.c.
-> Disassembles current memory blocks using rz_asm_disassemble.
-> Compiles and executes extended regex queries against op.buf_asm.
-> The command is abortable using rz_cons_is_breaked().
-> Seamlessly integrates into the existing /ao command switch using public Rizin APIs.

(Note: I used Google's Gemini AI assistant to help format my code, and I have reviewed the code to ensure architectural soundness).

Test plan

Build the project using Meson/Ninja and enter the developer environment.

Open an empty memory block: rizin -

Verify default bytes with pd 3 (e.g., 0x00000000 jnbe 0x63).

Run the new regex search to match instructions:

[0x00000000]> /ao jnbe
0x00000000 jnbe 0x63

You can also write custom assembly as a single string and search for it:

[0x00000000]> "wa mov eax, 1; mov ebx, 2; add eax, ebx"
[0x00000000]> /ao mov

Closing issues

Addresses #6663

[ksd-2005]

Ran into the same question when looking at this issue, so what I found is :
in the build graph

librz/arch/meson.build already lists rz_search_dep among rz_arch's dependencies, and librz/search/meson.build only depends on rz_util, rz_io, rz_magic, rz_crypto, rz_hash. So an asm search collection can't live in librz/search/ next to string_search.c — pulling RzAsm in there would make the dependency cycle rz_search -> rz_arch -> rz_search.

That leaves your first option (expose the collection API to core) as the workable one.
RzSearchHit, RzSearchOpt, RzSearchFindOpt and rz_search_on_io() are already RZ_API in rz_search.h. The only pieces behind the module boundary are rz_search_collection_new_bytes_space() (RZ_IPI, librz/search/collection.c:51) and the RzSearchFindBytesCallback typedef in search_internal.h.

On thread safety: core->rasm can't be shared across find threads - rz_asm_set_pc() mutates it, The collection's private user data would need one RzAsm per worker, configured from core's arch/bits/cpu/syntax/endianness up front.

For transparency: I picked this issue up in the issue thread on Aug 11 and I'm working on an implementation along these lines. Happy to combine efforts rather than duplicate - let me know what you'd prefer.

[Yashwin-i]

Thanks for the detailed architectural analysis! I already have the core disassembly and regex engine prototyped and working here in this draft. Regarding the module boundary, I want to hold off on promoting rz_search_collection_new_bytes_space to the public RZ_API until the core maintainers confirm that is their preferred approach, as it affects the public ABI. Let's see what they advise before we rewire the headers!

[ksd-2005]

On ABI point:

Promoting an RZ_IPI symbol to RZ_API is additive to the export table rather than a break.
The build uses -fvisibility=hidden (once check the meson.build file), so those symbols are just hidden from librz_search.so, marking one RZ_API exports it without touching any existing signature or struct layout.
So it's a question of whether maintainers want that symbol in the public surface , so i think it doesn't lead to ABI break.

Since it's small change , so I will keep building on my branch against the promoted symbol so there's something concrete to look at when they check in - easy to change later if needed.

On coordinating:
are you planning to carry on with the threaded implementation yourself? I am interested in working on the threaded version, want to make sure we're both not writing the same code.

[Yashwin-i]

Thanks for the heads-up! Since altering the headers impacts the public API surface, I'm going to hold off on doing that on my branch and wait for the maintainers to weigh in on their preferred architectural direction first. Let's see what they advise before we coordinate any further code changes.

[Yashwin-i]

Hey @ksd-2005, I’ve actually already coded the multithreaded RzSearchCollection approach locally to test it out. I also sorted out the cascading header dependencies (moving RzSearchIsEmptyCallback and RzSearchFreeCallback over to rz_search.h to keep the compiler happy).

That said, since exposing these internal collections alters the public ABI, I am holding off on pushing the changes to this PR until one of the core maintainers gives the green light. I’ve got the fully working code ready to push, so no need for you to duplicate the effort!

@Rot127 (or other maintainers) - let me know if you are comfortable with this API promotion, or if there is a different internal bridge you'd prefer I use!

[wargio]

We cannot multithread asm search as RzArch is not thread safe.(yet)

[wargio]

this should be in cmd_search.c as it makes no sense to have it in a new file. also this should be a static function.

[Yashwin-i]

Hey @wargio, thanks for the review and the clarification on RzArch thread-safety!

Since the original issue description explicitly mentioned multithreading, I initially went down that path by creating a separate file and trying to hook into the search collection APIs. However, your feedback makes perfect sense. I've completely scrapped the multithreaded approach for now.

I did a clean reset of my branch, moved the single-threaded implementation directly into cmd_search.c as a static function, and force-pushed the clean commit. Let me know if this implementation looks good to go or if you'd like any other tweaks!

[Rot127]

Sorry, I thought RzArch is already thread safe. Will review this next week. Please ping me if I forget it.

[wargio]

RzArch is not implemented yet, and the current RzAnalysis/RzAsm code is mostly thread safe, but not all of it. also there are still some archs that are not thread safe. (see #4055)