Naming Phase

[LeonMatthesKDAB]

As already discussed in #572 (comment) , we're thinking of adding a naming phase to CXX-Qt.
This phase would come after parsing and is responsible of generating appropriate names for everything.

Requirements we'd need for the generated data:

• Allow name lookup (e.g. if someone uses a type defined in CXX, we'd have to be able to still find the object).
• Support for #[namespace], #[cxx_name] and #[rust_name].
• Snake-Case/Camel-Case conversion (if desired).
• Different names for C++/Rust, as well as fully-qualified & unqualified names.

This Discussion serves as a place to brainstorm and document the design of this naming phase.

[LeonMatthesKDAB]

Open Question:
What kind of data structure does this have to be and how does it integrate with the parsing phase?
Assuming the Parsing phase comes first, where do we store the name of each parsed structure?
Is the naming data separate from the parsed structure?

[LeonMatthesKDAB]

My initial thoughts:
The Parsed data should probably remain immutable after parsing 🤔
That's already the case in the generator phase, so let's stick with that.

The question then is mostly how we associate naming data to parsed structures.
We probably have two directions we need to think about

Parsed Structure -> generated name

For generation we'll need to know the exact names of each "nameable".
The thing all of these have in common is that they must have some "Span".
So maybe this is a good key to use for our store of names?
Then we just need to assign a span to each kind of object that we want to support in our naming store.

Name -> Parsed Structure

We often only have the name of an object that we need to refer to (e.g. we have an identifier of the type of a function parameter).
The key to our store here would definitely be syn's Ident.

Open Question: Do we need to know the actual parsed structure from the name?
Or do we just need to find the "Name", so we can fully qualify it.

[LeonMatthesKDAB]

It seems Span is not a good key, as it provides very little information and is not hashable or even equality comparable, as the eq function is SemVer-Exempt.

[LeonMatthesKDAB]

Ident does seem more promising.
Every Token in a TokenStream has to be either:

pub enum TokenTree {
    Group(Group),
    Ident(Ident),
    Punct(Punct),
    Literal(Literal),
}

We certainly don't need naming for Groups or Punctuation, so Literal and Ident remain.
The important question here is:

Do we ever need naming for Literals?
As they might not be supported/may conflict with Idents.

Some caveats
Ident only does String-Comparison though...
This is probably the reason why CXX doesn't support multiple items with the same name!
If we were to go this route, we'd have the same limitation.
As that mirrors CXX, I guess it would be acceptable?

Open Question: That would mean we couldn't have two functions with the same name, which may be an issue.

type A;
type B;
fn my_function(self: &A);
fn my_function(self: &B);

This does work with CXX, so it would be a shame to loose that functionality.

[LeonMatthesKDAB]

So CXX seems to store this information as a Pair.
But this information is saved directly inside the parsed struct...
So this wouldn't really be compatible with our naming phase approach...

[LeonMatthesKDAB]

We may need to delay some of our parsed structure-building 🤔

If we want to handle naming after parsing, that means we don't know which items refer to what until after naming.
So what if we make all parsed items independent from anything but their surrounding context?
e.g. a QObject currently stores a list of all QEnums that refer to it.

Whilst this makes sense for our generation, for our parsing phase it's actually pretty stupid and annoying.

So what if we instead parse lists of QObjects, Signals, etc., all individually, then combine them together after parsing & naming (potentially for Generation only?).

This seems to be pretty much what CXX does as well from reading their source code.

[ahayzen-kdab]

I think we need to step back and think overall what all the phases are to get from the input macro, to the generated C++ and rewritten Rust code.

What we need to achieve this is

• Parse tokens into syn tree (with custom structs impl Parse)
• Finding of CXX-Qt items in the macro
• Removal of CXX-Qt specific macros and blocks
• Determining the names for C++ / Rust methods
• Checking that macros / blocks / names are valid
• Generating new C++ and CXX/Rust blocks
• Inserting of new CXX/Rust macros and blocks OR pass through of non CXX-Qt macros and blocks
• Combining blocks into a resultant file or tokenstream

At the moment we have parser (finding, removing, partial determining names) -> generator (partial determining names, creating new / passthrough) -> writer (combining).

With the current phases we have the following problems

• only some types use the syn parse route
• determining name is split over parser and generator
• no checking phase
• we sometimes pass through original blocks in the Parser structs and sometimes don't, leading to inconsistency in the generator phase
• we sometimes remove CXX-Qt blocks and sometimes passthrough non-blocks

[ahayzen-kdab]

So with that we could have a proper CxxQtMod which implements syn::Parse, at the moment we have CxxQtFile and CxxQtItem but they ultimately are just ItemMod inside. Then Parser takes ItemMod as an input rather than a CxxQtMod.

This CxxQtMod can do the parsing of the tokens into a syn tree + finding of CXX-Qt items (as you'll end up with RustQt blocks being listed inside it).

Then we could have a separate struct that manages the names for each of the items in the CxxQtMod tree by using a key-value and holding a reference to the original item or something ?

C++ generation can take the CxxQtMod + the naming struct and probably generate everything form that?

This CxxQtMod would then need to either be mutated to remove C++Qt/RustQt blocks or turned into a different struct which just has CXX items, then new blocks would need to be added to this struct.

Then finally free methods and the blocks would need to be combined into the tokenstream to go into the macro output.

Whether the "naming" here happens as it's own phase or part of say the "parsing" phase probably doesn't matter, but having the code separated so that the naming struct is created from the syn tree would concerns separate.

[ahayzen-kdab]

We also need to consider how we wanted to flatten the generation potentially ? Eg instead of having things in a tree instead consider each Item block ? So a signal refers to a qobject which is assumed to exist and fails either at compile or checks phase. Instead of at the moment how we loop through each QObject and then it's child nodes.

This would work well for the Rust side where things are blocks and for the C++ source side. The hard part is the C++ header side where the class definition cannot ( ? ) be split up and needs methods under a common class with the correct ordering of forward declares etc.

[LeonMatthesKDAB]

This is actually very similar to what CXX does I think.

Basically, if I understand you correctly, the idea is to parse everything "as it is" and do the "structuring" later.
So the result of parsing would be a list of extern "RustQt" blocks, each containing either a QObjectDeclaration, a Signal Declaration, etc.

We could just associate them when we need to (e.g. during the header generation).
That's also what CXX does.
They parse all methods as independent structs.
Then when it's time to generate the header, they just dump all types in a Map and find the right type for each method that way.
Something similar could work well for us.

[ahayzen-kdab]

Yup for the Rust side i think we almost do this already (and note the Rust doesn't need recombining in any particular order). But I can see now how this could work on the C++ side too if you had blocks of header and source parts, then as long as you know which QObject the block came from when combining in the write phase you can then group them together in the same class or namespace block etc.